> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lighton.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create text tokens

> This endpoint can be used to convert strings into tokens.

It is a simple proxy forwarding your requests to the desired model.

Any LightOn model is deployed on a vLLM-based image.

**Supported Input:**
- `prompt`: Simple text string to tokenize
- `messages`: Array of chat messages to tokenize (alternative to prompt)



## OpenAPI

````yaml /api-reference/openapi-v2.yaml post /api/v2/tokenize
openapi: 3.0.3
info:
  title: Paradigm API
  version: xenial-xerus (v2)
  description: >-
    A versatile and adaptable tool designed to integrate Generative AI into your
    applications
servers:
  - url: https://paradigm.lighton.ai
security: []
tags:
  - name: Models
    description: Operations about AI models
  - name: Files Search
    description: Operations about files search
  - name: Files
    description: Operations about files
  - name: Upload Sessions
    description: Operations about upload sessions
  - name: Workspaces
    description: Operations about workspaces
  - name: Users
    description: Operations about users
  - name: Companies
    description: Operations about companies
  - name: SCIM
    description: Operations about SCIM
  - name: Feedbacks
    description: Operations about feedbacks
  - name: Reporting
    description: Operations about reporting
  - name: Monitoring
    description: Operations about monitoring
  - name: Platform Status
    description: Operations about platform status
paths:
  /api/v2/tokenize:
    post:
      tags:
        - Models
      summary: Create text tokens
      description: |-
        This endpoint can be used to convert strings into tokens.

        It is a simple proxy forwarding your requests to the desired model.

        Any LightOn model is deployed on a vLLM-based image.

        **Supported Input:**
        - `prompt`: Simple text string to tokenize
        - `messages`: Array of chat messages to tokenize (alternative to prompt)
      operationId: api_v2_tokenize_create
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TokenizeRequest'
            examples:
              TextTokenizationExample:
                value:
                  model: alfred-4.2
                  prompt: Hello, how are you today?
                summary: Text tokenization example
              MessagesTokenizationExample:
                value:
                  model: alfred-4.2
                  messages:
                    - role: system
                      content: You are a helpful assistant.
                    - role: user
                      content: Hello!
                summary: Messages tokenization example
          application/x-www-form-urlencoded:
            schema:
              $ref: '#/components/schemas/TokenizeRequest'
          multipart/form-data:
            schema:
              $ref: '#/components/schemas/TokenizeRequest'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TokenizeResponse'
          description: ''
      security:
        - bearerAuth: []
components:
  schemas:
    TokenizeRequest:
      type: object
      description: Request serializer for tokenize endpoint.
      properties:
        model:
          type: string
          description: >-
            Model to use for tokenization, must exist and be configured from the
            admin
        prompt:
          type: string
          description: The text to tokenize
        messages:
          type: array
          items:
            type: object
            additionalProperties: {}
          description: List of messages to tokenize (alternative to prompt)
      required:
        - model
    TokenizeResponse:
      type: object
      description: Response serializer for tokenize endpoint results.
      properties:
        total_tokens:
          type: integer
          description: Total number of tokens in the input text
        request_model:
          type: string
          description: The requested model for tokenization
        model_used:
          type: string
          description: The model used for tokenization
        tokenizer_type:
          type: string
          description: The type of tokenizer used
        id:
          type: string
          description: The id of the response
        model:
          type: string
          description: The model used for tokenization
      required:
        - id
        - model
        - model_used
        - request_model
        - tokenizer_type
        - total_tokens
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: >-
        Bearer authentication header of the form `Bearer <token>`, where
        `<token>` is your auth token.

````