Tokenize endpoint

Break down text into individual tokens to understand how a model will process your input. This is useful for debugging, token counting, and understanding model behavior.

Embedding modelsIt is currently not possible to use the tokenize endpoint on embedding models. Only Large Language Models are supported.

Prerequisites

A Paradigm API key: if you do not have one, go to your Paradigm profile and generate a new API key.
The desired LLM available in Paradigm: If you want to use a new model, you must add it to Paradigm from the admin interface.

Usage methods

There are several ways to call the endpoint:

With the python requests package (recommended)
Through a curl request: for quick testing or first-time use

Python `requests` package

You can directly send request to the API endpoint through the requests package.

import requests
import os

# Get API key and base URL from environment
api_key = os.getenv("PARADIGM_API_KEY")
base_url = os.getenv("PARADIGM_BASE_URL", "https://paradigm.lighton.ai/api/v2")

response = requests.post(
    url=f"{base_url}/tokenize",
    headers={
        'accept': "application/json",
        'Authorization': f"Bearer {api_key}"
    },
    json={
        "model": "alfred-4",
        "prompt": "This a test string"
    }
)

print(response.json())

You would then get a JSON answer as a dictionary:

{
    "id": "8c0d73b9-b18a-4893-a38e-a4338a7d4e0e",
    "tokens": [
        {"This": 1182},
        {"Ġa": 241},
        {"Ġtest": 1318},
        {"Ġstring": 3821}
    ],
    "text": "This a test string",
    "n_tokens": 4,
    "model": "alfred-4"
}

cURL request

If you prefer sending a request to Paradigm with a simple cURL command, here is an example:

curl --request POST \
  --url $PARADIGM_BASE_URL/tokenize \
  --header "Authorization: Bearer $PARADIGM_API_KEY" \
  --header 'content-type: application/json' \
  --data '{
  "model": "alfred-4",
  "prompt": "This a test string"
}'

API Fundamentals

Agents & Threads

Chat & AI Models

LightOn models

Document Management

Prerequisites

Usage methods

Python `requests` package

cURL request

​Prerequisites

​Usage methods

​Python requests package

​cURL request

Prerequisites

Usage methods

Python `requests` package

cURL request