> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lighton.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenize endpoint

> The tokenize API endpoint allows you to easily convert some text into a list of tokens for a specific model.

Break down text into individual tokens to understand how a model will process your input. This is useful for debugging, token counting, and understanding model behavior.

<Warning>
  **Embedding models**

  It is currently not possible to use the tokenize endpoint on embedding models.
  Only Large Language Models are supported.
</Warning>

## Prerequisites

* A **Paradigm API key**: if you do not have one, go to your Paradigm profile and generate a new API key.
* The desired **LLM available in Paradigm**: If you want to use a new model, you must add it to Paradigm from the admin interface.

## Usage methods

There are several ways to call the endpoint:

1. With the **python `requests` package** (recommended)
2. Through a **curl request**: for quick testing or first-time use

## Python `requests` package

You can directly send request to the API endpoint through the `requests` package.

```python theme={null}
import requests
import os

# Get API key and base URL from environment
api_key = os.getenv("PARADIGM_API_KEY")
base_url = os.getenv("PARADIGM_BASE_URL", "https://paradigm.lighton.ai/api/v2")

response = requests.post(
    url=f"{base_url}/tokenize",
    headers={
        'accept': "application/json",
        'Authorization': f"Bearer {api_key}"
    },
    json={
        "model": "alfred-4",
        "prompt": "This a test string"
    }
)

print(response.json())
```

You would then get a JSON answer as a dictionary:

```json theme={null}
{
    "id": "8c0d73b9-b18a-4893-a38e-a4338a7d4e0e",
    "tokens": [
        {"This": 1182},
        {"Ġa": 241},
        {"Ġtest": 1318},
        {"Ġstring": 3821}
    ],
    "text": "This a test string",
    "n_tokens": 4,
    "model": "alfred-4"
}
```

## cURL request

If you prefer sending a request to Paradigm with a simple cURL command, here is an example:

```shell theme={null}
curl --request POST \
  --url $PARADIGM_BASE_URL/tokenize \
  --header "Authorization: Bearer $PARADIGM_API_KEY" \
  --header 'content-type: application/json' \
  --data '{
  "model": "alfred-4",
  "prompt": "This a test string"
}'
```