Tokenizer API

One endpoint that counts tokens and prices them across the model catalogue. Counting runs against local byte-pair encoding tables, so your text is never sent to a model provider.

The API is a Pro feature. See plans.

Authentication

Send your Pro license key as a bearer token. An x-license-key header is accepted as an alternative.

Authorization: Bearer YOUR_LICENSE_KEY

Validity is checked against Creem and cached for a few minutes, so a cancelled or refunded subscription stops working shortly after.

Endpoint

POST https://mytokenhub.site/api/v1/tokenize

Request

Send either text for a single input, or texts for a batch of up to 200.

text string: The text to tokenize. Up to 2,000,000 characters.
texts string[]: A batch of inputs. Mutually exclusive with text.
modelIds string[]optional: Models to price against. Defaults to the three free-tier models. Also accepted as model_ids or models.
outputTokens numberoptional: Expected completion length, so output pricing is included. Defaults to 0.
cacheHitRate numberoptional: Share of input tokens you expect to hit the provider's prompt cache, between 0 and 1. Ignored for models with no published cached rate.
priceOverrides objectoptional: Your own rates in USD per million tokens, keyed by model id, for example {"gpt-4o":{"input":1.5,"output":6}}. An override drops that model's cached-input discount.
includePieces booleanoptional: Return the individual tokens of the first input, capped at 400.

Example

curl -X POST https://mytokenhub.site/api/v1/tokenize \
  -H "Authorization: Bearer YOUR_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Summarise the following incident report in three bullets.",
    "modelIds": ["gpt-5", "claude-sonnet-4.5", "gemini-2.5-pro"],
    "outputTokens": 300
  }'

Response

{
  "tier": "pro",
  "items": [
    {
      "index": 0,
      "chars": 57,
      "results": [
        {
          "modelId": "gpt-5",
          "label": "GPT-5",
          "provider": "OpenAI",
          "inputTokens": 11,
          "outputTokens": 300,
          "exact": true,
          "contextWindow": 400000,
          "exceedsContext": false,
          "cost": {
            "inputCost": 0.00001375,
            "cachedInputCost": 0,
            "outputCost": 0.003,
            "totalCost": 0.00301375
          }
        }
      ]
    }
  ],
  "totals": [
    {
      "modelId": "gpt-5",
      "label": "GPT-5",
      "inputTokens": 11,
      "outputTokens": 300,
      "totalCost": 0.00301375,
      "exact": true
    }
  ]
}

totals sums each model across every item, which is what you want for a batch. exact is false when the count was scaled from o200k_base rather than measured with the vendor's own tokenizer -- true only for OpenAI models.

Rate limits

60 requests per minute per license key. Responses carry X-RateLimit-Limit and X-RateLimit-Remaining; a 429 also carries Retry-After in seconds.

Errors

Errors are JSON with an error string.

400 Bad Request: Malformed body, or an unknown model id.
401 Unauthorized: Missing or invalid license key.
403 Forbidden: The license is inactive, expired or disabled.
413 Payload Too Large: An input exceeded the character limit.
429 Too Many Requests: Rate limit exceeded.
503 Service Unavailable: License validation is temporarily unavailable. Retry.

Model ids

All 44 ids, grouped by provider. Ids marked ~ return estimated counts.

OpenAI

gpt-5 gpt-5-mini gpt-5-nano gpt-4.1 gpt-4.1-mini gpt-4.1-nano gpt-4o gpt-4o-mini o3 o3-mini o4-mini gpt-4-turbo gpt-4 gpt-3.5-turbo gpt-oss-120b gpt-oss-20b

Anthropic

claude-opus-4.1~ claude-opus-4~ claude-sonnet-4.5~ claude-sonnet-4~ claude-3.7-sonnet~ claude-haiku-4.5~ claude-3.5-haiku~ claude-3-opus~

Google

gemini-2.5-pro~ gemini-2.5-flash~ gemini-2.5-flash-lite~ gemini-2.0-flash~

DeepSeek

deepseek-v3~ deepseek-r1~

xAI

grok-4~ grok-3~ grok-3-mini~

Support

Questions about the API go to support@mytokenhub.site.