Tokenizer: Understand Tokens to Truly Control LLM Costs

Nine out of ten developers working with LLMs cannot say how many tokens their last prompt consumed. Everyone knows that billing is based on tokens, but their understanding of a token often stops at "it is roughly a character." As a result, they estimate costs by guesswork, truncate context by character count, and fill prompts to the brim while thinking they are still short. The consequences are uncontrolled bills, overflowing context windows, and Chinese applications that inherently cost more than English ones without anyone understanding why.

A token is neither a character nor a word. Without understanding this smallest unit, cost optimization and context engineering are built on sand. This article makes tokenizers practical: how they work, differences in language efficiency, measurements with tiktoken, and techniques that actually save money.

1. A token is neither a character nor a word, but a subword unit

Humans read text as words or characters, while models recognize only numbers. The bridge between them is the tokenizer: it splits text into discrete fragments, looks up each fragment in a vocabulary to map it to an integer ID, and passes those IDs to the model. Each fragment is a token.

The key is that token granularity lies between characters and words. These are called subwords:

• A common whole word may be one token: "hello", "world", or "the".
• Rare or compound words are split apart: "unbelievable" → "un" + "believ" + "able", and "tiktoken" → "tik" + "token" because it is too new to appear as a complete word in the training corpus.
• Rare characters and emoji often require multiple tokens to assemble a single character.

One simple but overlooked fact is that LLMs charge by token and measure context windows in tokens. The number of input tokens determines both price and context usage, and the same applies to output tokens, which are usually more expensive. Tokens are therefore the "currency" of the LLM era: every line in your prompt costs money.

2. How tokenizers work: how BPE segments text

Mainstream LLMs use BPE (Byte Pair Encoding). It does not segment text according to predefined rules; it learns segmentation from data:

1. The initial vocabulary consists of 256 bytes, and text is first split into a byte sequence.
2. The algorithm finds the most frequently adjacent byte pair in the corpus, merges it into a new token, and adds that token to the vocabulary.
3. It repeats the previous step until the vocabulary reaches its target size, from tens of thousands to hundreds of thousands of tokens.

The result is that the more common a fragment is, the more likely it is to become a standalone token. High-frequency strings such as "the", "tion", and "http" become single tokens, while rare combinations remain split into smaller fragments. This is why common English words often occupy one token while uncommon words require several.

However, each model uses a different tokenizer, so token counts cannot be converted across models:

The same Chinese passage might use 100 tokens with cl100k_base but only 70 with o200k_base. There is no single answer to "How many tokens does this prompt contain?" You must tie the count to a specific model.

3. Language-efficiency differences: why Chinese is inherently more expensive

BPE learns merges from corpus frequency, and English dominates the training corpora of mainstream models. This creates an unfriendly reality for Chinese applications: expressing the same meaning in Chinese uses more tokens than in English.

Typical figures at the scale of cl100k_base:

• English: most common words occupy one token, averaging about 1 word ≈ 1.3 tokens, so 1,000 English words are about 1,300 tokens.
• Chinese: one Chinese character often occupies 1–2 tokens. Common characters such as 你、好、的、是 often use one token, but many Chinese characters require two or more. An equivalent Chinese expression often uses 1.5–2 times as many tokens as its English counterpart.
• Code: whitespace and indentation also consume tokens. Every indentation level, line break, and bracket counts. In a neatly formatted code sample, structural symbols and indentation can consume a substantial share.

This is not the model "discriminating" against Chinese. It follows from the corpus distribution used by BPE: English word combinations are merged extensively, while Chinese character combinations are not. For a Chinese AI product, the same budget buys a shorter context window, and a prompt of equivalent length costs more. Product pricing and design must account for this premise.

One evolution worth remembering is that o200k_base for GPT-4o was specifically optimized for multilingual text. Its Chinese efficiency is significantly better than cl100k_base, reducing token counts for the same Chinese passage by around 30%. Choosing a model means choosing not only its capabilities, but also its token efficiency.

4. Measuring it: count with tiktoken

Do not rely on intuition; measure it. OpenAI's open-source tiktoken library counts tokens exactly and offline, using only local computation with no API call or cost. The TypeScript ecosystem offers a WASM port under the same name:

import { encoding_for_model } from "tiktoken";

const enc = encoding_for_model("gpt-4o"); // Corresponds to o200k_base

const count = (text: string) => enc.encode(text).length;

// Also inspect how it tokenizes the text
const ids = enc.encode("tiktoken is great");
console.log(ids);                 // [82, 427, 2301, 374, 2294]  -- something like "tik" "token" " is" " great"
console.log(ids.length);          // 5

enc.free(); // Release WASM memory after use; long-lived processes must remember this

Here is a set of measurements using o200k_base:

Across the two tables, the row to watch most closely is the code row, where the character-to-token ratio approaches 2:1. You may see only a few short lines of code, but the tokenizer sees a screenful of indentation and symbols. When you feed an entire source file to the model as context, this ratio directly consumes your context-window budget.

5. Why this directly affects cost and quality

Token count is not only about money. It is connected to three other factors:

1. Long input = expensive. Input is billed by token, so a long prompt directly increases the cost of every call.
2. Long input = slow. Time to first token (TTFT) correlates with input length. The longer the prompt, the longer the user waits.
3. Output costs more. Output-token prices are usually three to five times input-token prices. It costs much more for the model to "ramble" than to "read the question."

For Chinese applications, all three effects are amplified by the efficiency disadvantage: the same feature naturally costs more and runs more slowly with a Chinese prompt than with an English one. This is also why shortening prompts has an especially high ROI in Chinese use cases: you are compressing the portion of the cost amplified by BPE.

6. Practical cost control: five levers that work immediately

Once you understand tokens, the direction of optimization becomes clear:

1. Trim prompts and remove filler. Delete phrases such as "You are a helpful, knowledgeable assistant skilled at..." and keep only executable rules. The system prompt is a fixed tax paid on every call, so it is worth optimizing repeatedly.
2. Limit few-shot examples. Three carefully chosen examples are often enough; do not pile up twenty. Every example adds input tokens and dilutes the model's attention.
3. Choose efficient representations. For internal data processing, log analysis, and similar scenarios that do not need a human touch, English prompts are cheaper. Use JSON or tables for structured data instead of describing it in natural language.
4. Enable prompt caching. When a stable system prompt and long document prefix hit the cache, input cost can fall to one-tenth with Anthropic or be cut in half with OpenAI. The rule is: put stable content first and frequently changing content last.
5. Use tokenizer-aware truncation. Trim context by token count, not character count. Otherwise, truncating Chinese in the middle of a token can create garbled fragments that waste tokens and may trigger abnormal model output.

7. Engineering rule: never estimate tokens by character count

The final point is a strict engineering rule: Any logic involving token limits must calculate them exactly with a tokenizer, never estimate them from character count.

• "I truncate at 8,000 characters" is wrong. For the same 8,000 characters, English might use 2,000 tokens while Chinese might use more than 6,000, a twofold or threefold difference. You either waste context-window capacity by truncating too early or exceed the limit and get an error.
• Character-based truncation can also cut through the middle of a token, producing an incomplete fragment. The correct approach is to count with a tokenizer until you reach the budget, then decode at a valid token boundary.
• tiktoken is entirely local, millisecond-fast, and free. Before adding text to the context or sending a prompt, run encode to get an exact count. This prevents limit errors and enables accurate billing.

Replacing the intuition of "count characters" with "count tokens" is one of the most basic and valuable mindset shifts in LLM engineering.

Summary

• Tokens are subword units, not characters or words. They are both the billing unit and the currency of context windows.
• BPE learns segmentation from frequency. Because its corpora are dominated by English, Chinese has inherently lower token efficiency and equivalent content costs 1.5–2 times as much.
• Each model has a different tokenizer. Token counts cannot be converted across models, so optimization must target a specific encoding.
• For code, token count can approach half the character count because of structural symbols and indentation. Be especially cautious when supplying source code as context.
• Whenever a token limit is involved, calculate it exactly with a tokenizer. Estimating by character count means flying blind.

The key line: Estimating tokens by character count is like weighing something with a ruler. The unit is wrong, so even the most meticulous optimization is futile. Real cost control begins with understanding how large one token actually is.