Token Cost Control for AI Products: Caching, Routing, and Measurement

AI products have a problem that traditional software does not: the more users use them, the more money you lose. Every call consumes tokens, and usage quietly creeps up. Add a chain-of-thought step, retrieve a few more pieces of context, or let the model take one more pass, and the bill rises. Many teams discover only after launch that token costs have consumed their gross margin.

Cost control is not something to tackle only after expenses become excessive. It should be built into the architecture from the start. Here are five levers, ranked by ROI.

1. Prompt Caching: The Biggest Savings with the Smallest Change

Today's leading models, including the Claude, Gemini, and GPT families, support prompt caching. Long, stable prefixes such as system prompts, lengthy documents, and few-shot examples are cached, and that portion is billed at a steep discount when subsequent requests hit the cache, as low as one-tenth of the standard price with Claude.

Requirements:

• The prefix must be stable: the first N tokens of every request must be identical.
• Volume must be high enough: caches have a limited lifetime, ranging from minutes to hours, so frequent hits are necessary for meaningful savings.

Implementation costs almost nothing: place the system prompt and invariant context first, and the model API automatically identifies the cacheable prefix. For customer support and assistant applications built around a "fixed system prompt + multi-turn conversations," this technique often cuts input token costs by more than 50% immediately.

This should be the top cost-reduction priority because the change is small and the payoff is substantial.

2. Model Routing: Use the Right Model, Not the Most Powerful One

A common mistake is to use the most expensive large model for every request. Most requests are actually simple tasks such as format conversion, classification, routing decisions, and straightforward Q&A. A small or fast model is entirely sufficient for these tasks while being ten times cheaper and ten times faster.

A typical tiered routing architecture looks like this:

Request arrives
  → Routing layer (small model/rules assess complexity)
      ├ Simple → Haiku/Mini/Flash (cheap and fast)
      └ Complex → Opus/Pro (expensive but powerful)

Ways to assess complexity:

• Rules based on length, keywords, or whether the request contains code.
• Let a small model inspect the request first, then escalate anything it cannot classify.

A system with mixed routing often costs only one-third as much as one that always uses a large model, with almost no difference in the user experience. The key is not to make a small model handle complex requests beyond its capabilities. Incorrect routing sacrifices quality, so routing accuracy must be monitored.

3. Prompt Trimming: Save Tokens to Save Money

Every token costs money. Overloading a prompt makes it more expensive and dilutes the model's attention. Several areas are worth trimming:

• Remove redundancy from system prompts. Many system prompts contain thousands of words of generic prose such as "You are a helpful assistant..." Reduce them to the essential rules to save tokens and improve focus.
• Trim the context. Did RAG retrieve 20 items? After reranking, provide only the four most relevant ones. Use summaries and a sliding window for conversation history instead of including everything.
• Limit few-shot examples. Three strong examples are enough; do not provide 20.
• Do not express structured data in natural language. Data that takes 1,000 characters to describe in prose might require only 200 characters in JSON.

The basic arithmetic is compelling: reducing a prompt from 4,000 tokens to 1,500 saves more than half the cost per call. Multiplied by daily call volume, that becomes a significant amount.

4. Batch Processing (Batch API)

If you have many non-real-time tasks, such as overnight content generation, bulk classification, or document processing, use each provider's Batch API. Accumulate requests into a batch and submit them together. Results arrive within a few hours, and the price is typically cut in half. Both Anthropic and OpenAI offer batch APIs with a 50% discount.

The tradeoff is latency, ranging from minutes to hours, so this approach is suitable only when immediate results are unnecessary. For tasks that can be batched, batching is a reliable way to save 50%.

5. Result Caching: Do Not Compute the Same Answer Twice

Many requests repeat the same search terms, document summaries, or FAQ questions. Add a result cache in front of the model:

• Semantic caching: go beyond exact matches. Embed the query and directly return an answer to a similar previous question when it meets the similarity threshold, using Redis and a vector similarity threshold.
• Exact caching: cache by a business key. Repeated requests for the same document through summarize(doc_id) return the cached result directly.

Use caching carefully. Do not indiscriminately cache time-sensitive or highly personalized requests, or you may return stale or incorrect results. For high-frequency, stable requests such as FAQs and summaries of popular content, however, costs approach zero as the cache hit rate increases.

6. Monitoring: Attribute Costs by Feature and User

You cannot optimize costs you cannot measure. Break token consumption down across several dimensions:

• By feature: which feature is consuming the most money? If one feature accounts for an unusually large share, the design is probably at fault, perhaps because it performs too many passes or uses excessively long context.
• By user: identify anomalous users, including abuse or a single user accounting for 50% of costs.
• By call-chain stage: determine how much is spent on query rewriting, the main answer, and self-checking.

Configure alerts for users exceeding daily consumption thresholds, sudden cost increases in a feature, and anomalies in overall tokens-per-request metrics. The cost curve of an LLM application is not as flat as that of traditional software. A single bug can burn through a substantial amount of money in a few hours. Operating without alerts means flying blind.

7. A Baseline for Combining the Techniques

A well-controlled AI application's cost architecture looks roughly like this:

• Send stable prefixes through prompt caching to reduce input costs.
• Route requests through model routing, assigning simple work to small models to reduce unit prices.
• Continuously trim prompts and provide context only as needed to reduce input costs.
• Send non-real-time tasks through batch processing to save 50%.
• Send high-frequency requests through result caching to bring costs close to zero.
• Apply dimensional monitoring and alerts throughout the system to prevent costs from spiraling out of control.

Combined, these five techniques can often reduce total costs to one-quarter or one-third of an unoptimized implementation, with almost no loss in user experience.

Summary

Token costs are not something to cut only after the bill arrives. They are part of the architecture. Prompt caching, model routing, prompt trimming, batch processing, and result caching are five levers to implement one by one in ROI order, backed by monitoring that breaks costs down across dimensions.

Remember this: the sustainability of an AI product depends on whether tokens are treated as a real cost during design. AI products that maintain healthy gross margins after launch do not necessarily have cheaper models. They treat cost control as a first-class concern from day one.