Token Optimization Guide: How to Reduce Cost and latency in LLM Prompts

Every word you send to a large language model costs money. Every word the model generates costs money. Every word in between — the system prompt, the retrieved context, the conversation history, the boilerplate instructions — costs money. And in production AI systems running thousands or millions of requests per day, those fractions of a cent add up faster than most people expect. Token optimization is the discipline of minimizing the number of tokens your prompts consume without sacrificing output quality, and it is one of the most high-leverage skills a prompt engineer or AI application developer can develop.

This guide covers everything from the fundamentals of how tokenization works to advanced strategies for compressing context, structuring system prompts, caching responses, and choosing the right model for the right task. Whether you're running a customer-facing chatbot, an internal RAG pipeline, or a batch processing system that analyzes thousands of documents, the techniques in this article will help you cut costs, reduce latency, and squeeze more value out of every token you spend.

What Are Tokens and Why Do They Matter?

A token is the basic unit of text that a large language model processes. It is not the same as a word — depending on the tokenizer the model uses, a token might represent a whole word, a part of a word, or even a single character. As a rough rule of thumb, 100 tokens correspond to approximately 75 words in English. A short paragraph might be 100 tokens; a page of text might be 500.

Tokens matter for three critical reasons:

• Cost: Most LLM API providers charge per 1,000 or 1,000,000 tokens. Input tokens and output tokens are usually priced separately, with output tokens typically costing more. If your prompt includes 2,000 tokens of context that could have been 800, you are paying for 1,200 wasted tokens on every single request.
• Latency: The more tokens a model has to process — both on input and output — the longer it takes to return a response. For real-time applications like chatbots or agents, latency directly impacts user experience. Output latency is especially painful because most models generate tokens autoregressively, one at a time.
• Context window limits: Every model has a maximum context window — the total number of tokens it can process in a single request. If your prompt exceeds the limit, you have to truncate, summarize, or split your data. Wasteful token usage means less room for the data that actually matters.

Understanding tokenization is the foundation of all optimization. Different models use different tokenizers: OpenAI's GPT models use a tokenizer called tiktoken (BPE-based), Anthropic's Claude uses its own tokenizer, and Google's Gemini uses yet another. The same sentence can tokenize differently — and cost a different number of tokens — across models. When optimizing for cost across multiple providers, you need to be aware of these differences.

How Tokenizers Work: A Practical Overview

Most modern LLMs use a subword tokenization approach called Byte Pair Encoding (BPE) or a close variant. BPE works by starting with individual characters and iteratively merging the most common pairs into larger units. The result is a vocabulary where common words and common word fragments are single tokens, while rare words are broken into multiple tokens.

Here is why this matters for optimization:

• Common English words like "the", "and", "is", and "prompt" are typically single tokens.
• Less common words, technical jargon, or proper nouns may be split into 2-5 tokens. For example, "tokenization" might be 3-4 tokens depending on the tokenizer.
• Numbers, code, and special characters often tokenize inefficiently. A long JSON document with lots of punctuation can consume significantly more tokens than the equivalent information in plain text.
• Whitespace matters. Some tokenizers treat leading spaces as part of the preceding token; others treat them separately. Consistent formatting can add up across a large corpus.

You can test this yourself using tools like OpenAI's tiktoken Python library or Anthropic's tokenizer playground. Paste in your prompt, look at the token count, and then experiment with rewrites to see how different phrasings compare. You will quickly develop intuition for which patterns are token-efficient and which are wasteful.

Strategy 1: Compress Your System Prompt

The system prompt is the instruction block sent with every API call. In production applications, system prompts often balloon to thousands of tokens as developers add example interactions, formatting rules, guardrails, and behavioral instructions. Because the system prompt is sent with every single request, it is often the single largest source of token waste.

Audit Your System Prompt

Start by measuring your current system prompt token count. If it is over 500 tokens, there is almost certainly room to optimize. If it is over 1,000 tokens, you are likely paying a premium on every request. Here's how to audit and compress:

Original (214 tokens):
"You are a helpful customer support assistant for a SaaS company called Acme. 
You should always greet the customer politely and professionally. You should 
never make up information that you don't know. If you don't know the answer, 
you should say so and offer to escalate to a human agent. You should always 
use the customer's name if they have provided it. You should keep your 
responses concise and avoid unnecessary repetition. You should format your 
responses with clear paragraphs and use bullet points where appropriate."

Optimized (96 tokens):
"Acme SaaS support assistant. Be concise, polite, use bullet points. 
If unsure, say so and offer human escalation. Use customer's name when 
provided. No fabricated information."

The optimized version conveys every single instruction from the original but uses less than half the tokens. The key techniques are:

• Remove filler words and pleasantries: "You should always" and "you should never" can almost always be trimmed.
• Use imperative verbs: "Greet the customer politely" → just "Be polite."
• Combine related instructions: Merge redundant guidance into single clauses.
• Abbreviate acceptable: Minor grammar shortcuts in system prompts do not meaningfully change model behavior.

Models interpret compressed system prompts just as well as verbose ones — often better, because there is less room for contradictory instructions. Test your optimized prompt against a set of representative inputs and compare the output quality. In most cases, it will be identical or negligibly different.

Move Conditional Instructions Out of the System Prompt

If your system prompt includes instructions that only apply sometimes (e.g., "If the customer asks about billing, do X; if they ask about technical issues, do Y"), consider using a routing or classification step instead. A small, fast model can classify the user's intent into a category, and then you append only the relevant instructions to the prompt. This keeps the base system prompt lean and adds context only when needed.

Strategy 2: Optimize Context for RAG Pipelines

Retrieval-augmented generation (RAG) pipelines are one of the most common production LLM architectures, and they are also one of the most token-hungry. A typical RAG prompt includes the system instruction, the user query, and several retrieved document chunks. If each chunk is 500 tokens and you retrieve 10 chunks, that's 5,000 tokens of context before the model has even started answering.

Right-Size Your Chunks

Chunk size is the most impactful RAG parameter for token efficiency. Common mistakes include:

• Chunks too large: If your chunks are 1,000+ tokens, much of each chunk may be irrelevant to the query. The model has to process all of it, wasting tokens and potentially getting distracted.
• Chunks too small: If your chunks are 50 tokens, you lose semantic context, and you may need to retrieve more chunks to get the same information.
• No overlap: Without overlap between chunks, information that spans a chunk boundary gets lost.

A good starting point is 200–400 tokens per chunk with 50–100 tokens of overlap. Measure recall (did the retrieved chunks contain the answer?) and token usage together. The goal is the smallest total context that still gives the model what it needs to answer correctly.

Limit Retrieval Count Aggressively

More retrieved chunks does not always mean better answers. Research consistently shows that models perform best when relevant information appears early in the context and when total context length is moderate. Retrieving 15 chunks when 5 well-chosen chunks contain the answer wastes tokens and can actually decrease answer quality by diluting the signal.

Start with retrieving 3–5 chunks, evaluate answer quality, and only increase if recall is poor. If you need more, consider re-ranking: retrieve 20 chunks, use a fast embedding model or cross-encoder to re-rank them, and send only the top 3–5 to the LLM. The re-ranking step is cheap compared to the LLM call and dramatically improves both quality and token efficiency.

Compass Retrieval: Summarize Before Sending

For large documents, an effective pattern is a two-stage approach:

1. Stage 1 (cheap model): Retrieve chunks and ask a small, fast model to extract only the sentences or facts relevant to the query.
2. Stage 2 (capable model): Send only the extracted facts to the more capable model to generate the final answer.

This pattern uses more calls but fewer total tokens on the expensive model, which is often a net cost win.

Strategy 3: Control Output Token Length

Output tokens are typically the most expensive part of an API call and the biggest contributor to latency. A model that writes 800 tokens when 200 would suffice is costing you 4x on output and making users wait 4x longer.

Set Explicit Length Limits

Models respect explicit output length instructions when they are specific. "Be concise" is vague. "Respond in no more than 3 sentences" or "Limit your response to 150 words" is actionable and effective. Include length constraints in your system prompt or user instructions:

Before: "Explain the concept of depreciation in accounting."
After: "Explain depreciation in accounting in 2-3 paragraphs, max 150 words."

Use Structured Output Formats

Asking for structured output (JSON, tables, bullet lists) naturally constrains verbosity compared to free-form prose. A model asked to "list the top 5 risk factors as a JSON array with keys 'rank', 'factor', and 'description'" will produce more concise, predictable output than one asked to "discuss the risk factors."

"Return your analysis as a JSON object with the following structure:
{
  "summary": "one sentence",
  "key_points": ["point 1", "point 2", "point 3"],
  "recommendation": "one sentence"
}
Do not include any text outside the JSON object."

This format typically yields 60–150 output tokens regardless of the complexity of the input, compared to 400+ tokens for an unstructured prose response covering the same ground.

Use max_tokens as a Hard Cap

Most APIs allow you to set a max_tokens parameter that hard-caps the output length. Use this as a safety net — not your primary optimization lever (a truncated response is a poor user experience), but as protection against runaway outputs. A well-optimized prompt with a max_tokens cap is the belt-and-suspenders approach to output control.

Strategy 4: Trim Conversation History

In multi-turn chat applications, conversation history accumulates rapidly. By turn 10 of a conversation, the prompt might include thousands of tokens of prior exchanges — much of it no longer relevant to the current turn. Without management, token costs grow linearly with conversation length, and eventually the context window fills up entirely.

Techniques for History Management

• Sliding window: Keep only the most recent N turns. Simple and effective for most use cases. Start with N=4–6 turns.
• Summarize older turns: When a conversation exceeds a threshold, use a cheap model to summarize the older turns into a compact paragraph (100–200 tokens) and replace the raw history with the summary. This preserves context while dramatically reducing tokens.
• Selective retention: After each turn, classify which prior turns are still relevant and discard the rest. A simple classification prompt — "Which of the following prior messages contain information still relevant to the current question?" — can identify what to keep.
• Tool result pruning: If your agent calls tools (e.g., a search API, a database query), the raw tool outputs are often large. After the model has used them to answer, replace the raw output with a short summary ("Search returned 4 results about topic X") in the stored history.

Example: Summarization-Based History Compression

System prompt for the summarizer:
"You are a conversation compressor. Given the following conversation 
excerpt, produce a factual summary in under 150 tokens that preserves 
all key information, decisions made, and unresolved questions. 
Do not add any commentary."

User: [Paste the older turns to be compressed]

Store the summary in place of those turns in the conversation history.

This technique can compress a 2,000-token conversation history into 200 tokens while retaining the information the model needs to stay contextually aware.

Strategy 5: Choose the Right Model for Each Task

Token optimization is not just about reducing token count — it is also about not spending premium tokens on tasks that do not require a premium model. A common mistake in production systems is using a large, expensive model (GPT-4o, Claude Opus, Gemini Ultra) for every request, including simple ones that a smaller model could handle.

Model Tiering

Implement a model tiering strategy:

• Small/fast model (e.g., GPT-4o-mini, Claude Haiku, Gemini Flash): Handle classification, routing, simple extraction, formatting, and history summarization. These tasks cost fractions of a cent and complete in under a second.
• Mid-tier model (e.g., GPT-4o, Claude Sonnet): Handle most user-facing tasks, standard RAG queries, and moderate-complexity reasoning.
• Premium model (e.g., GPT-4o with advanced reasoning, Claude Opus): Reserve for complex reasoning, multi-step analysis, and tasks where output quality justifies the cost.

A routing prompt can classify incoming requests and send them to the appropriate tier:

"Classify the following user request into one of three complexity tiers:
- 'simple': basic factual questions, formatting, extraction, simple lookups
- 'moderate': multi-step reasoning, summarization, analysis, standard RAG
- 'complex': advanced reasoning, multi-constraint tasks, nuanced writing

User request: [request]
Output: a single word — simple, moderate, or complex"

This classification call costs almost nothing (a few hundred input tokens, a single output token) but can reduce your overall spend by 50–80% by routing most traffic to cheaper models.

Strategy 6: Leverage Prompt Caching

Some API providers now offer prompt caching — if you send the same prefix (typically the system prompt and any static context) across multiple requests, the provider caches the tokenized representation and charges less for those tokens on subsequent calls. This is particularly valuable for:

• Long system prompts sent with every request
• RAG systems where a large knowledge base is included in every prompt
• Agent frameworks where a long tool-definition block is sent with every call

To benefit from prompt caching, structure your prompts so that the static portion comes first (the system prompt, the tool definitions, the background context) and the variable portion comes last (the current user message). The cached prefix must be identical across requests — even a small change invalidates the cache.

Strategy 7: Batch Processing Optimization

For batch jobs that process many documents or requests, additional optimizations apply:

• Batch API endpoints: Providers like OpenAI and Anthropic offer batch APIs that accept large batches of requests and process them asynchronously, typically at a 50% discount. If your use case does not require real-time responses, this is one of the most effective cost reductions available.
• Parallel processing: For independent requests, use async/concurrent processing to maximize throughput and minimize total wall-clock time.
• Pre-filtering: Use a cheap model to filter out requests that do not need LLM processing at all. For example, in a document classification pipeline, a rules-based filter or small model can handle 60% of documents, leaving only the ambiguous 40% for the more expensive model.

Measuring and Monitoring Token Usage

You cannot optimize what you do not measure. Every production LLM system should have instrumentation that tracks:

• Input tokens per request: Both cached and uncached if caching is used.
• Output tokens per request: The largest lever for latency and cost.
• Cost per request: Calculated from token counts and the model's pricing.
• Cost per meaningful unit: For example, cost per successfully answered question, cost per document processed, cost per ticket resolved. This is a better KPI than raw cost.
• Token efficiency ratio: Total output tokens divided by total input tokens. A high ratio means the model is spending most of its budget generating useful output; a low ratio suggests excessive context.

Log these metrics, track them over time, and set alerting thresholds. A sudden spike in average input tokens often indicates a prompt that was changed and accidentally bloated, or a conversation history management bug that stopped compressing.

Common Token Waste Patterns to Eliminate

Here is a checklist of the most common sources of token waste in production systems:

1. Verbose system prompts that repeat instructions or include unnecessary pleasantries
2. Over-retrieved context in RAG systems — sending 10 chunks when 3 would do
3. Unbounded output with no length constraints, leading to rambling responses
4. Full conversation history stored and sent with every turn, never compressed
5. Redundant examples in few-shot prompts where 1-2 examples would suffice
6. Sending raw tool outputs in conversation history long after they have been used
7. JSON or XML formatting overhead where simpler delimiters would work
8. Using a premium model for simple routing or classification tasks
9. No prompt caching despite a large static prompt prefix
10. Chatty interjections in system prompts ("Great question!", "Let me think about this") that bleed into output

Audit your system against this list, and you will almost certainly find low-hanging fruit.

Putting It All Together: A Token Optimization Workflow

Here is a repeatable workflow you can apply to any LLM feature or application:

1. Measure the current token usage per request (input and output) and the cost per request.
2. Audit the system prompt for redundancy and verbosity. Compress it to the minimum that preserves behavior.
3. Review context (RAG chunks, conversation history, tool outputs). Right-size and prune aggressively.
4. Add output constraints — length limits, structured formats, and max_tokens caps.
5. Tier the model — route simple requests to cheaper models.
6. Enable caching for static prompt prefixes.
7. Re-measure and compare cost and quality. Iterate on any change that degrades quality.
8. Instrument and monitor ongoing token usage so regressions are caught early.

Token optimization is not a one-time project — it is an ongoing discipline. As prompts evolve, new features are added, and models are updated, token usage will creep upward. Make optimization a regular part of your prompt engineering review cycle.

Getting Started with PromptWright

Managing token-optimized prompts across a team, tracking their performance, and iterating on improvements is exactly what PromptWright is built for. With version-controlled prompt libraries, built-in testing and evaluation, and cost tracking integrations, PromptWright gives you the tools to keep your AI applications fast, affordable, and high-quality as they scale.

Ready to take control of your token spend? Sign up for PromptWright and start building an optimized, cost-efficient prompt library today.