For application and platform engineers, and for teams using AI coding tools.
Saving tokens is not mainly about shortening prompts. It is about making the model do less useless work. Repeated reads, irrelevant context, verbose output, failed attempts, and rework all belong in the optimization budget.
The goal is to reduce total cost per accepted task while meeting quality, safety, and latency requirements. This guide focuses on usage-priced model APIs; subscriptions and IDE plans need their own quota analysis.
How to use this guide
This is not a model-parameter manual. It is a set of cost lessons drawn from real Agent, Skill, MCP, AI coding, and API engineering work.
The most direct approach is to give this guide and your system details to an AI, then ask it to audit and improve the system using these principles. Start with the clearest waste: repeated context, oversized tool results, ineffective retries, unnecessarily expensive models, long outputs, and unstable cache layouts.
Reading map
| Problem | Go to |
|---|---|
| Measure savings and prioritize work | Task economics · Rollout |
| Inputs and history keep growing | Context · Caching |
| Agents load too many tools or spin | Tools and Skills · Execution control |
| Models and reasoning cost too much | Models and output · Other options |
1. Measure task cost, not just token totals
Key 1 | Separate tokens, money, and time. Fewer input/output tokens and calls save tokens. Cache discounts, model routing, and batch processing mainly change price. Streaming and concurrency mainly improve perceived latency. Define the target before treating “faster” as “cheaper.”
Key 2 | Include failures and retries. Total task cost includes every attempt, subagent, model review, and tool execution—not only the successful final call.
Caution | Lower cost must not hide lower quality. Track success rate, P95 latency, and critical-risk cases. If no tasks pass acceptance, cost per accepted task is undefined, not zero.
2. Remove irrelevant material, not critical evidence
Key 1 | Process data before sending it to a model. Extract article text from pages, collapse repeated logs, filter and aggregate tables, and search code before reading it. Prefer text for text documents and crop screenshots to relevant regions, but preserve sources, values, units, small print, and coordinate accuracy.
Key 2 | Assemble RAG context against an evidence budget. Filter by permissions and version, retrieve, deduplicate and rank, then fill a defined budget. Keep source locations. If evidence is insufficient, retrieve more instead of forcing an answer.
Key 3 | Persist state for long tasks instead of replaying the chat. Store raw material separately. Working context should contain goals, constraints, confirmed facts, artifact locations, open questions, and the next action. Summarize or compact at phase boundaries or near a budget limit—not automatically every turn.
Key 4 | Conversation continuation does not make history free. previous_response_id continues state, but historical input is still billed. When using server-side continuation, do not manually append the same full history again.
Caution | A summary must not be the only memory. Store permissions, constraints, and source evidence independently. A business summary and API Compaction are different mechanisms; preserve returned compaction items exactly as the API requires.
3. Cache stable, reusable prefixes—not vanity hit rates
Key 1 | Put stable content first and dynamic content last. Separate durable rules, tool definitions, and output contracts from the current request. Do not inject timestamps or random IDs into the stable prefix. Keep tool order and schemas deterministic, and check whether middleware duplicates content.
Key 2 | Count cold starts, writes, and expiry. Prompt caching reuses prefix computation, not answers, and usually does not reduce the input-token figure. Measure initial creation, later reads, expiry, and rebuilds; confirm write pricing with the provider.
Key 3 | Design tool-result and answer caches separately. The former prevents duplicate queries; the latter may avoid a whole model call. Include tenant, permission, data version, and query conditions in cache keys, with explicit invalidation. Semantic similarity is not permission to reuse data across users.
Caution | Shorter is not always cheaper. Replacing a cache-hit prefix with a summary adds summarization and cache-rebuild costs. Compare downstream total cost, not text length alone.
4. Load fewer tools and return less data
Key 1 | Keep a small common toolset available; discover large catalogs on demand. Use mechanisms such as Tool Search for infrequent tools. Expose a Skill's name and purpose first, and load its rules only after selection. Lazy loading adds discovery steps, and defer_loading is not a universal MCP client switch.
Key 2 | Trim results in the tool, not in the prompt. Search should return snippets, queries only required fields, and tests status plus failure summaries. Enforce count and size limits in code and return a truncation marker or cursor.
Key 3 | Keep intermediate data in programs. Let code filter, sort, sum, join, and convert formats. Let models interpret intent, handle exceptions, and explain results. Programmatic tool composition should also limit permissions, runtime, and response size.
Avoid: 100,000 orders → model reads and sums each row → total
Prefer: database filters and aggregates → total plus anomalies → model explains
5. Optimize model choice, reasoning, and output separately
Key 1 | Choose models with evaluations, not by defaulting to the largest. Test low-cost models for classification, extraction, and formatting; use models that meet the quality bar for complex reasoning and high-risk changes. A “small model → validation → escalation” route must beat alternatives across the whole chain, not just the first call.
Key 2 | Control reasoning effort and answer length independently. Evaluate lower effort on simple tasks and raise it for complex phases while tracking errors and rework. “Only give the conclusion” does not disable hidden reasoning, and an overly tight output cap can exhaust the reasoning budget before producing a complete answer.
Key 3 | Specify the deliverable. Say whether you need a conclusion or analysis, a complete file or diff, quoted evidence or source IDs. Structured output helps machine consumers, but business meaning still needs validation and the schema should allow missing or uncertain values.
Return only: conclusion, key evidence, and open questions.
Do not restate the request or repeat a summary. Mark insufficient evidence; do not guess.
6. Enforce agent stop conditions in code
Key 1 | Define four kinds of stop. Finish when acceptance passes; checkpoint when cost, turn, or time budgets are reached; interrupt repeated failure under unchanged conditions and no new evidence; hand off missing permissions or high-risk approvals. A prompt that says “use fewer tokens” is not enforcement.
Key 2 | Classify retries and prefer recovery. Back off network failures within limits, fix invalid parameters before retrying, and stop on permission errors. After timeouts, check whether the remote action completed. Make writes idempotent or deduplicated and resume from checkpoints instead of rerunning everything.
Key 3 | Multi-agent is a capability choice, not a savings switch. Split only independent, measurable tasks. Give subagents only the required objective, evidence, and budget; ask for results and sources rather than copying the entire parent conversation. Count every branch in total cost.
Caution | A budget stop is not task completion. Return completed work, unfinished items, and blockers; never report unrun checks as passed.
7. In AI coding, read less irrelevant code and make fewer unrelated changes
Locate first, read second, and submit a focused patch. Provide relevant modules and acceptance criteria; search symbols, call sites, and tests before opening large files. Skip dependencies and generated artifacts by default. If a patch fails, reread the relevant region instead of guessing repeatedly, and turn stable workflows into scripts.
Stop duplicate verification without removing required tests. Copy changes and permission, payment, or migration changes deserve different scopes. Stop after checks pass with no intervening change, failure, or unresolved concern; rerun when code or environment changes.
8. Choose other cost controls by workload
Consider Batch or lower-priority service for non-interactive classification and evaluation. Use Flex-like service only when longer waits and unavailable capacity are acceptable. Retry failed items only and price the degradation path; these options mainly change price and latency, not task tokens.
For frequent, stable tasks with clear acceptance criteria and strong examples, evaluate fine-tuning or distillation, including training, maintenance, and ongoing evaluation. When knowledge changes often, maintain retrieval sources first and confirm that the target model supports the required training method.
Do not trade readability for unverified compression. Single-letter fields, wholesale translation, Base64, and claims that “JSON is always cheaper” must be tested against real token counts, accuracy, and rework. Inputs saved but lost to mistakes and retries are not savings.
9. Recommended rollout
- Establish task-level cost and quality baselines.
- Trim irrelevant context and oversized tool responses.
- Remove duplicate calls, ineffective retries, and full reruns.
- Stabilize prompt prefixes and verify cache savings.
- Select model, reasoning effort, and output format through evaluations.
- Only then evaluate Batch, Flex, fine-tuning, or distillation.
Change one major variable at a time and record success rate, cost per accepted task, and P95 latency.