LLM cost control: budgets, caching, and when to stop
LLM cost control is four rules on a workflow you already run: a token ceiling for each item, a cache for the prefix that does not change, a smaller model for the easy step, and a stop when the run costs more than the labor it replaces. OpenAI discounts reused cached input by up to 90% when that prefix matches, and the discount is not a reason to leave a loop running [1]. A small operations team can enforce the four rules on the queue it has today. It does not need a new platform program to do it.
The bill grows inside the loop
Most of the spend in a live workflow is not the first successful test. It is the later cases. Each one resends standing instructions, tool definitions, and a source document, then adds one new record at the end. If the loop retries, that prefix rides along every time.
OpenAI's prompt cache stores the computed state for an unchanged prefix at the beginning of the prompt. A later request reuses that state only when the entire rendered prefix matches. If a tool name, a schema, or an earlier turn changes, the prefix after that change will not match [1]. The case-specific text still has to be processed. Caching does not make the new part of the prompt free.
Output cost also hides in tokens a person never reads. On OpenAI reasoning models, reasoning tokens are not visible in the API response, they still occupy context, and they are billed as output tokens [2]. A step that thinks harder can cost more than the short answer that follows it.
Set a per-run token budget
Set the ceiling on one item of work, not on the month. Two numbers are enough. The first is a token cap for that item. The second is a money cap equal to what it would cost a person on the team to finish the same item by hand.
On OpenAI reasoning models, the `max_output_tokens` parameter limits the total tokens the model generates, including reasoning tokens, visible output, and non-visible formatting tokens [2]. If generation hits that cap, the response can come back incomplete, with the reason set to `max_output_tokens`. That can happen before any visible output is produced, so you can pay for input and reasoning and still have nothing a clerk can use [2]. OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when you first use these models, then tightening the buffer once you know what your prompts actually consume [2].
Our team treats the ceiling as a stop in the workflow, not a sentence in the prompt. After every step, read the usage object. Add input tokens, cache-write tokens, cache-read tokens, and output tokens, including reasoning tokens when the API reports them. Price that sum from the rate card for the model you called. If the next step would cross the token ceiling or the labor-cost ceiling, do not call again. Hand the item to a person. The labor cap is the time of the person who already owns that kind of item. Write it down before the batch. A number you invent after the invoice arrives is not a control.
Cache the part that does not change
Split every prompt into a stable head and a variable tail. The head is the standing instructions, the tool list, and the policy, rate sheet, or procedure every case in the batch shares. The tail is the one record. Keep the head byte-for-byte stable. Edit the tail.
Google's Gemini API turns implicit caching on by default for Gemini 2.5 and newer models, passes on savings when a request hits, and raises the odds when large, common content sits at the beginning and similar prefixes arrive close together [4]. On that page, the minimum input for implicit caching is 4,096 tokens for the Gemini 3 Flash rows and for Gemini 3.1 Pro Preview, and 2,048 tokens for Gemini 2.5 Flash and Gemini 2.5 Pro [4]. Hits are reported on `usage.total_cached_tokens` [4]. A short instruction block under that minimum will not cache, no matter how often you repeat it.
OpenAI's minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later, it varies by request settings on earlier models, and tokens inside OpenAI's hidden system content do not count toward that minimum [1]. Caching is on by default for supported models, and reused tokens are discounted by up to 90% at the cached-input rate, with the exact price still depending on the model [1]. For GPT-5.6 and later, a cache write costs 1.25 times the uncached input rate and a later read costs 0.1 times that rate [1]. On those models, writing a prefix once and fully reusing it once costs 1.35 times an ordinary pass, versus 2 times with no cache, and one write plus nine full reads across ten requests costs 2.15 times, versus 10 times with no cache [1]. A prefix you use only once does not earn that discount.
OpenAI tells you to track `usage.input_tokens_details.cached_tokens`, `usage.input_tokens_details.cache_write_tokens`, input-token counts, and realized cost, and to price input from `response.usage` plus the model rate card [1]. Writes with no reads mean you paid to store a prefix the next case missed.
Anthropic uses the same split with its own multipliers. A 5-minute cache write is 1.25 times the base input price, a 1-hour cache write is 2 times the base input price, and a standard cache read is 0.1 times the base input price [3]. A hit at that standard rate costs 10% of the base input price, so a 5-minute cache pays off after one read and a 1-hour cache pays off after two reads [3]. Named models are cheaper to read: 0.025 times base input on Claude Fable 5.1 and Claude Mythos 5.1, and 0.05 times base input on Claude Opus 5.5 [3]. Use the row for the model you call.
Google Cloud bills the tokens that create a cache at the standard input price for both implicit and explicit caching, and implicit caching has no storage fee plus a 90% discount on cached tokens compared with standard input tokens [5]. Explicit caching adds storage based on how long the cache is kept. On Gemini 2.5 and later that explicit discount is 90%, and on Gemini 2.0 models it is 75% [5]. The default explicit lifetime on that page is 60 minutes unless you extend it [5]. If similar cases arrive far apart, batch them, or pay for the longer window and confirm the reads land before it expires.
Send the easy step to a smaller model
Do not send every row through the most capable model because it is the one you tested. Anthropic's pricing page tells builders to choose Haiku for simple tasks, Sonnet for most production workloads, and Opus for the most complex reasoning [3]. The same page lists Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens, and Claude Opus 5 at $5 per million input tokens and $25 per million output tokens [3]. That gap is the reason to split the job inside one provider. It is not a reason to switch vendors mid-queue.
On an invoice-exception queue, keep the policy and tool list at the front and one invoice at the end. The smaller model labels duplicates and missing purchase-order numbers. Only a true mismatch goes to the larger model, and only inside the per-item ceiling. If the larger model handles most of the batch, the first question is too vague. If a person redoes most of the labels, that step was too hard. Fix the step rather than adding another loop.
Stop when the run costs more than the labor
NIST released the AI Risk Management Framework on January 26, 2023, for voluntary use [6]. In the framework core, treatment of documented AI risks is prioritized based on impact, likelihood, and available resources or methods [7]. Resources required to manage AI risks are taken into account, along with viable non-AI alternative systems, approaches, or methods, to reduce the chance or the size of a bad outcome [7]. The same core calls for mechanisms, with responsibilities assigned and understood, to supersede, disengage, or deactivate an AI system whose performance or outcomes are inconsistent with intended use [7].
For an operations workflow, intended use is plain. The run should clear the item for less than a person would spend doing it. When the accumulated cost crosses that labor cost, stop. The non-AI alternative is the person who already owns the queue. NIST is writing about risk, not about a token invoice. Applying the disengage rule to a cost miss is our management choice. It is not a claim that the framework priced an API account.
A small desk in Charlotte can run this as a morning rule. The person who owns the queue asks whether the next model pass is still cheaper than finishing the item. If it is not, the loop ends and the person clears it. The workflow stays available for the next ordinary case.
Stop for any of these, and record which one fired:
- The token ceiling or the labor-cost ceiling for this item is reached.
- Another pass would repeat work a person can close in less time than the call would cost.
- The model returned an incomplete result because it hit the output cap, and a second paid attempt is not justified [2].
- The case sits outside the policy text you cached. Do not let another call invent a path. Hand the file over.
What to write down after the batch
Keep a short log: model name, input tokens, cached tokens, cache-write tokens, output tokens, reasoning tokens when the provider returns them, computed cost, the stop reason if you halted, and whether a person finished the item. Once a week, read it for three patterns. If cache reads are absent, the prefix is under the minimum, it changed between calls, or the cases arrived outside the cache window. If the larger model handled the easy rows, the first step is not earning its keep. If nothing hits the stop rule and the invoice still climbs, the ceiling is too loose. OpenAI's guidance is to calculate input cost from the usage object rather than from a guess about the prompt [1]. That log is the control: budget, cache read, small-model step, and stop.
Practical takeaways
- Set a per-item token cap and a labor-cost cap before the batch, and enforce both in the workflow after every call.
- Put stable instructions and source text at the front of the prompt, and confirm cache reads in the usage fields before you count on a discount.
- Send the narrow first step to the smaller model, and call the larger model only when the case is actually hard or the small model is unsure.
- Size the output cap so one call can finish a usable answer. A cap that is too tight can bill reasoning tokens and return nothing you can use.
- Stop the loop when the next call would cost more than finishing the item by hand, and treat that stop as the control working.
How we can help
Have more questions or want to get in touch? Our team helps operations leaders put a budget, a cache, a smaller-model step, and a stop rule on a workflow they already run. Reach us through our contact page, call (980) 322-4500, or email [email protected].
Citations
- OpenAI, "Prompt caching" (2026)
- OpenAI, "Reasoning models" (2026)
- Anthropic, "Pricing" (2026)
- Google AI for Developers, "Context caching" (2026-09-02)
- Google Cloud, "Context caching overview" (2026-09-22)
- NIST, "AI Risk Management Framework" (2023-01-26)
- NIST AI Resource Center, "AI RMF Core" (2023)
