AI Cost Control

01What I'm tracking

Inference spend is now the fastest-growing line item in most engineering budgets I see — often eclipsing compute, storage, and observability combined. My job on these systems is to keep AI economically rational: every token, GPU-second, and vector lookup either generates measurable user value or gets cut.

What's in this note

• The full taxonomy of where dollars go in an LLM system.
• Architectural patterns: routing, caching, RAG, agents, batching, serving.
• Engineering trade-offs: latency vs. cost vs. quality vs. risk.
• FinOps tooling, observability schemas, and budget guardrails.
• Org design, vendor negotiation, and a 5-stage maturity model.

02Cost Drivers Anatomy

Before optimizing, map the bill. AI cost decomposes into seven primary drivers. Most teams over-index on (1) and ignore (3)–(7), which is exactly where the avoidable waste lives.

Driver deep-dive

03Unit Economics & Formulas

Every product decision should resolve to a cost-per-action (CPA) number. If you can't compute CPA on demand, you cannot govern AI cost.

Core formula

CPA = Σ over calls c in action a:
      (input_tokens(c)  × P_in(model_c)
     +  output_tokens(c) × P_out(model_c)
     +  cached_tokens(c) × P_cache(model_c))
    +  retrieval_cost(a)
    +  tool_cost(a)
    +  infra_amortization(a)

Five ratios I won't ship without

A worked example

Illustrative blended rates. The point is not the absolute dollars — it's that stacked optimizations compound to ~13× without changing the user-visible product.

04Reference Cost-Aware Architecture

The architecture below pushes every request through layers that progressively get more expensive. The cheapest possible answer wins. Each layer has a clear cost owner.

Layer-by-layer cost intent

05Model Routing & Cascades

Routing is the single highest-leverage optimization. A well-designed router cuts spend 40–70% with no quality loss because most queries are easy. The goal: send each request to the cheapest model whose expected quality clears the SLA.

Three routing strategies

Designing the routing policy

1. Label the workload. Sample 1–5K real prompts, annotate difficulty / required capability / latency tolerance.
2. Calibrate thresholds. For each candidate model, measure quality on samples; pick the cheapest that meets the SLA.
3. Decide escalation signal. Options: log-prob/entropy, judge model verdict, JSON schema validation, regex/tool-call success, user thumbs-down.
4. Bound the cascade. Hard cap on escalations per request (e.g., 1) to prevent cost runaway.
5. Continuously re-evaluate. Cheap models improve quickly; re-run routing eval monthly.

06Caching Strategies

Caching is the cheapest dollar you'll ever save. A serious system runs at least four distinct caches.

Designing prompt caches that actually hit

Provider prompt caches (Anthropic, OpenAI, Google) only hit when the prefix is byte-identical. The prompt structure I enforce on teams I work with:

[ STABLE — cached ]
  system instructions
  tool / function definitions
  large reference documents
  few-shot examples

[ VOLATILE — not cached ]
  user query
  per-request retrieved snippets (or place AFTER stable block)
  timestamps, user_id, session id

Semantic cache: when it's a trap

• Don't use semantic cache for personalized, account-specific, or time-sensitive answers without per-tenant namespaces.
• Don't set similarity threshold too low — false positives serve wrong answers and erode trust.
• Do log every cache hit's similarity score and sample audit a slice weekly.
• Do invalidate by content-hash when source documents change.

07Context & Prompt Compression

Context length is the second-biggest token lever after model choice. The discipline: send the minimum context needed for a correct answer, no more.

The compression toolbox

The asymmetry: output tokens cost more

On most APIs, output tokens are 3–5× the price of input. Yet engineers focus on shrinking input. Equally important:

• Constrain output with a strict JSON/XML schema; reject and retry only with stronger constraints.
• Use max_tokens as a hard guardrail, not a hope.
• Discourage chain-of-thought in the visible answer unless a thinking budget is genuinely needed; if it is, use a separate thinking budget and don't include scratchpad in the final shown text.
• For listy answers, ask explicitly for "3 bullets, ≤ 12 words each."

08RAG Cost Optimization

RAG looks cheap because each call is small, but at scale embedding, storage, retrieval, and rerank fees stack. Worse, naive RAG inflates prompt size and the model bill it was meant to reduce.

RAG cost anti-patterns

• Embedding the whole corpus every nightly run. Use content hashes; re-embed only changed chunks.
• Top-k = 20 by default. Measure recall; most workloads peak at 4–8.
• 1536-dim embeddings for short SKU descriptions. Use a smaller model and dimension.
• Per-request reranking with a frontier model. Use a dedicated cross-encoder.
• No tenant scoping. Larger index = more recall noise + more tokens fed in.

09Agentic Cost Control

Agents are the most dangerous cost category because their cost is unbounded by default. A single user query can trigger dozens of model calls, tool calls, and retries. Naïve loops have killed budgets overnight.

The five non-negotiable bounds

Cost shape of agentic systems

cost(session) ≈ Σ steps × (avg_input_tokens + avg_output_tokens) × price
              + Σ tool_calls × tool_cost
              + Σ subagent_invocations × subagent_cost

Without compaction, avg_input_tokens grows with each step. The cost curve becomes quadratic. Always compact.

10Batching & Async Inference

If your workload is not user-facing in real time, batching is a 40–60% discount you should not leave on the table.

Three batching modes

Async, scheduled, and shadow workloads

• Push evals, content moderation backfills, and re-embedding jobs to the Batch API.
• Shadow-test cheaper models in parallel; never block the user path on them.
• Use off-peak windows for self-hosted fleets to amortize fixed GPU costs.

11Fine-tune vs. Prompt vs. Distill

The default impulse to "just fine-tune" is usually a cost mistake. Here is the decision order I enforce before anyone trains anything.

Distillation — the underused weapon

For high-volume, well-defined tasks (intent classification, extraction, routing decisions, ticket triage), distill frontier-model behavior into a fine-tuned small model. Typical economics:

• Capex: a few hundred to a few thousand dollars of frontier calls to generate the training set + GPU-hours for tuning.
• Opex: 5–20× cheaper per call than the teacher, often with comparable quality on the narrow task.
• Break-even: usually within weeks at > 1M calls/month.

12Self-Hosted Serving Economics

Self-hosting beats SaaS APIs only when utilization is high and predictable. Otherwise you pay for idle GPUs.

The break-even formula

break_even_qps = (gpu_hourly_cost × n_gpus) / (api_price_per_call × 3600 × utilization)

If your steady-state QPS is materially below break-even, stay on the API. If it's above and stable, consider self-hosting — and only then if you have the on-call expertise to operate it.

What actually moves self-hosted cost

13Observability & FinOps

You cannot control what you cannot attribute. Every LLM call must be tagged at emission with enough metadata to roll up by tenant, feature, environment, and team.

Minimum required telemetry per call

{
  "request_id":      "uuid",
  "timestamp":       "2026-05-18T14:22:01Z",
  "tenant_id":       "acme-corp",
  "user_id_hash":    "...",
  "feature":         "support.summary",
  "environment":     "prod",
  "team":            "support-platform",
  "model":           "claude-sonnet-4-6",
  "provider":        "anthropic",
  "input_tokens":     3421,
  "output_tokens":    412,
  "cached_tokens":   2980,
  "cache_layer":     "provider_prompt",
  "tool_calls":      [{"name":"search_kb","cost":0.0001}],
  "latency_ms":       1840,
  "cost_usd":         0.0061,
  "budget_bucket":   "support-platform/prod/monthly",
  "outcome":         "success",
  "quality_signal":  {"thumbs":"up"}
}

Four dashboards I require before launch

Tooling landscape

• LLM-native: Langfuse, Helicone, Arize Phoenix, Braintrust, OpenLLMetry, Datadog LLM Observability.
• Cloud FinOps: AWS Cost Explorer / CUR, GCP Billing BigQuery export, Azure Cost Management, plus Vantage, CloudHealth, Cloudability.
• Custom ledger: a simple append-only table in your warehouse (BigQuery / Snowflake) is often the most trustworthy single source of truth — emit from a thin SDK wrapper around every provider call.

14Quotas, Budgets & Guardrails

Soft guidance is not control. Cost guardrails must be enforced in the request path, not in a wiki page.

The three-tier guardrail stack

Concrete guardrails worth shipping

• Per-tenant daily $ cap (with override workflow).
• Per-feature monthly $ cap tied to product manager sign-off.
• Max tokens per request at the SDK wrapper level — refuse if exceeded.
• Max steps per agent session with graceful fallback.
• Production-only model allowlist — prevent staging models leaking to prod.
• Cost-anomaly autoresponder — auto-page on > N% deviation from forecast.
• Eval-traffic isolation — separate API key + budget; never pollute prod cost reports.

15Vendor & Contract Strategy

List prices are starting points, not endings. Above ~$50K/month of spend, real money is in commercial structure.

Negotiation levers (in order of impact)

1. Volume commitments with quarterly true-ups — 10–30% off list at meaningful scale.
2. Multi-year deals with model-class portability — protect against being locked to a deprecated SKU.
3. Reserved throughput / provisioned capacity — predictable price, predictable latency.
4. Cross-product bundles when the vendor sells more than inference (eval, embeddings, fine-tuning).
5. Free / discounted dev & eval traffic — surprisingly often granted.
6. Right-to-export embeddings / tuned weights — exit clause that protects future negotiation.

Multi-provider posture

Even if one vendor is your strategic primary, maintain a tested fallback for two reasons:

• Resilience: outages happen; cost spikes during incidents are real.
• Leverage: a vendor that knows you can route away will price differently.

Hide vendor specifics behind an internal LLM gateway with a unified schema; this also gives you the natural place to apply routing, caching, and budgets.

16Org Model & RACI

Cost control fails when no one owns it. The default state is "AI is everyone's tool and no one's bill."

The "AI Platform" team charter

If you do not have one, propose it. A small (3–8 person) AI Platform team owns:

• The LLM gateway / SDK wrapper (single point of vendor swapping, routing, caching, budgeting).
• The cost ledger and dashboards.
• Shared retrieval infrastructure.
• Eval and red-team harnesses.
• Production guardrails (PII, abuse, cost).

This team's ROI is measured as $ saved per $ spent on the team, and at scale it is consistently > 10×.

17AI Cost Maturity Model

18Anti-Patterns & How to Diagnose Them

19What I keep on my desk

Not commandments — a checklist I re-read before sign-off. If every item is yes, the system is cost-disciplined enough to scale.

Visibility

• Every LLM call is tagged with tenant, feature, environment, team, model.
• There is a single ledger (warehouse table) that is the source of truth for AI spend.
• Cost-per-successful-action is published per feature, refreshed daily.
• Forecast-to-month-end exists, with alerting at deviation thresholds.

Architecture

• All provider calls go through an internal LLM gateway.
• The gateway implements: routing, prompt caching, semantic caching, budgets, retries.
• Every prompt is structured stable-first, volatile-last.
• RAG uses hybrid retrieval with reranking and a hard top-k cap.
• Agents have hard caps on steps, tokens, dollars, and identical-tool-call loops.

Governance

• Each feature has a named budget owner and a monthly cap.
• Soft/hard/circuit-break guardrails are wired in the request path.
• Eval traffic uses separate API keys and its own budget bucket.
• A quarterly cost review covers spend, unit economics, and vendor posture.

Continuous improvement

• Routing thresholds re-evaluated monthly on a held-out set.
• New model releases are benchmarked for $/quality before promotion.
• High-volume stable tasks are reviewed annually for distillation.
• An "AI cost office hours" or equivalent exists; teams know who to ask.

20Glossary

Terms used in this note

21References & Sources

Annotated bibliography behind the cost-driver map, unit-economics formulas, cost-aware architecture, routing cascades, four cache layers, RAG pipeline levers, agentic bounds, batching economics, fine-tune/distill decision tree, self-hosted break-even, FinOps telemetry schema, guardrail stack, vendor posture, RACI, maturity model, anti-patterns, and desk checklist. Section tags (e.g. §05) show where each source is used. Diagrams and worked examples are my synthesis unless noted.

Scope. Synthesis of FinOps, LLM-inference, RAG, and platform-engineering sources (May 2026). Hero KPI ranges (10–100× model cost delta, 50–90% savings from caching/routing, 3–5× agentic inflation) blend FrugalGPT, provider pricing docs, and field benchmarks — directional ranges, not guarantees for your workload. Dollar figures in §03 worked example use illustrative blended rates; verify against current list prices before budgeting. Not vendor, financial, or procurement advice.

Citations are numbered continuously [1]–[n] within this section.

FinOps, cloud cost accountability & AI spend governance (§01, §13–§14, §16–§17)

1. FinOps Foundation, FinOps Framework & FinOps Principles. 2024–25. Capabilities model for bringing financial accountability to variable cloud spend — backbone for §13 FinOps cadence, §14 budget guardrails, and §17 maturity stages 3–4. finops.org/framework — §13, §14, §17.
2. FinOps Foundation, FinOps for AI working group & community guidance. 2024–25. Applying FinOps disciplines to GPU and inference billing — §01 hero framing and §13 ledger/dashboard requirements. finops.org — §01, §13.
3. AWS, Cost and Usage Report (CUR) & Cost Explorer documentation. Granular attribution of cloud/GPU spend — §13 tooling landscape and §02 driver 4 (serving infra). docs.aws.amazon.com/cur — §02, §13.
4. Google Cloud, BigQuery billing export & Cloud Billing APIs. Warehouse-native cost rollups — §13 custom-ledger pattern and §16 FinOps RACI. cloud.google.com/billing — §13, §16.

LLM token economics, pricing & inference cost trends (§01, §03, hero KPIs)

1. OpenAI, API Pricing & usage documentation. 2025–26. Input/output token rates, Batch API discount, prompt caching — §03 CPA formula, §03 worked example, §06 L3 cache, §10 batching. openai.com/api/pricing — §03, §06, §10.
2. Anthropic, API Pricing & Prompt Caching documentation. 2025–26. Cached-input pricing (≈10–25% of base input), batch discounts — §06 prompt-cache layer and §03 output/input asymmetry. docs.anthropic.com/prompt-caching — §03, §06, §07.
3. Google Cloud, Vertex AI Generative AI pricing & Context caching. 2025–26. Gemini context-cache TTL and token billing — §06 four-layer cache and §04 architecture prompt-cache node. cloud.google.com/vertex-ai/pricing — §04, §06.
4. Stanford HAI, 2025 AI Index Report — Technical Performance & Economy chapters. 2025. Inference-cost declines, model tiers, and adoption economics — background for §01 hero 10–100× tier delta and §05 monthly re-eval guidance. hai.stanford.edu/ai-index — §01, §05, §11.
5. McKinsey & Company, The Economic Potential of Generative AI: The Next Productivity Frontier. June 2023. Gen-AI cost as a scaling constraint for enterprises — §01 “fastest-growing line item” context and §15 vendor negotiation at scale. mckinsey.com — §01, §15.

Cost-aware routing, cascades & model selection (§05, FIG 3, §18)

1. Chen, L. et al., “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.” arXiv:2305.05176, 2023. LLM cascade and prompt-adaptation strategies; reported large cost reductions with quality preservation — §01 hero 50–90% range, §05 cascade routing, §05 callout on cost-per-success. arxiv.org/abs/2305.05176 — §01, §05, hero.
2. Ong, I. et al., “RouteLLM: Learning to Route LLMs with Preference Data.” arXiv:2406.18647, 2024. Learned routers sending queries to strong vs. weak models — §05 classifier router and §04 L3 cost-aware router. arxiv.org/abs/2406.18647 — §04, §05.
3. Shnitzer, A. et al., “Large Language Model Routing with Benchmark Datasets.” arXiv:2309.15789, 2023. Router design and evaluation methodology — §05 routing-policy calibration steps. arxiv.org/abs/2309.15789 — §05.
4. Yan, E., “Patterns for Building LLM-based Systems & Products.” eugeneyan.com, 2023–25 (ongoing). Practical patterns for routing, eval, and cost-quality trade-offs — cited in §05 design steps and §08 RAG anti-patterns mindset. eugeneyan.com/writing/llm-patterns — §05, §08, §19.

Caching — prompt, semantic & retrieval layers (§06, FIG 4, §04)

1. Anthropic, Prompt caching — implementation guide & pricing. Byte-identical prefix requirements, TTL, cache breakpoints — §06 stable/volatile prompt structure and FIG 4 L3. docs.anthropic.com — §04, §06.
2. OpenAI, Prompt caching (automatic caching on supported models). 2024–25. Provider-side prefix reuse — §06 engineering rule on stable-first prompt ordering. platform.openai.com/docs/guides/prompt-caching — §06.
3. Zilliz / GPTCache project, “GPTCache: An open-source semantic cache for LLM applications.” 2023–25. Vector-similarity caching patterns and false-positive risks — §06 L2 semantic cache and semantic-cache trap bullets. github.com/zilliztech/GPTCache — §06, §18.
4. Redis, Vector search & caching best-practices documentation. Exact-match and TTL caches at the edge — §04 L1 edge cache and §06 L1 exact-match layer. redis.io/docs — §04, §06.

Context compression, long-context limits & RAG economics (§07–§08, FIG 5)

1. Lewis, P. et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 2020. Foundational RAG architecture — §08 pipeline stages and §07 chunk-and-retrieve compression lever. arxiv.org/abs/2005.11401 — §07, §08.
2. Liu, N. F. et al., “Lost in the Middle: How Language Models Use Long Contexts.” TACL 2024. Models under-use middle context — supports §07 hard top-k caps and §08 rerank-cap lever over stuffing. arxiv.org/abs/2307.03172 — §07, §08.
3. Es, S. et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation.” 2023. RAG eval metrics (faithfulness, recall) — §08 stage-6 eval and §13 judge-model cost control. arxiv.org/abs/2309.15217 — §08, §13.
4. TruLens documentation — LLM app evaluation & tracing. 2024–25. Feedback functions and RAG triad — §08 eval stage and §13 observability tooling list. trulens.org — §08, §13.
5. Pinecone, Weaviate, Qdrant — vector-database sizing & embedding-dimension guides. 2024–25. Storage and query cost trade-offs — §08 embed stage (dim 256–512 vs 1536) and §02 driver 2. pinecone.io/learn — §02, §08.

Agentic systems, loops & unbounded cost risk (§09, hero 3–5×, §18)

1. Yao, S. et al., “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. Tool-use loops that multiply LLM calls — §09 cost-shape formula and step-budget rationale. arxiv.org/abs/2210.03629 — §09.
2. LangChain / LangGraph documentation — agent runtimes, recursion limits & checkpoints. 2024–25. Step caps and state management patterns — §09 five non-negotiable bounds and loop detector. langchain-ai.github.io/langgraph — §09, §14.
3. Shinn, N. et al., “Reflexion: Language Agents with Verbal Reinforcement Learning.” NeurIPS 2023. Iterative self-correction loops — background for §09 quadratic cost curve without compaction. arxiv.org/abs/2303.11366 — §09.
4. Helicone & Langfuse case studies on agent cost spikes. 2024–25. Runaway session patterns and guardrail responses — §09 “pattern I've seen” callout and §13 anomaly dashboard. helicone.ai/blog — §09, §13.

Batching, async inference & throughput (§10, §12)

1. OpenAI, Batch API documentation. ~50% discount, 24-hour completion window — §10 provider Batch API row and async workload bullets. platform.openai.com/docs/guides/batch — §10.
2. Anthropic, Message Batches API. 2024–25. Async batch processing for non-latency-sensitive workloads — §10 batching modes table. docs.anthropic.com/message-batches — §10.
3. Yu, G.-I. et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models.” OSDI 2022. Continuous batching concepts — precursor to modern inference servers in §10 micro-batching row. usenix.org/osdi22 — §10, §12.

Fine-tuning, LoRA, distillation & the tune-last discipline (§11, FIG 6)

1. Hu, E. J. et al., “LoRA: Low-Rank Adaptation of Large Language Models.” ICLR 2022. Parameter-efficient fine-tuning — §11 LoRA/PEFT card and §02 driver 3 amortization. arxiv.org/abs/2106.09685 — §02, §11.
2. Hinton, G., Vinyals, O., & Dean, J., “Distilling the Knowledge in a Neural Network.” NeurIPS 2014 Deep Learning Workshop. Knowledge distillation foundation — §11 distillation economics and FIG 6 decision tree. arxiv.org/abs/1503.02531 — §11.
3. OpenAI, Model distillation guide & fine-tuning API documentation. 2024–25. Teacher-student workflows and when distillation pays — §11 break-even bullets and §18 tune-before-prompt anti-pattern. platform.openai.com/docs/guides/distillation — §11, §18.
4. Dettmers, T. et al., “QLoRA: Efficient Finetuning of Quantized LLMs.” NeurIPS 2023. Lower-cost fine-tune runs — §11 LoRA card capex range. arxiv.org/abs/2305.14314 — §11.

Self-hosted serving, GPU economics & inference stacks (§12, §02 driver 4)

1. Kwon, W. et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP 2023 (vLLM). PagedAttention and throughput gains — §12 vLLM row and §04 self-hosted OSS tier. arxiv.org/abs/2309.06180 — §04, §12.
2. vLLM project documentation. 2024–25. Continuous batching, prefix caching, quantization — §12 lever table and §10 micro-batching. docs.vllm.ai — §10, §12.
3. NVIDIA, TensorRT-LLM documentation. 2024–25. FP8/INT quantization and speculative decoding — §12 quantization and speculative-decoding rows. nvidia.github.io/TensorRT-LLM — §12.
4. Hugging Face, Text Generation Inference (TGI) documentation. Production serving stack — §12 vLLM/TGI comparison and glossary entry. huggingface.co/docs/tgi — §12, §20.
5. NVIDIA & cloud-provider GPU pricing pages (H100/H200, reserved vs spot). 2025–26. Break-even inputs for §12 formula and §18 self-host-at-low-utilization anti-pattern. nvidia.com/data-center — §12, §18.

Observability, LLM tracing & cost ledgers (§13, §04 ledger, §19)

1. Langfuse documentation — LLM tracing, cost tracking & evaluations. 2024–25. Per-call metadata schema aligned with §13 telemetry JSON — §13 tooling and §19 visibility checklist. langfuse.com/docs — §04, §13, §19.
2. Helicone — LLM observability & cost analytics. 2024–25. Gateway proxy pattern for attribution — §04 cost ledger and §13 LLM-native tooling list. docs.helicone.ai — §04, §13.
3. Braintrust — eval + production logging for LLM apps. 2024–25. Cost-per-success tracking — §03 CPA denominator and §13 unit-economics dashboard. braintrust.dev/docs — §03, §13.
4. Arize AI, Phoenix — open-source LLM observability. 2024–25. Tracing and eval for RAG — §08 eval stage and §13 tooling landscape. docs.arize.com/phoenix — §08, §13.
5. Datadog, LLM Observability product documentation. 2024–25. Enterprise tracing and cost correlation — §13 tooling landscape and §16 AI Platform charter. docs.datadoghq.com/llm_observability — §13, §16.
6. OpenTelemetry, generative-AI semantic conventions (draft). 2024–25. Standard spans for model calls — §13 minimum telemetry schema design. opentelemetry.io/gen-ai — §13.

Guardrails, quotas, platform teams & vendor posture (§14–§16, §15)

1. Google SRE Book — chapters on monitoring, alerting & capacity planning. O'Reilly, 2016–18. Circuit-breaker and throttle patterns — §14 three-tier guardrail stack and §14 circuit-break row. sre.google/sre-book — §14.
2. Team Topologies (Skelton & Pais) — platform team as internal product. 2019. AI Platform charter in §16 — gateway ownership and enablement vs feature-team autonomy. — §16.
3. Partnership on AI & NIST, AI Risk Management Framework — resource allocation & monitoring. 2023. Governance patterns for production AI — §14 prod model allowlist and §16 guardrails ownership. nist.gov/ai-rmf — §14, §16.
4. Flexera / Gartner cloud-cost negotiation research (enterprise licensing). 2023–25. Volume-commitment and true-up structures — §15 negotiation levers 1–2. flexera.com/blog/finops — §15.

Author synthesis

1. Truong, L., AI Cost Control — personal working notes. May 2026. Original diagrams (FIG 1–7), cost-aware architecture, maturity model, anti-pattern table, desk checklist, and §03 stacked-optimization example. LinhTruong.com — all sections.