Agentic AI

Agentic AI Cost Optimization: Cut LLM Inference Costs by 60–80% in Production

AI agents that work beautifully in demos can become budget nightmares in production. A multi-step ReAct agent making 12 GPT-4o calls per user request, at 2 000 tokens each, costs ~$0.29 per session — that's $290 per 1 000 users. This guide gives you the concrete engineering techniques to cut that bill by 60–80% without degrading quality.

Md Sanwar Hossain April 6, 2026 18 min read LLM Cost Optimization

1. Understanding Where Costs Come From

LLM API costs are priced per token — typically split between input tokens (cheaper) and output tokens (more expensive). For GPT-4o in 2026, input costs ~$2.50/M tokens and output ~$10/M tokens. In an agentic loop, costs compound quickly:

• System prompt repetition: Your 800-token system prompt is sent on every single LLM call in the loop. A 10-step agent sends it 10 times.
• Full conversation history: ReAct agents append every observation back into context. By step 8, you may be sending 6 000 tokens of history just to ask a simple follow-up question.
• Redundant tool calls: Agents frequently call the same retrieval tool with semantically identical queries. Without caching, each call is billed fresh.
• Oversized output requests: Setting max_tokens=4096 "just to be safe" when the task only needs 200 tokens wastes output quota and increases latency.
• Wrong model for the task: Using GPT-4o to classify a support ticket into one of 5 categories costs 20× more than using GPT-4o-mini for the same job.

Before optimizing, instrument your agent to log token counts per call, per step, and per session. You cannot optimize what you cannot measure. Libraries like LangSmith, Langfuse, and Helicone provide this telemetry out of the box.

2. Semantic Caching: The Highest-ROI Optimization

Exact-match caching (Redis key = hash of prompt) helps for identical prompts, but agents rarely send pixel-perfect identical requests. Semantic caching embeds the query, finds similar cached responses within a cosine similarity threshold, and returns the cached answer. This eliminates 30–50% of LLM API calls in most production workloads.

# Semantic cache implementation with Redis + pgvector
import numpy as np
from openai import OpenAI
import redis
import json

client = OpenAI()
r = redis.Redis(host="localhost", port=6379)

SIMILARITY_THRESHOLD = 0.95  # tune based on your quality requirements
CACHE_TTL = 3600  # 1 hour

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",  # 5x cheaper than ada-002
        input=text
    )
    return response.data[0].embedding

def cosine_similarity(a: list, b: list) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def semantic_cache_lookup(query: str) -> str | None:
    query_emb = get_embedding(query)
    # scan cached embeddings (use pgvector for scale)
    for key in r.scan_iter("cache:emb:*"):
        cached = json.loads(r.get(key))
        sim = cosine_similarity(query_emb, cached["embedding"])
        if sim >= SIMILARITY_THRESHOLD:
            return cached["response"]  # cache HIT
    return None  # cache MISS

def llm_call_with_cache(query: str) -> str:
    cached = semantic_cache_lookup(query)
    if cached:
        return cached  # free!

    # cache miss — call the API
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    )
    result = response.choices[0].message.content
    emb = get_embedding(query)
    cache_key = f"cache:emb:{hash(query)}"
    r.setex(cache_key, CACHE_TTL, json.dumps({"embedding": emb, "response": result}))
    return result

For production scale, replace Redis scan with pgvector's <-> cosine distance operator or use a purpose-built semantic cache service like GPTCache or Zep. The embedding cost (~$0.02/M tokens) is orders of magnitude cheaper than the LLM calls you're saving.

3. Intelligent Model Routing

Not every agent step requires your most capable (and expensive) model. A router classifies each subtask and sends it to the cheapest model that can handle it reliably. Empirical data across production agentic systems shows:

• ~60% of agent subtasks are simple classification, extraction, or formatting — perfect for GPT-4o-mini, Claude Haiku, or Gemini Flash.
• ~30% require moderate reasoning — suitable for GPT-4o, Claude Sonnet, or Gemini Pro.
• ~10% require frontier-level reasoning — justify GPT-4o or o3.

# Model router for an agentic pipeline
from enum import Enum

class TaskComplexity(Enum):
    SIMPLE = "simple"     # extraction, classification, formatting
    MODERATE = "moderate" # multi-step reasoning, synthesis
    COMPLEX = "complex"   # planning, code generation, long-form analysis

MODEL_MAP = {
    TaskComplexity.SIMPLE: "gpt-4o-mini",    # $0.15/M input, $0.60/M output
    TaskComplexity.MODERATE: "gpt-4o",        # $2.50/M input, $10/M output
    TaskComplexity.COMPLEX: "o3",             # reserved for hard problems
}

def classify_task(task_description: str) -> TaskComplexity:
    """Use a cheap model to classify the task complexity"""
    prompt = f"""Classify this task complexity as simple/moderate/complex.
Simple: extraction, formatting, yes/no, classification.
Moderate: multi-step reasoning, synthesis, explanation.
Complex: long-form analysis, planning, mathematical reasoning.

Task: {task_description}
Answer with one word: simple, moderate, or complex."""

    response = openai.chat.completions.create(
        model="gpt-4o-mini",  # use cheap model for the classification itself
        messages=[{"role": "user", "content": prompt}],
        max_tokens=5,
    )
    label = response.choices[0].message.content.strip().lower()
    return TaskComplexity(label) if label in [c.value for c in TaskComplexity] else TaskComplexity.MODERATE

def routed_llm_call(task: str, messages: list) -> str:
    complexity = classify_task(task)
    model = MODEL_MAP[complexity]
    response = openai.chat.completions.create(model=model, messages=messages)
    return response.choices[0].message.content

4. Token Budgeting & Prompt Compression

Set explicit token budgets for every agent task and compress inputs aggressively before they hit the API:

• LLMLingua / Selective Context: These prompt compression libraries remove low-entropy tokens from your context (conjunctions, filler phrases, redundant sentences) with 3–4× compression ratios while losing less than 5% of semantic content.
• Rolling window history: Instead of sending the entire conversation history, keep only the last N turns. For most agents, N=5–8 captures all the relevant context.
• Summarize-and-compress: When history exceeds a token threshold, run a cheap summarization step (GPT-4o-mini) to compress it before the next expensive call.
• Dynamic max_tokens: Set max_tokens based on expected output length. A "classify into 5 categories" task needs max_tokens=10, not 4096.
• Structured output schemas: JSON mode with a tight schema forces the model to be concise. Unstructured free-text responses are often 2–3× more verbose than necessary.

5. Agent Context Window Management

The biggest token bill in long-running agents comes from context growth. A 10-step agent that starts with 2 000 tokens and grows by 500 per step ends up sending 6 750 tokens on step 10 — nearly 4× what step 1 cost. Use these strategies:

• Hierarchical summarization: After every N steps, compress the scratchpad into a concise summary. Store full detail in an external memory store (vector DB) and retrieve only relevant snippets as needed.
• Tool result trimming: When a tool returns 10 KB of raw text, extract only the relevant 200 tokens before appending to context. Use a cheap extraction LLM or a regex/parser.
• Separate working memory from long-term memory: Keep the agent's active context small (current step + recent history) and offload older memories to retrieval-augmented storage.
• Context budget enforcement: Implement a hard limit on input tokens per call. If you hit the limit, trigger a compression step automatically before proceeding.

6. Batching & Async Inference

OpenAI's Batch API and Anthropic's batch endpoints offer 50% cost reduction for asynchronous workloads. If your agent performs background tasks (document processing, scheduled analysis, bulk data enrichment) that don't need real-time responses, batch them:

# OpenAI Batch API — 50% cheaper, up to 24h turnaround
import json
from openai import OpenAI

client = OpenAI()

# Prepare batch file
requests = []
for i, document in enumerate(documents_to_process):
    requests.append({
        "custom_id": f"doc-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o",
            "messages": [
                {"role": "system", "content": "Extract key entities."},
                {"role": "user", "content": document}
            ],
            "max_tokens": 500
        }
    })

# Upload and submit batch
with open("/tmp/batch_requests.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

batch_input_file = client.files.create(
    file=open("/tmp/batch_requests.jsonl", "rb"),
    purpose="batch"
)
batch = client.batches.create(
    input_file_id=batch_input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)
print(f"Batch submitted: {batch.id}")

7. Speculative Decoding & Draft Models

When self-hosting models (Ollama, vLLM, TGI), speculative decoding dramatically reduces token generation latency and effective compute cost. A small "draft" model generates candidate tokens in parallel; the large model verifies them in batch rather than sequentially. In practice, this yields 2–3× throughput improvement on typical agentic outputs.

• Use Llama 3.2 3B as a draft model for Llama 3.1 70B — 40–60% throughput gain.
• vLLM supports speculative decoding out-of-the-box with --speculative-model flag.
• Combine with prefix caching: vLLM and SGLang cache the KV attention state for repeated prefixes (your system prompt), eliminating redundant computation on every agent call.

8. Self-Hosted vs API Trade-offs

At high volume (>10M tokens/day), self-hosting open-weight models becomes economically attractive. The break-even analysis depends on your cloud GPU costs:

9. Cost Monitoring & Alerting

You cannot optimize what you cannot see. Implement these observability practices from day one:

• Per-agent cost attribution: Tag every LLM call with the agent name, workflow ID, and user segment. Export to DataDog, Grafana, or Langfuse for dashboards.
• P95 cost alerts: Alert when a session's token spend exceeds a percentile threshold (e.g., "P95 session cost > $0.50").
• Cost per task type: Track cost per agent action type to identify which tool calls are disproportionately expensive.
• Cache hit rate KPI: Target >40% semantic cache hit rate. Below 20% indicates your cache TTL or similarity threshold needs tuning.
• Model mix dashboard: Track % of calls served by each model tier. Ensure your router is correctly sending simple tasks to cheap models.

# Cost tracking middleware (Python)
import time
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class LLMCallMetrics:
    agent_name: str
    model: str
    input_tokens: int
    output_tokens: int
    cache_hit: bool
    latency_ms: float
    cost_usd: float = field(init=False)

    # 2026 pricing (update as needed)
    PRICES = {
        "gpt-4o":      {"input": 2.50e-6, "output": 10.0e-6},
        "gpt-4o-mini": {"input": 0.15e-6, "output": 0.60e-6},
        "o3":          {"input": 10.0e-6, "output": 40.0e-6},
    }

    def __post_init__(self):
        pricing = self.PRICES.get(self.model, {"input": 2.50e-6, "output": 10.0e-6})
        self.cost_usd = (
            self.input_tokens * pricing["input"] +
            self.output_tokens * pricing["output"]
        )

def tracked_llm_call(agent_name: str, model: str, messages: list,
                     cache_hit: bool = False) -> tuple[str, LLMCallMetrics]:
    start = time.time()
    response = openai.chat.completions.create(model=model, messages=messages)
    elapsed_ms = (time.time() - start) * 1000
    usage = response.usage
    metrics = LLMCallMetrics(
        agent_name=agent_name, model=model,
        input_tokens=usage.prompt_tokens, output_tokens=usage.completion_tokens,
        cache_hit=cache_hit, latency_ms=elapsed_ms
    )
    # emit to your metrics pipeline (DataDog, Prometheus, Langfuse...)
    emit_metrics(metrics)
    return response.choices[0].message.content, metrics

10. The Cost Optimization Checklist