LLM Application Observability: Traces, Evals, and Monitoring

Traditional web application monitoring can usually get by with QPS, latency, and error rates. LLM applications introduce three complications: outputs are nondeterministic (the same request produces a different result each time), execution chains are long and expensive (one user request may trigger multiple rounds of model and tool calls), and quality matters more than "whether there was an error" (a 200 OK response that fails to answer the question is worse than a 500).

LLM applications therefore need a dedicated observability system with three layers: trace, eval, and metric.

1. Trace: Connect the Entire Execution Chain

A user request in an LLM application often flows like this:

User request
 → Routing decision
 → Query rewriting (1 model call)
 → Vector retrieval + rerank
 → Assemble the prompt (2 model calls: primary answer + self-check)
 → Tool call (read DB)
 → Final answer

Any intermediate step can fail: retrieval may miss the relevant content, rewriting may drift off course, or a tool may return nothing. Without a trace, all you see is "the user asked X, and the model answered Y," with no way to identify which stage failed.

A trace should record every span:

• Every model call: prompt, model, parameters, output, latency, and tokens.
• Every tool or retrieval operation: input parameters, returned results, and latency.
• Parent-child relationships across the entire chain (which child calls were nested within a single response).

For tooling, the open-source Langfuse / LangSmith are de facto standards in this category (they are decoupled from specific frameworks, and OpenTelemetry can handle this as well). Once integrated, they show the complete tree-structured trace for every request, and you can open any node to inspect its input and output.

Practical lesson: A trace is the "debugger" for an LLM application. Launching an AI application without traces means that when something goes wrong, your only option is to reproduce the user's input and guess blindly. That is extremely inefficient.

2. Eval: "Continuous Integration" for Quality

A trace tells you "what happened"; an eval tells you "whether it was good." This is the layer LLM applications most need to add and most often lack.

The core approach is to build an evaluation set containing dozens to hundreds of input → expected properties/reference answer cases. Whenever you change a prompt, switch models, or tune retrieval, run the set and compare the results quantitatively.

There are three types of evaluation:

1. Retrieval evaluation (for RAG)

• Context Recall: How much of the content that should have been retrieved was retrieved.
• Context Precision: How much of the retrieved content was relevant.

2. Generation evaluation

• Faithfulness: Whether the answer is fully supported by the context (to prevent hallucinations).
• Answer Relevancy: Whether the answer addresses the question.

3. Business evaluation (custom)

• Format validity (whether the output is valid when JSON is required).
• Safety (whether the output violates safety rules).
• Business correctness (score against a manually labeled set of "good answers").

Tools include RAGAS (specifically for RAG) and promptfoo / DeepEval (general-purpose eval frameworks). Most use an "LLM as judge" to score results automatically, eliminating the need to label each result manually.

The value of evals lies in regression testing: without them, changing prompts is guesswork; with them, every change has an objective comparison. Ideally, evals run in CI, and any prompt or model change that lowers evaluation scores is blocked.

3. Metric: Latency, Tokens, Cost, and Errors

The metric layer serves operations and SLOs:

• Latency distribution: p50 / p95 / p99. LLM application latency has high variance, so averages are misleading; inspect the tail.
• Token usage and cost: Break these down by prompt / completion, by user, and by feature. This is the unique "operating cost" of an LLM application; without monitoring, spending can spiral out of control.
• Error rate: Distinguish "system errors" (timeouts and rate limits) from "business errors" (JSON parsing failures and rejection by safety rules).
• Tool success rate: Track the success rate of every tool. A sudden drop for one tool often means something upstream has changed.
• Human intervention rate / user retry rate: These are user behavior metrics. Repeated retries mean a poor experience, even when no error is reported.

Alerts should cover abnormal spikes in token costs (possibly abuse), sudden increases in p95 latency (possibly instability on the model side), and surging error rates. LLM application cost curves are less stable than those of traditional applications; a single bug can burn through a meaningful amount of money within a few hours.

4. Integrate All Three Layers into the Development Loop

Observability is not something to "add after launch"; it should be embedded in the development process:

During development: use traces to debug a single request → change the prompt/code
Before release: run the eval set → quantify whether it improved
After launch: monitor metrics → anomaly alerts → feed bad cases back into the eval set

The critical feedback loop is: production bad case → eval set → prevent regressions. When a user reports that "the answer is wrong," add that case to the evaluation set and label the expected result. The next change will automatically check it. This makes the eval set more representative over time and the system more stable with continued use.

5. Trade-offs Between Cost and Privacy

A complete trace records the full prompt and output, creating two risks:

1. Cost: Trace data itself can be enormous (long prompts × high QPS). Set a sampling rate, or retain complete data only for errors and slow requests.
2. Privacy: User input may contain PII or sensitive data. Redact phone numbers, email addresses, and secret patterns before persisting traces, or self-host the tracing service (self-hosted Langfuse keeps data inside your own data center).

Summary

All three layers are essential to LLM application observability:

• Trace: Identify "which stage failed" (Langfuse/LangSmith).
• Eval: Quantify "whether a change made things better" (RAGAS/promptfoo + an evaluation set).
• Metric: Monitor "whether production is healthy" (latency/tokens/cost/alerts).

With all three layers in place, plus the "production bad case → eval set" feedback loop, your AI application changes from "leaving things to chance after launch" into an engineering system that is "debuggable, regression-testable, and cost-controlled." The ROI is extremely high: most AI application quality problems are not caused by weak models, but by the lack of observability needed to discover and pinpoint them.