Tail Sampling for LLM and Agent Failures: Retaining Critical Traces Once Outcomes Are Known

A fixed 1% head-sampling rate is insufficient as the sole source of AI forensic evidence. Empty responses, structured-output failures, and tool loops have not yet surfaced as errors when the root span is created; if the request is discarded at ingress, the model, retrieval, and tool spans cannot be recovered after the task completes.

Distributed Tracing explains context propagation, while AI Agent Observability defines task steps. This article focuses on buffering a complete Trace first, then waiting for the outcome before deciding whether to export it.

First Ensure Every Trace Reaches the Same Decision-Maker

A tail sampler must see the complete Trace. Production deployments usually use a stateless receiver tier that applies consistent hashing on trace_id, routing the same Trace to a single sampler instance. Otherwise, each instance sees only fragments, causing incorrect decisions and potentially incomplete exports.

Target services must use AlwaysOn or propagate the sampled flag; their traces must not first be discarded by a low-rate head sampler. Static assets and health checks can be excluded up front so that only AI requests enter the dedicated pipeline.

Encode Both Technical Errors and Semantic Failures

The policy retains ERROR traces, slow Traces, and a small successful baseline. The root span also records a bounded enum such as ai.outcome=schema_invalid|tool_loop|grounding_failed|policy_blocked; do not make the sampler parse natural language.

processors:
  tail_sampling/ai:
    sampling_strategy: trace-complete
    # Example values must be determined from Trace duration and export-latency distributions.
    decision_wait: 90s
    # p99 arrival skew from the root span to the final batch of child spans, plus headroom.
    decision_wait_after_root_received: 5s
    num_traces: 180000
    expected_new_traces_per_sec: 1500
    decision_cache:
      sampled_cache_size: 1000000
      non_sampled_cache_size: 1000000
    policies:
      - name: technical-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-runs
        type: latency
        latency:
          threshold_ms: 12000
      - name: semantic-failures
        type: string_attribute
        string_attribute:
          key: ai.outcome
          values: [schema_invalid, tool_loop, grounding_failed]
      - name: success-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

With retain-on-any-match behavior, measure policy overlap. Use a finite set of risk tiers for high-value tenants; never put an unbounded set of tenant IDs into a policy. Tool failure classification can reuse the Tool Calling Failure Taxonomy.

Capacity Depends on the Wait Window and Trace Size

A fixed twenty-second delay may expire before the root span and ai.outcome arrive for a long-running Agent. Determine decision_wait from the p99 duration between the first span and root span, batch export latency, and child-span arrival skew, plus headroom; ninety seconds is only an example. trace-complete waits for the entire Trace. decision_wait_after_root_received preserves the empirically measured late-arrival window after the root span arrives while also allowing short Traces to be decided earlier.

The concurrent Trace count is approximately the number of new Traces per second multiplied by their residence time in seconds, plus burst headroom. The example can reach 135,000 Traces, and num_traces must also cover peak traffic and routing skew. Memory also depends on the number of spans and the size of events and attributes. decision_cache should be substantially larger than num_traces so that late spans inherit the original decision. Late data cannot change an existing decision, and enlarging the cache cannot compensate for a wait window that is too short.

After replaying the production distribution, size memory and configure memory_limiter and scaling thresholds. Monitor sampling_trace_dropped_too_early, sampling_trace_removal_age, sampling_late_span_age, decision latency, queues, rejections, decision-cache evictions, and RSS; exact metric names depend on the deployed version. Scaling a hash ring can split in-flight Traces, so proceed gradually and observe the system for two wait windows.

Validate with Ground-Truth Data

Construct successful, ERROR, slow-request, semantic-failure, and late-child-span cases. Assign each one a test ID, then verify the retained set, completeness, and decision latency in the backend. Replay de-identified traffic and calculate recall by class; targets might be 100% for severe errors, above 99% for semantic failures, approximately 2% for the successful baseline, and zero Collector rejections.

Tail-sampled Traces can enter a case library through Agent Replay Evaluation, but sampling attributes are only for selection and must not store the complete Prompt in tags. Content retention is governed by a separate privacy policy.

Roll out at 5%, 25%, and 100%. Each stage should cover one decision_wait plus the maximum task duration, while comparing error discovery rates, ingestion volume, and memory usage. Alerts should include the retention reason.

Fall Back to Predictable Sampling Under Overload

When memory exceeds 80%, evictions rise, or a queue approaches capacity, stop increasing the rollout and reserve protected capacity for a minimal tail-sampling lane. It still receives every span from the target AI Traces but retains only ERROR and a bounded set of ai.outcome values. The scope can be narrowed to critical services or risk tiers, but reduced coverage outside that scope must be recorded. Revert to the previous policy, remove the slow-Trace rule, or exclude noncritical workloads; do not keep increasing num_traces indefinitely or shorten the wait window until it ends before the outcome arrives.

The successful baseline can use a separate, quota-limited, parent-based probabilistic-sampling path that randomly retains 1% while the outcome is unknown. It cannot guarantee retention of ERROR traces because the head sampler cannot see the final status. If the minimal tail-sampling lane remains unstable, the only option is to narrow the protected scope and record the gap; head sampling must not be described as an equivalent substitute.

Keep the hash topology stable during rollback, and let old instances complete one decision window before exiting. Rollback is successful only when RSS falls, rejections stop, critical errors remain queryable, and ingestion returns within budget. Afterward, revise the capacity model using actual Trace sizes.

Tail sampling improves the visibility of critical failures within the same storage budget. It preserves reliable evidence only when routing is complete, outcomes are stable, capacity has been replay-tested, and degradation is predictable.

Further Reading

• Distributed Tracing
• AI Agent Observability
• Tool Calling Failure Taxonomy
• Agent Replay Evaluation