LLM Observability in FastAPI: A Code-First Approach

Why Traditional Monitoring Fails for LLM Applications

Large Language Models (LLMs) introduce unique operational challenges. Unlike deterministic APIs, LLM responses vary based on prompts, context, and sampling parameters. Traditional monitoring tools focus on infrastructure metrics like latency and error rates, but these miss critical LLM-specific signals such as:

• Token consumption and cost
• Prompt construction latency
• Context window limits
• Response quality and hallucination risks

Without visibility into these factors, debugging becomes a guessing game. This article shows how to implement end-to-end observability for LLM-powered FastAPI applications using OpenTelemetry.

Core Principles of LLM Observability

LLM observability requires tracking the full request lifecycle:

1. User input and context retrieval
2. Prompt construction and model configuration
3. Model inference and token usage
4. Post-processing and quality evaluation

By instrumenting each stage with OpenTelemetry, you gain insights into:

• Why a model generated a specific response
• Which documents influenced the output
• Where latency occurred in the pipeline
• Whether responses passed safety checks

Implementing Observability in FastAPI

1. Instrumenting the Request Pipeline

Use OpenTelemetry to create trace spans for each logical stage:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@router.post("/query")
def handle_query(request: Request):

with tracer.start_as_current_span("query-processing"):

# Context retrieval

with tracer.start_as_current_span("retrieve-context"):

context = get_relevant_docs(request.input)

# Prompt construction

with tracer.start_as_current_span("build-prompt"):

prompt = construct_prompt(request.input, context)

# Model inference

with tracer.start_as_current_span("llm-inference"):

response = model.generate(prompt)

# Quality evaluation

with tracer.start_as_current_span("evaluate-response"):

validate_response(response)

return response

2. Capturing Critical Metadata

Add semantic attributes to spans for deeper analysis:

• model.name: “gpt-4”
• prompt.tokens: 256
• response.cost: $0.03
• retrieval.documents: [“doc123”, “doc456”]

3. Exporting and Visualizing Traces

Configure OpenTelemetry to export traces to any compatible backend:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="https://otel-collector:4317")
trace.set_exporter(exporter)

Supported backends include Jaeger, Grafana Tempo, and LLM-specific platforms like Phoenix.

Best Practices for LLM Observability

1. **Use Semantic Conventions**: Follow OpenTelemetry’s LLM-specific attributes for consistency.
2. **Track Cost Signals**: Record token usage and inference costs per request.
3. **Attach Evaluation Results**: Add quality checks directly to traces for root-cause analysis.
4. **Avoid Vendor Lock-In**: Use OpenTelemetry’s vendor-neutral format for flexibility.

Conclusion

LLM observability isn’t just about adding telemetry—it’s about building visibility into every decision point of your AI system. By implementing code-first instrumentation with OpenTelemetry, you gain precise control over how LLM interactions are observed and analyzed. This approach enables:

• Debugging hallucinations and low-quality responses
• Optimizing token usage and inference costs
• Identifying performance bottlenecks
• Ensuring compliance with safety standards

Start by instrumenting your FastAPI endpoints with trace spans, capturing metadata at each stage, and exporting to your preferred observability backend. With this foundation, you’ll transform LLM operations from a black box into a transparent, debuggable system.