Discord’s Distributed Tracing in Elixir: Zero Performance Penalty

Introduction

Scaling distributed systems while maintaining performance is a challenge even for tech giants. Discord engineers recently shared how they added distributed tracing to Elixir’s actor model without sacrificing speed or scalability. Their solution, a custom Transport library, addresses a critical gap in instrumenting actor-based architectures—proving that observability and performance can coexist.

Why Elixir’s Actor Model Needs Tracing

The Problem with Message-Passing Systems

Unlike HTTP-based microservices, Elixir’s actor model relies on direct message passing between processes. This design lacks a built-in metadata layer for trace context propagation. While OpenTelemetry works within individual services, it fails to track interactions between Elixir processes—leaving blind spots in complex systems like Discord’s chat infrastructure.

Discord’s Requirements

• Ergonomic adoption: Developers needed a seamless integration
• Flexibility: Support both raw messages and GenServer abstractions
• Zero-downtime deployment: Critical for a service with millions of users

The Transport Library Solution

Envelope Pattern for Trace Context

Discord’s breakthrough came in the form of a simple yet powerful Envelope struct:

defmodule Discord.Transport.Envelope do

defstruct [:message, trace_carrier: []]

def wrap_message(message) do

%__MODULE__{

message: message,

trace_carrier: :otel_propagator_text_map.inject([])

}

end
end

This wrapper adds trace metadata to every message without altering existing code. The library provides drop-in replacements for GenServer’s call and cast functions, ensuring compatibility during gradual migration.

Gradual Rollout Strategy

The team prioritized backward compatibility. Their handle_message function normalizes both old-style messages and new Envelope-wrapped ones. This approach allowed Discord to:

• Deploy tracing incrementally across services
• Avoid service restarts during migration
• Maintain full functionality during rollout

Performance Optimizations

Dynamic Sampling for Massive Fanout

Discord’s architecture faces unique challenges when messages are sent to large groups. Their solution: dynamic sampling based on recipient count:

• 1 recipient: 100% sampling
• 100 recipients: 10% sampling
• 10,000+ recipients: 0.1% sampling

This prevents trace data from overwhelming their observability systems while maintaining critical visibility.

Cost-Saving Optimizations

Two key optimizations reduced overhead:

1. Conditional Propagation: Only sampled traces include context, saving CPU cycles
2. Sampling Flag Filtering: gRPC requests now check sampling status before full deserialization

These changes dropped CPU usage in the sessions service from 55% to 45% during high-load scenarios.

Real-World Impact

During a recent incident, tracing revealed 16-minute delays in guild processes—information metrics and logs alone couldn’t provide. The detailed traces also exposed cascading failures that impacted user interactions.

Conclusion

Discord’s Transport library demonstrates how to add distributed tracing in Elixir without compromising performance. By wrapping message passing instead of retrofitting HTTP-style metadata, they preserved Elixir’s architectural strengths while gaining critical observability. For developers working with actor-based systems, this approach offers a pragmatic blueprint for balancing scalability and traceability.

Want to explore this solution in action? Check out Discord’s open-source Transport library and see how they’re pushing the boundaries of distributed systems engineering.