Sampling

Control which traces are recorded and exported in OpenTelemetry Rust with built-in and custom samplers

Sampling is a mechanism to control the noise and overhead introduced by OpenTelemetry by reducing the number of traces collected and sent to the backend.

Understanding Sampling

From opentelemetry-sdk/src/trace/sampler.rs:40-81, sampling can be implemented at different stages:

• Head sampling - Decision made when creating the span (what OpenTelemetry SDK provides)
• SDK layer - Implemented in span processors or exporters
• Out of process - In agents or collectors

Sampling Flags

Two properties control data collection:

1. is_recording() - If true, the span records events, attributes, and status. Span processors receive all recording spans.
2. Sampled flag - Set in SpanContext::trace_flags(). Indicates the span will be exported. Only sampled spans reach exporters.

A span can be recording but not sampled (is_recording=true, sampled=false). This allows local metrics without backend export.

The combination is_recording=false and sampled=true is not allowed by the OpenTelemetry API.

Sampling Decisions

From opentelemetry-sdk/src/trace/sampler.rs:22-33:

pub enum SamplingDecision {
    /// Span will not be recorded and all events and attributes will be dropped.
    Drop,

    /// Span data will be recorded, but not exported.
    RecordOnly,

    /// Span data will be recorded and exported.
    RecordAndSample,
}

Sampling Result

pub struct SamplingResult {
    /// The decision about whether or not to sample
    pub decision: SamplingDecision,

    /// Extra attributes to be added to the span by the sampler
    pub attributes: Vec<KeyValue>,

    /// Trace state from parent context, may be modified by samplers
    pub trace_state: TraceState,
}

Built-in Samplers

OpenTelemetry SDK provides several built-in samplers defined in opentelemetry-sdk/src/trace/sampler.rs:142-167:

pub enum Sampler {
    /// Always sample the trace
    AlwaysOn,

    /// Never sample the trace
    AlwaysOff,

    /// Respects the parent span's sampling decision or delegates to a sampler for root spans
    ParentBased(Box<dyn ShouldSample>),

    /// Sample a given fraction of traces (0.0 to 1.0)
    TraceIdRatioBased(f64),

    /// Jaeger remote sampler (requires feature flag)
    #[cfg(feature = "jaeger_remote_sampler")]
    JaegerRemote(JaegerRemoteSampler),
}

AlwaysOn Sampler

Samples every trace:

use opentelemetry_sdk::trace::{SdkTracerProvider, Sampler};

let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::AlwaysOn)
    .build();

AlwaysOff Sampler

Never samples any traces:

let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::AlwaysOff)
    .build();

TraceIdRatioBased Sampler

Samples a percentage of traces based on the trace ID:

// Sample 25% of traces
let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::TraceIdRatioBased(0.25))
    .build();

// Sample 100% (equivalent to AlwaysOn)
let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::TraceIdRatioBased(1.0))
    .build();

// Sample 0% (equivalent to AlwaysOff)
let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::TraceIdRatioBased(0.0))
    .build();

TraceIdRatioBased uses the trace ID for sampling decisions, ensuring all spans in a trace have the same sampling decision.

ParentBased Sampler

Respects the parent span’s sampling decision, delegating to another sampler for root spans:

// Use AlwaysOn for root spans, respect parent for child spans
let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::ParentBased(Box::new(Sampler::AlwaysOn)))
    .build();

// Use 10% sampling for root spans
let provider = SdkTracerProvider::builder()
    .with_sampler(Sampler::ParentBased(
        Box::new(Sampler::TraceIdRatioBased(0.1))
    ))
    .build();

From opentelemetry-sdk/src/trace/sampler.rs:211-236, the ParentBased sampler:

• If parent is sampled → child is sampled
• If parent is not sampled → child is not sampled
• If no parent → delegates to the provided sampler

Environment Variables

Configure sampling via environment variables (from opentelemetry-sdk/src/trace/config.rs:60-100):

# Always sample
export OTEL_TRACES_SAMPLER=always_on

# Never sample
export OTEL_TRACES_SAMPLER=always_off

# Sample 50% of traces
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.5

# Parent-based with AlwaysOn delegate
export OTEL_TRACES_SAMPLER=parentbased_always_on

# Parent-based with ratio delegate
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

Custom Samplers

Implement the ShouldSample trait for custom sampling logic:

use opentelemetry::{
    trace::{Link, SpanKind, TraceId},
    Context, KeyValue,
};
use opentelemetry_sdk::trace::{
    ShouldSample, SamplingResult, SamplingDecision,
};

#[derive(Debug, Clone)]
struct CustomSampler;

impl ShouldSample for CustomSampler {
    fn should_sample(
        &self,
        parent_context: Option<&Context>,
        trace_id: TraceId,
        name: &str,
        span_kind: &SpanKind,
        attributes: &[KeyValue],
        links: &[Link],
    ) -> SamplingResult {
        // Custom sampling logic here
        let decision = if name.starts_with("important_") {
            SamplingDecision::RecordAndSample
        } else {
            SamplingDecision::Drop
        };

        SamplingResult {
            decision,
            attributes: vec![],
            trace_state: parent_context
                .and_then(|cx| cx.span().span_context().trace_state().clone())
                .unwrap_or_default(),
        }
    }
}

Attribute-Based Sampling

Sample based on span attributes:

#[derive(Debug, Clone)]
struct AttributeSampler {
    sample_attribute: String,
    sample_value: String,
}

impl ShouldSample for AttributeSampler {
    fn should_sample(
        &self,
        parent_context: Option<&Context>,
        trace_id: TraceId,
        name: &str,
        span_kind: &SpanKind,
        attributes: &[KeyValue],
        links: &[Link],
    ) -> SamplingResult {
        let decision = attributes.iter()
            .any(|kv| {
                kv.key.as_str() == self.sample_attribute
                    && kv.value.as_str() == self.sample_value
            })
            .then_some(SamplingDecision::RecordAndSample)
            .unwrap_or(SamplingDecision::Drop);

        SamplingResult {
            decision,
            attributes: vec![],
            trace_state: Default::default(),
        }
    }
}

// Use the custom sampler
let sampler = AttributeSampler {
    sample_attribute: "user.tier".to_string(),
    sample_value: "premium".to_string(),
};

let provider = SdkTracerProvider::builder()
    .with_sampler(sampler)
    .build();

Rate-Limiting Sampler

Implement rate limiting for high-volume scenarios:

use std::sync::Mutex;
use std::time::{Duration, Instant};

#[derive(Debug)]
struct RateLimitingSampler {
    max_per_second: usize,
    state: Mutex<SamplerState>,
}

#[derive(Debug)]
struct SamplerState {
    count: usize,
    window_start: Instant,
}

impl RateLimitingSampler {
    fn new(max_per_second: usize) -> Self {
        Self {
            max_per_second,
            state: Mutex::new(SamplerState {
                count: 0,
                window_start: Instant::now(),
            }),
        }
    }
}

impl ShouldSample for RateLimitingSampler {
    fn should_sample(
        &self,
        parent_context: Option<&Context>,
        trace_id: TraceId,
        name: &str,
        span_kind: &SpanKind,
        attributes: &[KeyValue],
        links: &[Link],
    ) -> SamplingResult {
        let mut state = self.state.lock().unwrap();
        let now = Instant::now();

        // Reset counter every second
        if now.duration_since(state.window_start) >= Duration::from_secs(1) {
            state.count = 0;
            state.window_start = now;
        }

        let decision = if state.count < self.max_per_second {
            state.count += 1;
            SamplingDecision::RecordAndSample
        } else {
            SamplingDecision::Drop
        };

        SamplingResult {
            decision,
            attributes: vec![],
            trace_state: Default::default(),
        }
    }
}

Composite Samplers

Combine multiple sampling strategies:

#[derive(Debug)]
struct CompositeSampler {
    samplers: Vec<Box<dyn ShouldSample>>,
}

impl ShouldSample for CompositeSampler {
    fn should_sample(
        &self,
        parent_context: Option<&Context>,
        trace_id: TraceId,
        name: &str,
        span_kind: &SpanKind,
        attributes: &[KeyValue],
        links: &[Link],
    ) -> SamplingResult {
        // Sample if ANY sampler says to sample
        for sampler in &self.samplers {
            let result = sampler.should_sample(
                parent_context,
                trace_id,
                name,
                span_kind,
                attributes,
                links,
            );

            if result.decision == SamplingDecision::RecordAndSample {
                return result;
            }
        }

        SamplingResult {
            decision: SamplingDecision::Drop,
            attributes: vec![],
            trace_state: Default::default(),
        }
    }
}

Jaeger Remote Sampler

With the jaeger_remote_sampler feature, fetch sampling configuration from a remote service:

#[cfg(feature = "jaeger_remote_sampler")]
use opentelemetry_sdk::trace::Sampler;
use opentelemetry_sdk::runtime::Tokio;

#[cfg(feature = "jaeger_remote_sampler")]
let sampler = Sampler::jaeger_remote(
    Tokio,
    reqwest::Client::new(),
    Sampler::TraceIdRatioBased(0.1), // Default sampler
    "my-service",
)
.with_endpoint("http://localhost:5778")
.build()?;

let provider = SdkTracerProvider::builder()
    .with_sampler(sampler)
    .build();

Best Practices

Use ParentBased for distributed traces

Always use ParentBased in production to ensure consistent sampling decisions across service boundaries. This prevents partial traces.

Start with conservative sampling

Begin with a low sampling rate (e.g., 1-10%) and increase based on traffic volume and backend capacity.

Sample important operations at 100%

Use custom samplers to always sample critical operations like errors, slow requests, or specific user tiers.

Consider RecordOnly for metrics

Use RecordOnly decision to collect local metrics without backend export overhead.

Test sampling in development

Use AlwaysOn in development and testing environments to ensure instrumentation is working correctly.

Complete Example

use opentelemetry::global;
use opentelemetry_sdk::trace::{SdkTracerProvider, Sampler};
use opentelemetry_sdk::Resource;
use opentelemetry_otlp::SpanExporter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Configure a parent-based sampler with 10% ratio for root spans
    let sampler = Sampler::ParentBased(
        Box::new(Sampler::TraceIdRatioBased(0.1))
    );

    let exporter = SpanExporter::builder()
        .with_tonic()
        .build()?;

    let provider = SdkTracerProvider::builder()
        .with_resource(Resource::builder()
            .with_service_name("my-service")
            .build())
        .with_sampler(sampler)
        .with_batch_exporter(exporter)
        .build();

    global::set_tracer_provider(provider.clone());

    // Your application code here
    // Only ~10% of root traces will be sampled

    provider.shutdown()?;
    Ok(())
}

Next Steps

Span Processors

Configure how sampled spans are processed and exported

Overview

Return to tracing overview