Signals

Understand the three types of telemetry signals in OpenTelemetry: traces for request flows, metrics for measurements, and logs for events.

Overview

OpenTelemetry defines three core telemetry signals that work together to provide comprehensive observability:

• Traces - Track request flows through distributed systems
• Metrics - Collect numerical measurements over time
• Logs - Capture discrete events and messages

Each signal serves a distinct purpose and is optimized for different use cases.

Traces

Traces track the progression of a single request as it flows through services in a distributed system. A trace is composed of one or more spans, which represent units of work.

Core Concepts

Spans

A span represents a single operation within a trace. Spans form a tree structure where parent spans can have multiple child spans.

use opentelemetry::{global, trace::{Span, Tracer}, KeyValue};

let tracer = global::tracer("my_service");

// Create a span
let mut span = tracer.start("database_query");
span.set_attribute(KeyValue::new("db.system", "postgresql"));
span.set_attribute(KeyValue::new("db.name", "customers"));

// Do work...

span.end();

Span Context

Every span has a context that includes:

• Trace ID - Unique identifier for the entire trace
• Span ID - Unique identifier for this span
• Trace Flags - Sampling and other flags
• Trace State - Vendor-specific context

use opentelemetry::trace::{SpanContext, TraceFlags, TraceId, SpanId};

let span_context = SpanContext::new(
    TraceId::from_hex("4bf92f3577b34da6a3ce929d0e0e4736").unwrap(),
    SpanId::from_hex("00f067aa0ba902b7").unwrap(),
    TraceFlags::SAMPLED,
    false,
    Default::default(),
);

Working with Spans

Active Span Management

OpenTelemetry provides utilities to manage the currently active span:

use opentelemetry::trace::{self, Tracer, get_active_span};

let tracer = global::tracer("my_tracer");

// Create and mark a span as active
let span = tracer.start("parent_span");
let active = trace::mark_span_as_active(span);

// Any span created here will be a child of `parent_span`

// Access the active span
trace::get_active_span(|span| {
    span.set_attribute(KeyValue::new("custom.attribute", "value"));
});

// Drop the guard to deactivate the span
drop(active);

Simplified Span Management

The in_span method provides a convenient way to create and manage spans:

use opentelemetry::{global, trace::Tracer};

let tracer = global::tracer("my_tracer");

tracer.in_span("parent_span", |cx| {
    // Spans created here will be children of `parent_span`
    
    tracer.in_span("child_span", |cx| {
        // Nested operation
    });
});

Async Spans

For async code, use FutureExt to propagate span context:

use opentelemetry::{Context, global, trace::{FutureExt, TraceContextExt, Tracer}};

async fn some_work() {
    // Async operations here
}

let tracer = global::tracer("my_tracer");
let span = tracer.start("my_span");

// Perform async work with this span as the currently active parent
some_work()
    .with_context(Context::current_with_span(span))
    .await;

Span Attributes, Events, and Links

Attributes

Attributes are key-value pairs that provide additional context:

span.set_attribute(KeyValue::new("http.method", "GET"));
span.set_attribute(KeyValue::new("http.url", "https://example.com/api"));
span.set_attribute(KeyValue::new("http.status_code", 200));

Events

Events represent significant points in time during a span’s lifetime:

use opentelemetry::trace::{Event, Span};

span.add_event(
    "cache_miss",
    vec![KeyValue::new("cache.key", "user:123")],
);

Links

Links associate a span with one or more other spans, useful for batch operations:

use opentelemetry::trace::{Link, SpanBuilder, Tracer};

let span = tracer
    .span_builder("batch_process")
    .with_links(vec![
        Link::with_context(span_context_1),
        Link::with_context(span_context_2),
    ])
    .start(&tracer);

Metrics

Metrics provide quantitative measurements about a service at runtime. Unlike traces which track individual requests, metrics aggregate data over time.

Instrument Types

OpenTelemetry provides several instrument types, each optimized for different measurement patterns.

Counter

Counters track values that only increase (e.g., requests served, errors):

use opentelemetry::{global, KeyValue};

let meter = global::meter("my_service");
let counter = meter.u64_counter("requests_total").build();

counter.add(1, &[
    KeyValue::new("method", "GET"),
    KeyValue::new("endpoint", "/api/users"),
]);

UpDownCounter

UpDownCounters track values that can increase or decrease (e.g., active connections, queue size):

let updown_counter = meter
    .i64_up_down_counter("active_connections")
    .build();

// Connection opened
updown_counter.add(1, &[KeyValue::new("pool", "primary")]);

// Connection closed
updown_counter.add(-1, &[KeyValue::new("pool", "primary")]);

Histogram

Histograms measure the distribution of values (e.g., request duration, payload size):

let histogram = meter
    .f64_histogram("request_duration_seconds")
    .with_description("HTTP request duration in seconds")
    .with_boundaries(vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0])
    .build();

histogram.record(0.0234, &[
    KeyValue::new("method", "GET"),
    KeyValue::new("status", "200"),
]);

Gauge

Gauges record independent measurements that represent the current state:

let gauge = meter
    .f64_gauge("cpu_temperature")
    .with_description("Current CPU temperature in Celsius")
    .build();

gauge.record(72.5, &[KeyValue::new("core", "0")]);

Observable Instruments

Observable instruments use callbacks to report values, ideal when the metric is calculated or managed elsewhere:

use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};

let queue_size = Arc::new(AtomicU64::new(0));
let queue_size_clone = queue_size.clone();

let _observable_gauge = meter
    .u64_observable_gauge("queue_size")
    .with_description("Current number of items in queue")
    .with_callback(move |observer| {
        let size = queue_size_clone.load(Ordering::Relaxed);
        observer.observe(size, &[]);
    })
    .build();

The callback is automatically invoked by the SDK before each export (typically every 60 seconds).

How Metrics Work

In OpenTelemetry, raw measurements are aggregated in memory before export:

1. Record - Measurements are recorded with instruments
2. Aggregate - Values are aggregated in memory (sum, count, min/max, histogram buckets)
3. Export - Aggregated metrics are periodically exported (e.g., every 60 seconds)

use opentelemetry_sdk::metrics::SdkMeterProvider;
use opentelemetry_sdk::Resource;

let exporter = opentelemetry_stdout::MetricExporterBuilder::default().build();

let provider = SdkMeterProvider::builder()
    .with_periodic_exporter(exporter) // Export every 60 seconds
    .with_resource(
        Resource::builder()
            .with_service_name("my_service")
            .build(),
    )
    .build();

Best Practices

Reuse Instruments: Create instruments once and reuse them. Avoid creating new instruments for each measurement.

// Good: Create once, reuse
let counter = meter.u64_counter("requests").build();
for request in requests {
    counter.add(1, &[]);
}

// Bad: Creating repeatedly
for request in requests {
    let counter = meter.u64_counter("requests").build(); // Don't do this!
    counter.add(1, &[]);
}

Clone for Sharing: Instruments are cheaply cloneable and can be shared across threads.

use std::sync::Arc;

let counter = meter.u64_counter("operations").build();

// Clone to share with another thread
let counter_clone = counter.clone();
std::thread::spawn(move || {
    counter_clone.add(1, &[]);
});

Logs

The OpenTelemetry Logs Bridge API provides integration between existing logging libraries and OpenTelemetry. It’s designed for logging library authors, not application developers.

Using Logs with Tracing

Application developers should use familiar logging libraries like tracing:

use tracing::{error, info, warn};
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_sdk::logs::SdkLoggerProvider;
use tracing_subscriber::prelude::*;

// Set up OpenTelemetry logging
let provider = SdkLoggerProvider::builder()
    .with_simple_exporter(opentelemetry_stdout::LogExporter::default())
    .build();

let otel_layer = OpenTelemetryTracingBridge::new(&provider);

tracing_subscriber::registry()
    .with(otel_layer)
    .init();

// Use normal logging
info!("Server started on port 8080");
warn!(user_id = 123, "Rate limit approaching");
error!(
    name: "database_error",
    error_code = "CONNECTION_FAILED",
    message = "Failed to connect to database"
);

Log Attributes

Logs can include structured attributes:

error!(
    name: "my-event-name",
    target: "my-system",
    event_id = 20,
    user_name = "otel",
    user_email = "[e-mail]",
    message = "This is an example message"
);

Instrumentation Scope

All signals can be associated with an instrumentation scope that identifies the library or component:

use opentelemetry::{global, InstrumentationScope};

let scope = InstrumentationScope::builder("my_library")
    .with_version(env!("CARGO_PKG_VERSION"))
    .with_schema_url("https://opentelemetry.io/schemas/1.17.0")
    .build();

let tracer = global::tracer_provider().tracer_with_scope(scope);

Next Steps

Resources

Learn about resource detection and service identification

Context Propagation

Understand how to propagate context across services