Log Processors

Configure batch and simple log processors for production and development

Overview

Log processors are responsible for receiving log records from loggers and forwarding them to exporters. OpenTelemetry SDK provides two built-in processors:

• SimpleLogProcessor: Exports logs immediately (for development/debugging)
• BatchLogProcessor: Buffers logs and exports in batches (for production)

+-----+---------------+   +-----------------------+   +-------------------+
|     |               |   |                       |   |                   |
| SDK | Logger.emit() +---> (Simple)LogProcessor  +--->  LogExporter      |
|     |               |   | (Batch)LogProcessor   +--->  (OTLPExporter)   |
+-----+---------------+   +-----------------------+   +-------------------+

LogProcessor Trait

All processors implement the LogProcessor trait:

pub trait LogProcessor: Send + Sync + Debug {
    /// Called when a log record is ready to be processed and exported
    fn emit(&self, record: &mut SdkLogRecord, scope: &InstrumentationScope);

    /// Force flush cached logs to the exporter
    fn force_flush(&self) -> OTelSdkResult;

    /// Shutdown the processor
    fn shutdown_with_timeout(&self, timeout: Duration) -> OTelSdkResult;

    /// Check if logging is enabled for given severity/target/name
    fn event_enabled(&self, level: Severity, target: &str, name: Option<&str>) -> bool {
        true // Default: all events enabled
    }

    /// Set the resource for the log processor
    fn set_resource(&mut self, resource: &Resource) {}
}

SimpleLogProcessor

The SimpleLogProcessor exports log records immediately when they are emitted. It’s synchronous and suitable for debugging and testing, but not recommended for production due to performance overhead.

Characteristics

• Synchronous export on each log emission
• No batching or buffering
• Immediate visibility of logs
• Higher overhead per log record
• Simpler error handling

Usage

use opentelemetry_sdk::logs::{SdkLoggerProvider, SimpleLogProcessor};
use opentelemetry_stdout::LogExporter;

let exporter = LogExporter::default();
let provider = SdkLoggerProvider::builder()
    .with_simple_exporter(exporter)
    .build();

Or manually:

use opentelemetry_sdk::logs::SimpleLogProcessor;

let exporter = LogExporter::default();
let processor = SimpleLogProcessor::new(exporter);

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

When to Use

Development & Debugging

Use SimpleLogProcessor when:

• Developing and debugging locally
• Running tests that need immediate log visibility
• Prototyping with small log volumes
• Using simple exporters like stdout

NOT for Production

Avoid in production because:

• Each log blocks the emitting thread
• High overhead for high-throughput scenarios
• Network latency affects application performance
• No batching efficiency

Runtime Compatibility

When using SimpleLogProcessor with different OTLP exporter clients:

Exporter Feature	Runtime Required	Emit From
grpc-tonic	Tokio runtime	Any thread
reqwest-blocking-client	No runtime needed	Non-tokio threads only
reqwest-client	Tokio runtime	Tokio runtime threads

Deadlock Risk: Using async exporters (e.g., with tokio::sleep) in SimpleLogProcessor can cause deadlocks if all runtime worker threads are blocked. Use BatchLogProcessor for async exporters.

BatchLogProcessor

The BatchLogProcessor buffers log records in memory and exports them in batches using a background thread. This is the recommended processor for production environments.

Characteristics

• Asynchronous export via background thread
• Configurable batching and scheduling
• Reduced export overhead
• Bounded memory usage
• Optimal for high-throughput scenarios

Basic Usage

use opentelemetry_sdk::logs::{BatchLogProcessor, SdkLoggerProvider};
use opentelemetry_stdout::LogExporter;

let exporter = LogExporter::default();
let processor = BatchLogProcessor::builder(exporter).build();

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

Configuration

Configure batch behavior using BatchConfigBuilder:

use opentelemetry_sdk::logs::{BatchLogProcessor, BatchConfigBuilder};
use std::time::Duration;

let exporter = LogExporter::default();

let processor = BatchLogProcessor::builder(exporter)
    .with_batch_config(
        BatchConfigBuilder::default()
            .with_max_queue_size(2048)           // Max buffered logs
            .with_max_export_batch_size(512)     // Max logs per export
            .with_scheduled_delay(Duration::from_secs(5))  // Export interval
            .build(),
    )
    .build();

let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

Configuration Parameters

Parameter	Default	Environment Variable	Description
max_queue_size	2048	OTEL_BLRP_MAX_QUEUE_SIZE	Maximum number of logs buffered in queue
max_export_batch_size	512	OTEL_BLRP_MAX_EXPORT_BATCH_SIZE	Maximum logs in a single export batch
scheduled_delay	1s	OTEL_BLRP_SCHEDULE_DELAY	Delay between consecutive exports

max_export_batch_size must be less than or equal to max_queue_size.

Export Triggers

Logs are exported when:

1. Batch size reached: When max_export_batch_size logs are buffered
2. Scheduled delay: Every scheduled_delay milliseconds
3. Force flush: When force_flush() is called
4. Shutdown: When shutdown() is called

Example: Production Configuration

use opentelemetry_sdk::logs::{BatchLogProcessor, BatchConfigBuilder};
use std::time::Duration;

let processor = BatchLogProcessor::builder(exporter)
    .with_batch_config(
        BatchConfigBuilder::default()
            .with_max_queue_size(4096)        // Higher capacity
            .with_max_export_batch_size(1024) // Larger batches
            .with_scheduled_delay(Duration::from_secs(10))  // Less frequent exports
            .build(),
    )
    .build();

Runtime Compatibility

When using BatchLogProcessor with OTLP exporters:

Exporter Feature	Runtime Required	Notes
grpc-tonic	Tokio runtime	LoggerProvider created in tokio runtime
reqwest-blocking-client	No runtime needed	Works with regular main or tokio::main
reqwest-client	Not supported	Use grpc-tonic or reqwest-blocking-client
hyper	Not supported	Use grpc-tonic or reqwest-blocking-client

Force Flush

Explicitly export buffered logs:

// Flush all buffered logs
provider.force_flush().unwrap();

This is useful before:

• Checkpointing application state
• Deploying new versions
• Handling critical errors

Shutdown

Always shutdown before application exit:

// Flush and shutdown
provider.shutdown().unwrap();

Tokio Deadlock: When using tokio’s current_thread runtime, call shutdown() from a separate thread or use spawn_blocking, as it’s a blocking call that can deadlock.

// Safe shutdown with current_thread runtime
tokio::task::spawn_blocking(move || {
    provider.shutdown().unwrap();
}).await.unwrap();

Dropped Logs

If the queue is full, logs are dropped and counted. The count is logged at shutdown:

// If queue is full, logs are dropped
// Increase max_queue_size or export more frequently

Monitor dropped logs and adjust configuration as needed.

Custom Log Processors

You can implement custom processors for specialized behavior:

use opentelemetry::logs::Severity;
use opentelemetry::InstrumentationScope;
use opentelemetry_sdk::error::OTelSdkResult;
use opentelemetry_sdk::logs::{LogProcessor, SdkLogRecord};
use opentelemetry_sdk::Resource;
use std::time::Duration;

#[derive(Debug)]
struct FilteringProcessor<P: LogProcessor> {
    delegate: P,
    min_severity: Severity,
}

impl<P: LogProcessor> LogProcessor for FilteringProcessor<P> {
    fn emit(&self, record: &mut SdkLogRecord, scope: &InstrumentationScope) {
        // Only forward logs at or above min_severity
        if record.severity_number().unwrap_or(Severity::Info) >= self.min_severity {
            self.delegate.emit(record, scope);
        }
    }

    fn force_flush(&self) -> OTelSdkResult {
        self.delegate.force_flush()
    }

    fn shutdown_with_timeout(&self, timeout: Duration) -> OTelSdkResult {
        self.delegate.shutdown_with_timeout(timeout)
    }

    fn event_enabled(&self, level: Severity, target: &str, name: Option<&str>) -> bool {
        level >= self.min_severity && self.delegate.event_enabled(level, target, name)
    }

    fn set_resource(&mut self, resource: &Resource) {
        self.delegate.set_resource(resource);
    }
}

Usage:

use opentelemetry_sdk::logs::SimpleLogProcessor;

let simple = SimpleLogProcessor::new(exporter);
let filtering = FilteringProcessor {
    delegate: simple,
    min_severity: Severity::Warn,
};

let provider = SdkLoggerProvider::builder()
    .with_log_processor(filtering)
    .build();

event_enabled Optimization

The event_enabled() method allows early filtering to skip expensive logging operations:

if !processor.event_enabled(Severity::Debug, "my-module", Some("my-event")) {
    return; // Skip expensive work
}

// Only execute if event is enabled
let expensive_data = compute_expensive_data();
log_record.add_attribute("data", expensive_data);

Processor Comparison

Feature	SimpleLogProcessor	BatchLogProcessor
Export Mode	Synchronous	Asynchronous (background)
Buffering	None	Yes
Overhead	High (per log)	Low (amortized)
Latency	None	Configurable
Memory Usage	Minimal	Bounded by queue size
Production Ready	No	Yes
Best For	Development, debugging	Production
Dropped Logs	Never	If queue full
Background Thread	No	Yes

Best Practices

Use BatchLogProcessor in Production

Always use BatchLogProcessor for production deployments:

• Lower overhead
• Better performance under load
• Configurable batching behavior

Configure Based on Load

Adjust batch configuration based on your log volume:

// High-volume service
.with_max_queue_size(8192)
.with_max_export_batch_size(2048)
.with_scheduled_delay(Duration::from_secs(2))

// Low-volume service
.with_max_queue_size(512)
.with_max_export_batch_size(128)
.with_scheduled_delay(Duration::from_secs(30))

Always Shutdown Cleanly

Call shutdown() before application exit to flush remaining logs:

// Register shutdown hook
let provider = SdkLoggerProvider::builder()
    .with_log_processor(processor)
    .build();

// On exit
provider.shutdown().unwrap();

Monitor Dropped Logs

Check for dropped log messages in production and adjust max_queue_size if needed.

Use event_enabled for Performance

Implement event_enabled() in custom processors to skip expensive operations for filtered logs.

Troubleshooting

Logs not exported

1. Forgot to shutdown: Always call provider.shutdown()

   provider.shutdown().unwrap();
   

2. Batch not full: Wait for scheduled_delay or call force_flush()

   provider.force_flush().unwrap();
   

3. Queue full: Increase max_queue_size or export more frequently

High latency

If using SimpleLogProcessor in production, switch to BatchLogProcessor:

let processor = BatchLogProcessor::builder(exporter).build();

Memory usage too high

Reduce max_queue_size or increase export frequency:

.with_max_queue_size(1024)  // Smaller buffer
.with_scheduled_delay(Duration::from_secs(1))  // More frequent exports

Logs dropped

Increase queue size or export more frequently:

.with_max_queue_size(8192)  // Larger buffer
.with_scheduled_delay(Duration::from_millis(500))  // More frequent

See Also

log Appender

Bridge logs from the log crate

tracing Appender

Bridge logs from the tracing crate

Bridge API

Understanding the core API

Overview

Return to logs overview