Observability Architecture in 2026: OpenTelemetry, Continuous Profiling & Exemplars

Metrics (Prometheus/Datadog) ──────────────── Silo A
Logs    (Elasticsearch/Splunk) ─────────────── Silo B
Traces  (Jaeger/Zipkin) ────────────────────── Silo C

Before OTel:                         With OTel:
Jaeger SDK → Jaeger only             OTel SDK → OTel Collector → Any backend
Datadog SDK → Datadog only           One API → Vendor-neutral → Honeycomb/Grafana/Datadog/Jaeger
New Relic SDK → NR only              Switch backends by changing Collector config

# otel-collector-config.yaml
receivers:
  otlp:                            # Receive from apps using OTLP protocol
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:                           # Batch for efficiency
    timeout: 1s
    send_batch_size: 1024
  
  tail_sampling:                   # Sample only interesting traces (costly requests, errors)
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic-low
        type: probabilistic
        probabilistic: {sampling_percentage: 1}  # 1% of normal traces

exporters:
  otlp/tempo:
    endpoint: http://tempo:4317    # Traces → Grafana Tempo
  prometheusremotewrite:
    endpoint: http://mimir/api/v1/push  # Metrics → Grafana Mimir
  loki:
    endpoint: http://loki:3100    # Logs → Grafana Loki

service:
  pipelines:
    traces:   {receivers: [otlp], processors: [batch, tail_sampling], exporters: [otlp/tempo]}
    metrics:  {receivers: [otlp], processors: [batch], exporters: [prometheusremotewrite]}
    logs:     {receivers: [otlp], processors: [batch], exporters: [loki]}

# Auto-instrumentation (zero code changes — inject as agent):
# pip install opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-sqlalchemy
# OTEL_SERVICE_NAME=checkout-service opentelemetry-instrument python app.py
# → Automatically instruments HTTP, SQLAlchemy, Redis, httpx calls

# Manual instrumentation for custom business logic:
from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("checkout.service")

async def process_payment(order_id: str, amount: float):
    with tracer.start_as_current_span("process_payment") as span:
        # Add business context as span attributes:
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.currency", "GBP")
        
        try:
            result = await stripe_client.charge(amount)
            span.set_attribute("payment.stripe_id", result.id)
            return result
        except stripe.CardError as e:
            # Mark span as error with details:
            span.set_status(StatusCode.ERROR, str(e))
            span.record_exception(e)
            raise PaymentFailedException(str(e)) from e

Without Exemplars:
  Graph: p99 latency = 8.2 seconds (spike at 14:32)
  Action: Manually search Jaeger for traces around 14:32 → 10 minutes

With Exemplars:
  Graph: click the spike at 14:32
  → Tooltip shows "TraceID: abc123 (duration: 8.2s)"
  → Click TraceID → Jaeger opens that exact trace instantly
  Action: 10 seconds to root cause

# Prometheus exemplar with trace ID linkage:
from prometheus_client import Histogram
from opentelemetry import trace

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint', 'status']
)

def record_request(method, endpoint, status, duration):
    # Attach current trace ID as exemplar:
    current_span = trace.get_current_span()
    trace_id = format(current_span.get_span_context().trace_id, '032x')
    
    REQUEST_DURATION.labels(method, endpoint, status).observe(
        duration,
        exemplar={'traceID': trace_id}  # Links this data point to the trace
    )

# Prometheus SLO alerting (using pyrra or sloth):
# SLO: 99.9% of requests succeed in < 500ms over 30 days

# Error budget: 0.1% of 30-day requests = 43.2 minutes of allowed downtime

# Alert 1: Fast burn (critical — page immediately)
# If current burn rate would exhaust 5% of error budget in 1 hour:
- alert: HighErrorRateFastBurn
  expr: |
    (rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])) > 0.001 * 14.4
  # 14.4 = (5% of budget / 1 hour) / (1 hour / 30 days)
  severity: critical

# Alert 2: Slow burn (warning — needs same-day attention)
# If current burn rate would exhaust 10% of budget in 6 hours:
- alert: HighErrorRateSlowBurn
  expr: |
    (rate(http_requests_total{status=~"5.."}[1h]) / rate(http_requests_total[1h])) > 0.001 * 6
  severity: warning