Observability: Logging, Metrics, and Tracing

← Back to Architecture Hub

Observability: Logging, Metrics, and Tracing

1. Metrics: The "Health" Dashboard

Metrics are numbers over time.

Tool: Prometheus (The standard).
Visualization: Grafana.
You track the "Golden Signals": Latency, Traffic, Errors, and Saturation (CPU).
The Rule: If a number goes above 90%, send an automated Alert to the developer's phone. This is how you find problems BEFORE the user calls you.

2. Structured Logging: Searching the Text

A log like "User 123 logged in" is useless in 2026.

You need Structured JSON Logging.
{"event": "login", "user": 123, "duration_ms": 50, "shard": "US-WEST"}
By using JSON, you can use tools like ElasticSearch or Loki to find EVERY user who had a slow login in the "US-WEST" region in 1 second.

3. Distributed Tracing: The Request Map

In a system where a single click calls 20 services, how do you find the "One Slow Service"?

Use OpenTelemetry and Jaeger.
Every request gets a Trace-ID.
You see a "Gantt Chart" of the whole request: "Gateway took 5ms -> Order Service took 2s -> Payment Service took 10ms." Aha! The problem is in the Order Service. Tracing turns days of debugging into seconds of clicking.

4. Sampling: The Cost of Vision

Storing every single log and trace for 1 billion users is Exceedingly Expensive.

In 2026, we use Sampling.
We only save 100% of the "Errors" and only 1% of the "Successes."
This gives you the visibility you need to fix bugs without spending your entire budget on storage.

Frequently Asked Questions

Is 'Monitoring' the same as 'Observability'? No. Monitoring is "Is the server on?" Observability is "WHY is the server slow?" Monitoring tells you there is a fire; Observability tells you exactly which wire caused the spark.

What is OpenTelemetry? It is a "Universal Language" for observability. In the past, every tool (Datadog, New Relic) had its own code. Today, you write your code using OpenTelemetry, and you can swap your dashboard provider in 5 minutes without changing a single line of your actual code.

Key Takeaway

Observability is the "Vision" of the architect. By mastering the Three Pillars and the discipline of Distributed Tracing, you gain the ability to manage thousands of servers with total confidence. You graduate from "Feeling the system" to "Seeing the Truth."

Part of the Software Architecture Hub — engineering the vision.