The specific engineering discipline that turns "our distributed system is a black box that occasionally breaks and we don\'t know why" into "we can query any signal across any service, correlate metrics/traces/logs in one place, and pinpoint root cause in minutes." Three primary signals: Metrics (Prometheus/VictoriaMetrics/Mimir — numeric time series, aggregated, low cost, ideal for SLOs and dashboards), Distributed traces (OpenTelemetry/Jaeger/Tempo — per-request span reconstruction across services, ideal for debugging distributed flows), and Logs (Loki/Elasticsearch/Splunk — structured events with full context, ideal for forensics and audit). Plus emerging: continuous profiling (Pyroscope, Parca, eBPF). Understanding OpenTelemetry as the vendor-neutral standard, cardinality management for metrics, sampling strategies for traces, SLO discipline with error budgets, and symptom-based alerting — the composed observability stack for modern production systems.
Observability is not "monitoring plus dashboards" — it\'s the specific engineering discipline for understanding what a distributed system is doing when things go wrong, at scale where no human can hold the full picture in their head. Consider concretely what modern production looks like. A single user request hits a load balancer, routes through an API gateway, calls the recommendation service, which calls the user profile service, the inventory service, and the ML ranking service. Each service is 5-100 pods. Each pod is running some framework generating structured events. Latency for that one user request could be dominated by any hop — the DB query in inventory, the model inference in ranking, the cache miss in user profile, network jitter to a cross-AZ replica. When latency spikes at 3 AM, the on-call engineer has minutes to figure out where. The naive approach — SSH into machines, tail -f on log files, run nagios checks, page a human when CPU exceeds 80% — collapses at scale. Modern observability replaces this with three composed signals: (a) metrics (Prometheus, VictoriaMetrics, Mimir, Thanos) — numeric time series with labels, aggregated at scrape time, cheap to store long-term, ideal for dashboards and SLO tracking; (b) distributed traces (OpenTelemetry, Jaeger, Tempo, Zipkin) — reconstruction of individual request flows as trees of spans across services, ideal for debugging where a slow request went; (c) logs (Loki, Elasticsearch/OpenSearch, Splunk) — structured event records with full context, ideal for forensic investigation and audit. Plus emerging (a 4th signal): continuous profiling (Pyroscope, Parca, eBPF-based) — CPU flame graphs continuously sampled in production, ideal for performance regressions. Understanding these signals — their tradeoffs, their composition via OpenTelemetry, the SLO/SLI discipline that makes them actionable — is Expert-tier competence for modern production engineering.
http_requests_total{service="checkout",status="500"} → count over time). Aggregated at scrape time (Prometheus pull model, typically 15s scrape interval) — individual events are counted/summed/bucketed before storage, not stored per-event. Cheap: a service producing 10K req/sec becomes 10K/s in a counter, stored as one point per scrape interval (~1 point / 15s). Perfect for dashboards, alerts, SLO tracking. Foundation of the RED method (Rate, Errors, Duration for services), USE method (Utilization, Saturation, Errors for resources), and Google\'s Four Golden Signals (Latency, Traffic, Errors, Saturation). Limitations: high-cardinality labels (user_id, request_id) explode series count — Prometheus struggles at >1M active series, causing OOM. Can tell you WHAT is happening (error rate up 3×) but not WHY (which request, which code path). No per-request context. Distributed traces: reconstruction of individual request flow as a tree of spans across services. Trace = collection of spans sharing a trace_id (propagated via W3C traceparent header). Span = one operation (HTTP call, DB query, function call) with start/end times, attributes (http.method, db.statement), events (log messages), and parent span reference. When request enters system: root span created with new trace_id. When service calls another: trace context injected into outbound headers; receiving service extracts and creates child span. Result: complete tree of what happened for that request across every service touched. Ideal for answering "where did the slow request spend time?" Historical: Google Dapper 2010 → Zipkin (Twitter 2012) → Jaeger (Uber 2016) → OpenTelemetry (2019 merger of OpenTracing + OpenCensus, CNCF de facto standard by 2023). Limitations: 100% sampling is prohibitive at scale (10K req/sec × 20 spans/req × 500 bytes = 100 MB/s of trace data per service); need sampling strategies (head-based at ingress or tail-based buffer-and-decide with bias toward errors/slow). Logs: structured event records with full context. Modern practice: JSON structured (not plain text — unstructured logs are grep-able but not query-able). Each log line has timestamp, level (DEBUG/INFO/WARN/ERROR), message, and structured attributes (user_id, request_id, error_code, etc.). Ideal for forensic investigation (what exactly did this user do?), audit trail (compliance), business events, and detailed debugging when metrics/traces are insufficient. Stack: Loki (Grafana Labs 2019, indexed labels + raw log bodies stored cheaply in object storage — inspired by Prometheus, "like Prometheus but for logs"), Elasticsearch/OpenSearch (full-text search + structured fields, expensive at scale), Splunk (enterprise incumbent, powerful but expensive). Limitations: log volume explodes without discipline — a chatty service producing 1000 log lines/sec × 500 bytes × 1000 pods = 500 MB/s of log data per service. Need sampling, retention policies, filtering. OTel logs signal GA in 2024. The Expert insight: no single signal is sufficient. Metrics tell you something is wrong. Traces tell you where. Logs tell you why. Together, correlated via OpenTelemetry, they form the modern observability stack. Standard modern discipline.The specific engineering task M.66 addresses is understanding how to compose the three signals for effective observability at scale, using OpenTelemetry as the vendor-neutral instrumentation standard and SLO/SLI discipline as the operational framework. Modern observability has four primary primitives: (a) OpenTelemetry (OTel) — the instrumentation standard. Launched 2019 as merger of OpenTracing (community, tracing-focused) + OpenCensus (Google, metrics + tracing). CNCF incubating 2021, de facto standard by 2023. Provides: SDKs per language (Java, Python, Go, JavaScript, .NET, Rust, etc.); Collector (agent + gateway pattern receiving, processing, exporting to any backend); Semantic conventions (standardized attribute names — http.method, db.system, rpc.service — so signals from any service are queryable consistently); auto-instrumentation for popular frameworks (Spring, Django, Express, etc.). Signal types: metrics, logs, traces, profiles (beta), events. Standard modern instrumentation choice. (b) Three signal pipelines composed. Metrics: OTel SDK → Collector → Prometheus (short-term) → VictoriaMetrics/Mimir/Thanos (long-term). Traces: OTel SDK → Collector (with tail sampling) → Jaeger/Tempo/Zipkin (storage + UI). Logs: application logging library → Collector or Loki/Fluent Bit → Loki/Elasticsearch/Splunk. All three linked via trace_id + span_id in log context for correlation. (c) SLO/SLI discipline. SLI = Service Level Indicator (measurement, e.g., "fraction of requests completing under 200ms"). SLO = Service Level Objective (target, e.g., "99.5% of requests under 200ms"). SLA = Service Level Agreement (external commitment). Error budget = 100% - SLO (e.g., 0.5% for 99.5% SLO). Burn-rate alerts fire when error budget consumed too fast (e.g., 14.4× burn over 1 hour + 6× burn over 6 hours → high severity page). Multi-window multi-burn-rate pattern from Google SRE book. Standard modern operational discipline. (d) Symptom-based vs cause-based alerting. Symptom-based: alert on user-visible issues (SLO burn, error rate p99 spike, checkout latency > 1s). Cause-based: alert on internal metrics (CPU > 80%, disk > 90%, memory > 85%). Modern practice: symptom-based for pages (someone must wake up), cause-based for warnings (informational, next business day). Alert fatigue is real problem: teams paged 20+ times per week burn out and start ignoring pages. Standard modern discipline.
http.request.method; New Relic uses http.method; Splunk expects httpMethod. Cross-vendor queries impossible. New service = pick which vendor to instrument for. Fragmentation. (c) COSTS EXPLODE AT SCALE — Datadog metrics pricing is per-custom-metric; at millions of series, monthly bills reach hundreds of thousands to millions of dollars. Splunk ingest pricing per-GB; a chatty service can cost more than its own compute. (d) NO SIGNAL PORTABILITY — a service might export to Datadog for metrics but Splunk for logs; no unified view. Sending same data to multiple backends means duplicate instrumentation. Standard failure of pre-OTel era. Common in enterprises with mature but siloed observability stacks.// FAIL MODE: vendor lock-in · inconsistent conventions · costs explode · no portabilityhttp.request.method, db.system, service.name are identical across every service. Standard modern instrumentation. (b) OTel Collector: deployed as DaemonSet (agent per node, receives from local pods) + Deployment (gateway, regional aggregation). Processes signals: batching, retries, filtering, tail sampling for traces, redaction of PII. Exports to any backend via receivers/processors/exporters pipeline. Vendor-neutral. (c) Backend composition: Prometheus (short-term metrics, 15-day retention) + VictoriaMetrics/Mimir (long-term, 2-year retention); Jaeger/Tempo (traces, 7-day retention); Loki/Elasticsearch (logs, 30-day retention). All correlated via trace_id + span_id + service.name. Grafana as unified query UI across all three. (d) SLO discipline: define SLIs per user journey (checkout latency p99 under 500ms, checkout success rate > 99.5%). Set SLOs. Compute error budgets (0.5% for 99.5% success). Configure multi-window multi-burn-rate alerts: page on 14.4× burn/1h + 6× burn/6h (fast burn threatening budget); warn on 3× burn/6h + 1× burn/24h (slow burn worth attention). Standard Google SRE pattern. (e) Symptom-based alerting: pages only fire when user-visible SLO threatened. CPU/memory/disk are informational warnings, not pages. On-call receives 1-3 pages/week instead of 30. Signal-to-noise ratio high. Alert fatigue eliminated. (f) Result: incident response < 15 minutes to root cause typically. Correlation of trace_id from failed request → logs at exact context → metrics showing when problem started. Vendor portability: swap Prometheus for VictoriaMetrics without touching application code. Cost control: tail sampling reduces trace volume 99%+ while keeping all error/slow traces.// FIT: OTel + composed backend + SLO + symptom alerts · <15min MTTR · vendor-neutralEach earlier attempt fails specifically. SSH + Nagios doesn\'t scale past 10 machines. Single-signal can\'t answer WHY. Vendor-locked explodes costs and loses portability. The Expert pattern: OpenTelemetry SDK for vendor-neutral instrumentation with consistent semantic conventions. OTel Collector for signal processing + tail sampling + export flexibility. Composed backend: Prometheus/VictoriaMetrics for metrics, Jaeger/Tempo for traces, Loki/Elasticsearch for logs — all correlated via trace_id. SLO discipline with error budgets. Multi-window multi-burn-rate alerts. Symptom-based pages, cause-based warnings. Continuous profiling (Pyroscope/Parca) as 4th signal. Standard modern observability. §02 covers the three pillars in depth (metrics cardinality, trace sampling, structured logs). §03 covers OpenTelemetry, SLO/SLI discipline, alerting patterns.
The historical arc of observability traces specifically how each primitive matured to solve problems that dominated at the time. 2000s: Nagios era. Individual machines, threshold-based alerts on CPU/memory/disk, plain-text log files. Standard early monitoring. Doesn\'t scale past 10-50 machines. 2010: Google Dapper paper. "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Sigelman et al.) — the foundational paper on distributed tracing. Introduces span/trace concepts, sampling strategies, header-based context propagation. Influences all subsequent tracing systems. 2012: Prometheus at SoundCloud. Metrics system with labels + PromQL query language. Pull-based scrape model. Google-style monitoring philosophy. Joined CNCF 2016 as second graduated project (after Kubernetes). 2012-2013: Zipkin at Twitter. Open-source implementation of Dapper concepts. First widely-used distributed tracing system. 2013-2014: ELK stack matures. Elasticsearch (search) + Logstash (ingestion) + Kibana (visualization). De facto log aggregation stack. Powerful but resource-heavy. 2014: Grafana released. Visualization layer initially for Graphite/Prometheus. Rapidly becomes standard dashboard tool across all signals. 2016: Jaeger at Uber. Production-quality tracing at scale (Uber traces 100K+ RPS). Adopted by CNCF 2017, graduated 2019. 2017-2018: OpenTracing + OpenCensus competing. OpenTracing (CNCF, community-driven, tracing API only). OpenCensus (Google, metrics + tracing). Fragmented instrumentation landscape. 2019: OpenTelemetry launches. Merger of OpenTracing + OpenCensus. Unified API for all signals. CNCF incubating 2021. 2020: Grafana Loki. "Prometheus but for logs" — index labels only (like Prometheus), store raw log bodies in cheap object storage (S3). Dramatically cheaper than Elasticsearch for high-volume logs. Adopted rapidly. 2021-2022: OTel matures. Traces GA 2021, metrics GA 2022. Auto-instrumentation for major frameworks. Collector processor library grows. 2022: Continuous profiling matures. Pyroscope (2020, acquired by Grafana Labs 2023) and Parca (2021, Polar Signals) offer eBPF-based zero-instrumentation CPU profiling in production. Standard 4th signal by 2024. 2023: OTel de facto standard. Datadog, New Relic, Honeycomb, Splunk, Dynatrace all offer OTel receivers. Cloud providers (AWS, GCP, Azure) native OTel support. Migration path away from vendor SDKs. 2024: OTel logs GA + eBPF observability rises. Logs signal reaches GA. eBPF-based tools (Cilium, Pixie, Parca) offer zero-instrumentation network + performance observability. 2025: OTel Profiles beta + native histograms mature. Continuous profiling becomes 5th official signal type in OTel. Prometheus native histograms (sparse buckets, higher resolution) become production-standard replacement for classic histograms. The arc explains why modern observability is composed OTel + Prometheus + Jaeger/Tempo + Loki + Pyroscope with SLO discipline — each primitive matured to solve the specific bottleneck that dominated at that time, and their composition produces the current state-of-art.
The three signals differ fundamentally in shape, cost, and query patterns — understanding these differences is what makes composition effective. Consider concretely what each signal costs and constrains. Metrics are (name, labels, value, timestamp) tuples, aggregated at scrape time. A counter http_requests_total{service="checkout",method="POST",status="200"} is one time series; the value increments as requests flow. Storage cost per active series: ~4 bytes per sample × ~5760 samples/day (15s scrape) = 23KB/day/series. 1M active series = 23GB/day = ~700GB/month. Prometheus struggles above ~1M active series (memory pressure, slow queries); VictoriaMetrics/Mimir/Thanos scale to 10M+ via horizontal sharding + block storage. Cardinality is the killer: adding a user_id label to a service serving 10M users creates 10M time series — instant OOM. The rule: labels should be low-cardinality (service, endpoint, status_code — bounded); high-cardinality attributes (user_id, request_id, session_id) belong in traces/logs, not metric labels. Standard modern discipline. Distributed traces are trees of spans identified by trace_id. Each span: ~500 bytes (name, service, timing, ~10 attributes, ~2 events). A single user request touching 20 services with ~5 spans each = 100 spans × 500 bytes = 50KB per traced request. 10K RPS × 100 spans = 1M spans/sec = 500 MB/s of trace data per service — prohibitive at 100% sampling. Sampling is the specific management primitive. Head-based sampling (decide at request start, e.g., sample 1% deterministically by trace_id hash) is cheap but misses rare bugs. Tail-based sampling (buffer all spans of a trace until complete, then decide based on total latency, error status, or specific attributes) captures errors + slow requests reliably at the cost of memory + latency at the collector. Standard modern approach: tail sampling with rules like "keep 100% of traces with error, 100% of traces with duration > p99, 1% of others." Reduces trace volume by 99% while keeping all diagnostic value. Logs are structured event records: {timestamp, level, message, service, trace_id, user_id, ...}. Each log line: ~500 bytes typical. A service producing 1000 lines/sec × 500 bytes × 1000 pods = 500 MB/s. Elasticsearch cost: ~$3-5/GB indexed (full-text index expensive). Loki cost: labels indexed (small), log bodies in S3 (~$0.023/GB stored). A production year: Elasticsearch $30-100K, Loki $2-8K for same volume. Loki tradeoff: log body queries scan (slow), while label filters fast. Both handle 100K events/sec+. Standard modern discipline: use Loki for volume-heavy application logs, Elasticsearch for search-heavy (compliance/security), Splunk for enterprise requirements. All emit via OpenTelemetry Collector.
http_requests_total{service="checkout",status="500"} = 47. Cardinality-sensitive: 2-3 labels (bounded values like status_code, method, service) generates a handful of time series per service. Adding user_id as a label would explode cardinality (10M users × existing labels = 10M+ series → Prometheus OOM). Metrics show that error rate in checkout is 47/12847 = 0.37% — enough to know something is wrong; not enough to know why. Standard modern discipline: keep labels bounded; put high-cardinality attributes in traces/logs. Prometheus at ~1M series capacity; VictoriaMetrics/Mimir horizontal-shard to 10M+. Distributed trace: per-request span tree. Root span (POST /checkout at api-gateway, 847ms total) has child spans (auth.validate 12ms, inventory.check 38ms, payment.process 782ms) which have their own children (db.query SELECT payment_methods in postgres, 761ms). Immediately shows: 761ms of the 847ms total was the postgres query in payment-service. Root cause visible in one glance. Trace context (trace_id abc123, span_id def456) propagates across service boundaries via W3C traceparent header. Each span: ~500 bytes with name, timing, service, attributes, events. A traced request with 20 services × 5 spans each = 100 spans × 500 bytes = 50KB. Cost: 10K RPS × 100 spans = 1M spans/sec = 500 MB/s per service — prohibitive at 100% sampling. Tail sampling (buffer all spans of trace, keep if error/slow/interesting, drop others) reduces volume 99%+ while keeping diagnostic value. Structured log: JSON event record with full context. {"timestamp":"2025-11-15T03:47:12.847Z", "level":"ERROR", "service":"payment-service", "trace_id":"abc123", "span_id":"def456", "message":"postgres slow query", "db.statement":"SELECT * FROM payment_methods WHERE user_id=?", "duration_ms":761}. Queryable: {service="payment-service"} |= "slow query" in Loki. Correlatable: trace_id links to the trace above; span_id links to the specific span. Log gives full context: the exact SQL statement that was slow. Cost: ~500 bytes per line. A service producing 1000 lines/sec × 500 bytes × 1000 pods = 500 MB/s. Loki stores raw bodies in S3 (~$0.023/GB); Elasticsearch fully indexes (~$3-5/GB effective). Loki 10-30× cheaper for volume-heavy logs. The Expert insight: no signal alone tells the full story. Metric says "checkout error rate 0.37%." Trace says "payment.process spent 761ms in postgres." Log says "SELECT * FROM payment_methods WHERE user_id=? took 761ms" — exact SQL for the DBA to add an index. Together, MTTR drops from hours to minutes. Correlation via OpenTelemetry (trace_id in every log line, span_id references) is the standard modern pattern. Each signal has its own scale/cost tradeoff — metric labels bounded, traces sampled, logs structured — and composition is Expert-tier discipline.Each unique label combination = one time series. Bounded labels (service, method, status) OK. High-cardinality (user_id, request_id) explodes. Prometheus OOM at ~1M series. Use histograms/exemplars for per-request detail.
Prometheus query language: rate(http_requests_total[5m]). Native histograms (sparse buckets, 2024+): higher resolution than classic buckets at similar storage cost. Standard modern discipline.
W3C traceparent header (00-{trace_id}-{span_id}-{flags}) injected into outbound HTTP/gRPC calls. Receiving service extracts, creates child span. Auto-instrumented by OTel SDK. Standard 2020+ pattern.
Collector buffers all spans of a trace; after trace completes, decides to keep or drop based on rules (keep errors, slow requests, specific attributes). Reduces volume 99%+ while keeping diagnostic traces. Standard at scale.
JSON format with trace_id, service.name, semantic conventions. Not plain text. Queryable via LogQL (Loki) or Elasticsearch DSL. Correlatable to traces via trace_id. Standard modern discipline; unstructured logs are 2005-era.
Loki: labels indexed (small), bodies in S3 (cheap). Fast label queries, slow body scans. 10-30× cheaper than Elasticsearch. Elasticsearch: full-text index (expensive but powerful). Choose by volume + query pattern.
The metric cardinality math (mech items i-ii) is worth walking through explicitly because it dictates every metric design decision. Consider concretely: Prometheus stores time series as (name + labels) → sequence of (timestamp, value) samples. Each active series occupies ~3KB of memory index + ~1-4 bytes per sample on disk. At 15s scrape interval: 5760 samples/day × 2 bytes average = 11KB/day/series, plus index. A service exposing http_requests_total with labels {method, endpoint, status_code} for 100 endpoints × 5 methods × 10 status codes = 5000 series — trivially cheap. Now add user_id label with 10M users: 5000 × 10M = 50B series — Prometheus OOM in seconds. This is the cardinality trap. Modern discipline: labels are attributes of the aggregate, not the individual request. Use histograms (http_request_duration_seconds_bucket{le="0.5"}) for latency distribution — a fixed number of buckets across bounded labels. For per-request detail: exemplars (link from histogram bucket to a sample trace_id) let you jump from aggregate metric to specific example trace. Prometheus native histograms (Prometheus 2.40+, 2022): sparse bucket representation with better resolution than classic buckets at similar storage cost — standard modern replacement. For very-high-cardinality workloads: use tracing (per-request) or logs (per-event) instead of metrics. VictoriaMetrics/Mimir/Thanos: horizontally-sharded Prometheus-compatible storage, scale to 10M+ active series via cluster + object storage. Standard modern architecture for large deployments. The trace sampling math (mech item iv): consider Uber-scale serving 100K RPS with 20-service architecture, ~10 spans per service per request = 200 spans/request × 100K = 20M spans/sec. At 500 bytes/span: 10 GB/s of trace data ingested. 100% sampling: prohibitive storage + processing cost, no marginal value (99% of traces are boring successful requests). Head-based sampling (decide at ingress, hash trace_id, keep 1%): cheap but misses rare errors + slow requests preferentially. Tail-based sampling: OTel Collector buffers all spans of a trace (memory cost, ~5s buffer window); after trace completes, evaluates rules — keep 100% of error traces, 100% of traces with duration > 500ms, 5% of others. Result: keep ~10-20% of traces (all diagnostic ones + representative sample of healthy) at 10-20% of storage cost while retaining full debugging value. Standard modern approach for tracing at scale. Understanding these numbers — cardinality costs, sampling ratios, storage economics — is Expert-tier competence.
The structured logging discipline (mech items v-vi) deserves specific attention because it\'s where most teams have technical debt. Consider concretely: legacy application logs are printf-style plain text — [2025-11-15 03:47:12] ERROR: payment failed for user 47129, order 8371, took 761ms. Grep-able but not query-able. To find "all payment failures for user 47129 in the last 24 hours" requires scanning every log line across every pod. Modern structured logging: JSON — {"timestamp":"2025-11-15T03:47:12.847Z", "level":"ERROR", "service":"payment-service", "trace_id":"abc123", "user_id":"47129", "order_id":"8371", "duration_ms":761, "message":"payment failed"}. Same information; now queryable: {service="payment-service", level="ERROR"} | json | user_id="47129" in LogQL. Correlatable: trace_id links to distributed trace showing the full request context; user_id enables user-scoped forensics. Modern logging libraries (structlog for Python, zap for Go, logback for Java) emit JSON natively. OpenTelemetry semantic conventions specify standard attribute names (service.name, trace_id, span_id, http.request.method, db.statement) so queries work identically across services. Loki vs Elasticsearch choice: Loki (Grafana Labs 2019, "like Prometheus but for logs") indexes labels only (bounded, low-cardinality) and stores raw log bodies compressed in S3-compatible object storage. Cost: ~$0.023/GB stored. Query pattern: fast label filter ({service="payment"}) then scan log bodies (slower). Excellent for volume-heavy application logs where you know the service/label but want to search within. Elasticsearch/OpenSearch: full inverted index on log body content. Cost: ~$3-5/GB effective (index inflates storage 2-5×). Query pattern: fast full-text search across all fields. Excellent for security/compliance (search for specific patterns across everything). Splunk: enterprise incumbent, similar to Elasticsearch, much more expensive. Standard modern choice: Loki for high-volume application logs; Elasticsearch/OpenSearch for security + compliance requiring full-text; Splunk if enterprise contract already in place. Both emit via OpenTelemetry Collector or Fluent Bit for vendor portability. Understanding this — structured logs + backend choice matched to query pattern — is Expert-tier competence.
Beyond individual signals, three primitives determine whether observability actually works operationally: OpenTelemetry as vendor-neutral instrumentation, SLO/SLI as the framework for defining "healthy," and symptom-based alerting as the discipline that keeps on-call humans effective. Each has specific mechanics. (a) OpenTelemetry (OTel): the CNCF observability standard that emerged from the 2019 merger of OpenTracing (tracing-only API) + OpenCensus (Google\'s metrics+tracing library). Provides three components. SDKs per language (Java, Python, Go, JS, .NET, Rust, Ruby, PHP, C++, Swift, etc.) implementing the OTel API — application code calls tracer.startSpan("my-op") or meter.createCounter("requests") and the SDK handles context propagation, sampling, batching, and export. Collector: standalone service that receives signals from SDKs, processes them (batching, filtering, sampling, redaction), and exports to any backend. Deployed as agent (DaemonSet per node) + gateway (Deployment for regional aggregation) pattern. Vendor-neutral: receivers include OTLP (OTel Protocol), Jaeger, Zipkin, Prometheus scrape; processors include batch, sampling, attributes, resource; exporters include Prometheus, Jaeger, Tempo, Loki, Datadog, New Relic, and virtually every observability backend. Semantic conventions: standardized attribute names so signals from any OTel-instrumented service use consistent naming — http.request.method, db.system, service.name, k8s.pod.name, etc. Cross-service queries work identically. Standard 2023+ instrumentation choice. (b) SLO/SLI discipline: the operational framework from the Google SRE book (Beyer et al. 2016). SLI (Service Level Indicator) = a specific measurement of service behavior, e.g., "the ratio of successful requests to total requests" or "the p99 latency for checkout completion." SLO (Service Level Objective) = the target value for an SLI, e.g., "99.5% of checkout requests complete successfully in a 30-day window." SLA (Service Level Agreement) = an external commitment to an SLO, typically with financial penalties. Error budget = 100% - SLO (e.g., 0.5% error budget for a 99.5% success SLO). Burn rate = current error rate ÷ acceptable error rate; burn rate > 1 means budget consumed faster than sustainable. Standard modern operational discipline. (c) Symptom-based alerting with multi-window multi-burn-rate: pages only fire when user-visible SLO is threatened; cause-based alerts (CPU, memory, disk) are informational warnings, not pages. Multi-window multi-burn-rate pattern (Google SRE book): fast-burn alert on 14.4× burn over 1 hour AND 6× burn over 6 hours (page — will exhaust monthly budget in ~2 days if unchecked); slow-burn alert on 3× burn over 6 hours AND 1× burn over 24 hours (warn — will exhaust monthly budget in ~30 days). Filters transient blips (single 1-hour window can be misleading) while catching real problems. Standard modern alerting pattern that dramatically reduces alert fatigue.
tracer.startSpan() + meter.createCounter() + logger integration; auto-instrumentation for HTTP servers, DB clients, gRPC, message queues, etc. Collector deployed agent+gateway: receivers (OTLP, Jaeger, Zipkin, Prometheus scrape), processors (batching, filtering, tail sampling, PII redaction, attribute manipulation), exporters (to Prometheus/Jaeger/Loki/Datadog/New Relic/any backend). Semantic conventions specify standardized attribute names (http.request.method, db.system, service.name) so services from different teams/languages produce consistently-queryable signals. Vendor-neutral: swap Prometheus for VictoriaMetrics without touching application code — only Collector config changes. Standard modern instrumentation. SLO/SLI discipline (Google SRE book 2016): SLI is a specific measurable indicator (fraction of successful requests, p99 latency, availability of a specific endpoint). SLO is the target value for the SLI (99.5% success over 30 days, p99 < 500ms). SLA is an external commitment with typically financial penalties. Error budget = 100% - SLO — the "allowed unreliability." A 99.5% SLO gives 0.5% error budget. At 1M requests/day × 30 days = 30M requests, 0.5% = 150K allowed errors. The team can spend this budget on deploys (some of which fail), maintenance windows, planned outages, or absorbing incidents. If budget exhausted mid-month: freeze deploys, focus on reliability. If budget healthy: ship features aggressively. Standard operational framework. Each additional 9 costs ~10× more (99% is 3.65 days/year down; 99.9% is 8.76 hours/year; 99.99% is 52.6 minutes/year; 99.999% is 5.26 minutes/year — five 9s requires distributed multi-region + heavy investment). Symptom-based alerting with multi-window multi-burn-rate: pages fire only on user-visible SLO threats (checkout success rate below target, p99 latency spike). Cause-based signals (CPU, memory, disk) are warnings/informational, not pages. Google SRE multi-window pattern: fast burn = 14.4× the acceptable burn rate over 1h AND 6× over 6h (page — at this rate, monthly budget exhausted in ~2 days, urgent). Slow burn = 3× over 6h AND 1× over 24h (warn — monthly budget exhausted in ~30 days, worth attention but not urgent). Two-window AND-gate filters transient blips (a single 1-hour window can be misleading) while catching real problems. Result: on-call receives 1-3 pages/week instead of 30. Every page is actionable and has a runbook. Signal-to-noise ratio high. Alert fatigue eliminated. Humans stay effective. Standard modern alerting discipline. The Expert insight: OTel provides the plumbing, SLO/SLI provide the framework for what "healthy" means, symptom-based multi-window alerts provide the operational discipline. Together: incident MTTR < 15 minutes, deploy cadence high (error budget allows fast iteration), team stays effective. Standard modern operational stack.Per-language library. tracer.startSpan(), meter.createCounter(), logger integration. Auto-instrumentation for frameworks. Standard 2023+ instrumentation choice; replaces vendor SDKs.
Agent (DaemonSet) + gateway (Deployment) pattern. Receivers → processors (batch, sample, redact) → exporters. Vendor-neutral routing. Standard deployment pattern.
SLI = measurement (success rate, p99 latency). SLO = target (99.5% success). Error budget = 100% - SLO. Framework for "healthy" (Google SRE book 2016). Each 9 costs 10×.
RED (Rate, Errors, Duration) for services. USE (Utilization, Saturation, Errors) for resources. Four Golden Signals (Latency, Traffic, Errors, Saturation) from Google SRE. Complementary frameworks.
Fast burn: 14.4× burn/1h AND 6× burn/6h → page. Slow burn: 3× burn/6h AND 1× burn/24h → warn. Filters blips, catches problems. Standard Google SRE pattern.
4th signal: CPU flame graphs sampled continuously in production. Pyroscope (Grafana), Parca (Polar Signals), eBPF-based zero-instrumentation. Standard 2022+ discipline.
The OpenTelemetry architecture (mech items i-ii) deserves specific attention because it\'s where instrumentation portability comes from. Consider concretely how a request flows through an OTel-instrumented system. Application code calls tracer.startSpan("checkout"); OTel SDK creates span, propagates context via traceparent header on outbound HTTP calls, captures duration + attributes + events. Metrics: meter.createCounter("http.server.requests").add(1, {method: "POST", route: "/checkout"}). Logs: standard logging library (structlog, zap, logback) automatically enriched with trace_id/span_id from active context. SDK batches signals in memory (default 5s or 512 items) and exports via OTLP protocol (gRPC or HTTP+protobuf) to a Collector endpoint (typically a sidecar or DaemonSet agent). Agent Collector receives, applies processors (batching, sampling, filtering by attributes, PII redaction), forwards to gateway Collector (regional aggregation). Gateway Collector applies tail sampling for traces (buffer 5s, decide based on rules), applies additional processing (attribute enrichment from k8s metadata, resource attribution), exports to backend of choice — Prometheus for metrics via remote_write; Jaeger/Tempo for traces via OTLP; Loki/Elasticsearch for logs. Standard modern architecture. Key benefits: (a) vendor portability — swap Prometheus for VictoriaMetrics by changing Collector exporter, no application changes; (b) processing centralization — apply sampling/redaction/enrichment once at Collector, not in every application; (c) protocol translation — receive Zipkin traces, export as OTLP; receive Prometheus scrape, export as Datadog metrics; (d) resource attribution — Collector k8s processor auto-enriches every signal with pod name, namespace, node, cluster from Kubernetes API. Standard modern discipline. Common failure modes: (i) shipping SDK-to-vendor direct (bypassing Collector) — loses processing benefits, couples applications to backend; (ii) 100% head sampling for traces — cost explodes at scale; (iii) missing semantic conventions — attribute names inconsistent across services, queries broken. Understanding OTel architecture is Expert-tier competence.
The SLO/SLI discipline (mech items iii-v) deserves specific attention because it\'s where reliability engineering becomes operational. Consider concretely how a team defines and operates SLOs. Step 1: identify user journeys (checkout, search, page load, API request). Step 2: define SLIs for each — measurable proxies for user happiness. For a checkout journey: (a) success SLI = successful_checkouts / total_checkout_attempts, (b) latency SLI = fraction of checkouts completing under 500ms. Step 3: set SLO targets: 99.5% success in 30 days, 99% under 500ms in 30 days. Step 4: compute error budgets: 0.5% error budget for success, 1% budget for latency. Step 5: configure burn-rate alerts (multi-window, symptom-based). Step 6: track budget consumption weekly; if healthy, ship features; if depleted, freeze deploys and focus on reliability. Standard Google SRE workflow. Choosing SLO targets: users don\'t distinguish 99.9% from 99.99% (both feel reliable); each additional 9 costs ~10× more engineering investment. Common targets: 99.9% for user-facing web (43 minutes/month allowed downtime — realistic for single-region), 99.95% for critical APIs (21.6 minutes/month — requires multi-AZ), 99.99% for payment/auth (4.3 minutes/month — requires multi-region + heavy investment), 99.999% for regulated systems (26 seconds/month — mostly aspirational, requires massive redundancy). Rarely justify 99.999% for typical products — user tolerance is usually 99.9-99.95%. Error budgets in practice: budget = 0.5% × 1M requests/day × 30 days = 150K allowed errors. Spent on: (a) planned deployments (some of which regress), (b) unplanned incidents, (c) maintenance windows. Team dashboard shows: "Budget remaining: 47K errors (31%). Burn rate 6h: 0.8× (healthy). Deploy velocity: high." If budget depletes to 0: freeze deploys, run reliability sprint. Aligns incentives: reliability teams and product teams have shared budget; product ships when budget is healthy, reliability wins when budget is depleted. Standard modern operational framework. RED/USE/Four Golden: complementary metric frameworks. RED (Weave Works — Rate, Errors, Duration) for request-driven services: RPS, error rate, latency percentiles. USE (Brendan Gregg — Utilization, Saturation, Errors) for resources: CPU util, memory pressure, disk I/O errors. Four Golden Signals (Google SRE — Latency, Traffic, Errors, Saturation) combines both. Standard modern dashboarding discipline: RED for each service, USE for each resource type, Four Golden for the top-level system dashboard. Understanding this framework is Expert-tier competence.
Below: each of three observability signals (Metrics · Distributed Traces · Logs) evaluated against three architecture profiles (Microservices at scale · ML/AI platform · Batch data processing). Watch how each signal fits each architecture — Traces × microservices is IDEAL (canonical distributed transaction reconstruction), Metrics × ML platform is IDEAL (numerical performance-heavy workload), Logs × batch is IDEAL (Spark stage debugging + data quality forensics). Off-diagonals still contribute value but with less leverage than the ideal fit. The takeaway: all three signals are needed at scale; the emphasis and investment shifts with architecture; OpenTelemetry standardizes the plumbing regardless.
The failure modes of observability are specific mechanisms by which "we have Prometheus + Grafana + ELK" turns into "we can\'t debug the 3am outage and half our team is burnt out from alert pages." Each anti-pattern is a real production pattern; Expert engineers avoid them by managing metric cardinality, applying tail-based trace sampling, using structured JSON logs, alerting on symptoms not causes, and standardizing on OpenTelemetry for instrumentation. Recognizing these saves years of "why is our observability so expensive and unhelpful" debugging.
Adding high-cardinality labels (user_id, request_id, session_id) to metrics causes exponential series explosion — every unique label combination creates a distinct time series. Prometheus stores each series with ~3KB memory index; 47M series × 3KB = 141GB just for index. The specific fix is to keep metric labels bounded (service, endpoint, status_code — enumerable values) and put per-request detail in traces or logs, using exemplars to link aggregate metrics to specific example traces. Specifically: (a) THE CARDINALITY MATH. Metric = name + label combination + timestamp + value. Each unique combination = one time series. Bounded labels: http_requests_total{service, method, status} with 20 services × 5 methods × 10 statuses = 1000 series (cheap). Add user_id with 10M users: 1000 × 10M = 10B series (impossible). Add request_id (unique per request): infinite growth. Standard failure. (b) THE PROMETHEUS RESOURCE MATH. Each active series uses ~3KB index memory + 1-4 bytes per sample. At 15s scrape: 5760 samples/day/series × 2 bytes = ~11KB/day/series. 1M active series = ~1TB/year storage + 3GB RAM. Prometheus struggles above 1M series; 10M+ requires VictoriaMetrics/Mimir/Thanos with horizontal sharding + object storage. 47M series in single Prometheus = certain OOM. (c) THE FIX. Rule: labels should be low-cardinality attributes of the aggregate, not identifiers of individual requests. Remove user_id, request_id, session_id from metric labels entirely. Keep service.name, http.request.method, http.response.status_code, k8s.pod.name (bounded to pod count). For per-request detail: use traces (each request = one trace, spans indexed by trace_id for lookup) or structured logs (per-event, query via LogQL/Elasticsearch DSL). (d) THE EXEMPLAR PATTERN. Modern Prometheus (2.26+) supports exemplars: attach a trace_id to a histogram bucket sample. Query: "show me the histogram of checkout latency; for the tail bucket (>1s), show me example trace_ids." Jump from aggregate metric directly to specific slow trace. Bridges the metric-trace gap without cardinality blowup. Standard modern discipline. (e) THE HISTOGRAM ALTERNATIVE. For latency distribution: use histograms with bounded bucket count. http_request_duration_seconds_bucket{le="0.5"} exposes a fixed 10-15 buckets per service/endpoint pair. Aggregatable across pods (sum bucket counts). Native histograms (Prometheus 2.40+, 2022): sparse bucket representation, higher resolution at similar cost. Standard 2024+ replacement for classic buckets. (f) THE RECOVERY. From 47M series: identify high-cardinality labels via prometheus_tsdb_head_series analysis; drop them via relabel_config at scrape time (remove label before storage); use recording rules to precompute needed aggregates from remaining low-cardinality metrics. Full recovery takes 1-2 weeks (memory releases as old series age out of head block). Standard incident response. Understanding this fix — that metric cardinality is a bounded resource and high-cardinality attributes belong elsewhere — is Expert-tier competence. Anti-pattern §05.i captures the failure to manage cardinality.
100% trace sampling is prohibitively expensive at scale — 50K RPS × 20 spans/request × 500 bytes = 500 MB/s of trace data = 40 TB/month. The specific fix is tail-based sampling at the OpenTelemetry Collector: buffer all spans of a trace, then decide based on rules (keep all errors, keep all slow requests, keep 1% of healthy). Preserves diagnostic value while dropping 99% of boring traces. Specifically: (a) THE SAMPLING VOLUME MATH. Modern microservices: single user request touches 10-30 services with 5-10 spans each. 50K RPS × 20 spans avg × 500 bytes/span = 500 MB/s. At 100% sampling: 40TB/month raw + backend index cost. Storage $2-5K, ingest processing $50-80K/month for typical tracing backends. Prohibitive. (b) THE HEAD SAMPLING PROBLEM. Head-based sampling: decide at request start (hash trace_id, keep 1% deterministically). Cheap: only 1% ever generated + stored. Fatal flaw: misses rare errors and slow requests preferentially. If your bug affects 0.1% of requests and you sample 1%, you keep 0.001% of buggy requests — statistical noise, invisible in traces. Standard failure of head sampling at scale. (c) THE TAIL SAMPLING FIX. OpenTelemetry Collector\'s tail_sampling processor buffers spans of a trace for a decision window (typically 5-30s), then evaluates rules against complete trace. Standard policies: keep 100% of traces with any error span, keep 100% of traces with latency > 500ms, keep 100% of traces from specific endpoints (payment, auth), keep 1-5% of everything else uniformly random. Result: 5-15% of traces kept, but 100% of diagnostic traces. Cost drops 85-95%; debug value stays. Standard modern discipline. (d) THE COLLECTOR CONFIG. OTel Collector tail_sampling processor: policies: [{name: errors, type: status_code, status_codes: [ERROR]}, {name: slow, type: latency, threshold_ms: 500}, {name: random, type: probabilistic, sampling_percentage: 5}]. Deploy at gateway Collector (regional aggregation) so all spans of a trace arrive at same node for buffering. Standard architecture. (e) THE MEMORY COST. Tail sampling requires buffering. At 500 MB/s of spans, 30s decision window = 15GB buffered per gateway Collector at any time. Multi-node with consistent hashing on trace_id (spans of same trace route to same Collector) — standard load balancer pattern. Manageable at scale with proper Collector sizing. (f) THE HEAD+TAIL COMPOSITION. Some workloads: use head sampling as a first cheap cut (keep 20% at ingestion), then tail sampling on the remaining 20% for finer filtering. Reduces buffering memory while preserving sampling quality. Standard modern pattern for very-high-volume services. (g) THE MEASUREMENT. Track: total spans emitted / total spans stored (target 5-20%); error trace retention rate (should be near 100%); trace query latency (should be under 5s for typical queries). Standard modern discipline. Understanding this fix — that tail sampling is the specific answer to "traces are too expensive at scale" — is Expert-tier competence. Anti-pattern §05.ii captures the failure to sample intelligently.
Unstructured plain-text logs are queryable only via full-text search — every meaningful query requires expensive scans across all log content. The specific fix is structured JSON logging: each log line is a JSON object with typed fields (timestamp, level, service, trace_id, user_id, duration_ms, message). Query becomes filter on indexed fields; correlation across metrics/traces via trace_id. Specifically: (a) THE PLAIN TEXT PROBLEM. Format: [2025-11-15 03:47:12] ERROR: payment failed for user 47129, order 8371, took 761ms. Fields embedded in prose; different services use different formats (some put timestamp first, others last; some quote strings, others don\'t; user_id might be "user_47129" or "u:47129" or embedded in message). Queries require regex parsing of every log line at query time — expensive. Aggregations impossible without pre-processing. Correlation across services impossible (no shared trace_id). Standard 2005-era logging. (b) THE STRUCTURED LOGGING FIX. JSON format: {"timestamp":"2025-11-15T03:47:12.847Z", "level":"ERROR", "service":"payment-service", "trace_id":"abc123", "span_id":"def456", "user_id":"47129", "order_id":"8371", "duration_ms":761, "message":"payment failed"}. Same information; now typed + queryable. LogQL query: {service="payment-service", level="ERROR"} | json | user_id="47129". Elasticsearch DSL: {"query": {"bool": {"filter": [{"term": {"service.name": "payment-service"}}, {"term": {"user_id": "47129"}}]}}}. Fast field queries. Aggregations native. Standard modern discipline. (c) THE LIBRARY CHOICE. Modern structured logging libraries: structlog (Python), zap (Go), logback with JSON encoder (Java), pino (Node.js), tracing/tracing-subscriber (Rust). All emit JSON with typed fields natively. Application code: logger.error("payment failed", user_id=47129, order_id=8371, duration_ms=761) — kwargs become structured fields, not concatenated into message string. (d) THE OPENTELEMETRY SEMANTIC CONVENTIONS. OTel specifies standard attribute names: service.name, trace_id, span_id, http.request.method, db.statement, user.id, etc. Every OTel-instrumented service produces logs with these standard fields. Cross-service queries work identically. Standard 2023+ practice. (e) THE TRACE CORRELATION. Structured logs enriched with trace_id + span_id from active OTel context (auto-instrumented in most SDKs). Query: "find all logs for trace_id abc123" — instantly shows exactly what happened across all services for that request. Bridges log-to-trace gap. Standard modern debugging pattern. (f) THE LOKI COST BENEFIT. Loki (Grafana Labs 2019) indexes labels only (bounded: service, level, k8s.pod.name), stores raw log bodies compressed in S3 (cheap). Label filter fast; body scans slower but tolerable when volume manageable. Cost: ~$0.023/GB stored. Elasticsearch/OpenSearch: full-text index inflates storage 2-5×, costs $3-5/GB effective. Loki 10-30× cheaper for high-volume application logs; Elasticsearch better for security/compliance requiring cross-field search. Choose backend by query pattern; both emit via OTel Collector or Fluent Bit for vendor portability. (g) THE MIGRATION. From plain text: (i) update logging library to structured JSON; (ii) add trace_id/span_id enrichment via OTel context; (iii) update log-parsing pipelines (Logstash/Fluent Bit) to parse JSON directly; (iv) update dashboards + alerts to use structured fields. Standard multi-week migration. Understanding this fix — that structured logging is table stakes for modern debugging — is Expert-tier competence. Anti-pattern §05.iii captures the failure to structure logs.
Cause-based alerts (CPU, memory, disk) fire on internal metrics that don\'t necessarily correspond to user-visible problems. The specific fix is symptom-based alerting with multi-window multi-burn-rate SLO burn detection: pages fire only when user-visible SLO is threatened; cause-based signals become informational warnings, not pages. Specifically: (a) THE CAUSE-BASED PROBLEM. Threshold alert on node_cpu_utilization > 80%: fires during any traffic spike, whether or not users are affected. Pods with 80% CPU may be perfectly healthy (running efficiently, serving traffic within SLO). Auto-scaling kicks in, adds capacity, CPU normalizes — but human already paged. Similar for memory (80% used may be healthy — JVM heap approaches limit before GC), disk (auto-expansion pending), network (bursty is normal). Result: 20-30 pages/week, most ignorable, alert fatigue high. Team stops trusting pages. Real incident gets ignored because "probably CPU noise again." Catastrophic. (b) THE ALERT FATIGUE MATH. Human effectiveness on-call: 2-3 pages/week healthy, 5-10 tolerable, 20+ dysfunction. Above 20/week: page fatigue, response time degrades, false-negative rate increases (real incidents ignored). Standard finding across on-call studies. Google SRE philosophy: every page should be actionable + urgent; if not, it\'s a warning. (c) THE SYMPTOM-BASED FIX. Alert on user-visible SLO violations. SLIs: checkout success rate, checkout latency p99, API error rate, page load time. SLO: 99.5% success in 30 days. Burn rate alert: fires when current error rate consumes budget faster than sustainable (14.4× burn over 1h = 2 days to exhaust monthly budget). Multi-window: also require slow-window confirmation (6× burn over 6h) — filters transient blips. Result: pages fire only when users are being harmed at rate that will exhaust budget. Standard modern discipline. (d) THE MULTI-WINDOW PATTERN. Google SRE book: (i) FAST BURN (page immediately): 14.4× burn over 1h AND 6× burn over 6h. Rationale: fast-burning problem that\'s also sustained. (ii) SLOW BURN (warn): 3× burn over 6h AND 1× burn over 24h. Rationale: slower problem worth attention but not urgent. AND-gate filters blips: momentary error spikes without sustained impact don\'t fire. Standard multi-window multi-burn-rate pattern. (e) THE CAUSE-BASED DEMOTION. Cause metrics (CPU, memory, disk) become dashboard warnings + Slack notifications, not pages. Auto-scaling handles capacity; monitoring alerts SRE team to investigate during business hours. If a cause-based signal correlates with user impact repeatedly, it becomes a symptom worth escalating — but only after evidence, not by default. Standard modern practice. (f) THE RESULT. On-call from 30 pages/week → 1-3 pages/week. Every page has a runbook. Every page is actionable. On-call effectiveness restored. Team retention improves. MTTR drops (each page gets full attention). Standard modern discipline. (g) THE MEASUREMENT. Track pages/week per on-call rotation; track page acknowledgment time; track false-positive rate (pages that resolved without human action). All should trend down as symptom-based alerting matures. Standard modern KPIs for observability. Understanding this fix — that alerting discipline is about human sustainability, not comprehensive coverage — is Expert-tier competence. Anti-pattern §05.iv captures the failure to alert on symptoms.
Proprietary vendor SDKs create instrumentation lock-in, inconsistent semantic conventions, and cost explosions. The specific fix is OpenTelemetry: vendor-neutral instrumentation with standardized semantic conventions, exported to any backend via Collector. Application code depends on OTel API only; backend choice is Collector configuration. Specifically: (a) THE LOCK-IN PROBLEM. Every service in the codebase depends on Datadog Tracing SDK or New Relic APM SDK. Application code: datadog.tracer.trace("checkout") or newrelic.agent.record_metric("..."). Migrating to different vendor requires: (i) removing every SDK call from every service (300 services), (ii) replacing with new vendor\'s SDK calls, (iii) re-testing every code path, (iv) coordinating rollout across teams. Years of work. Vendors know this — they price aggressively knowing switching cost is prohibitive. Datadog metrics: $0.90-2.60 per custom metric per month at scale = hundreds of thousands to millions of dollars annually for large deployments. Splunk ingest: $65/GB/month at typical enterprise volume = similarly explosive. Alternative: OpenTelemetry with commodity backends (Prometheus/VictoriaMetrics, Jaeger/Tempo, Loki) at 10-100× lower cost. (b) THE INCONSISTENT CONVENTIONS. Datadog uses http.method; New Relic uses request.method; Splunk expects httpMethod. Cross-vendor queries impossible. Each new service = decision "which vendor to instrument for." Fragmented observability. Merging teams = merging convention nightmare. (c) THE OPENTELEMETRY FIX. Application code depends on opentelemetry API only. tracer.startSpan("checkout"), meter.createCounter("orders"). OTel SDK handles context propagation, sampling, batching, export. Signals sent via OTLP (OTel Protocol) to Collector. Collector configuration decides backend: exporters: {datadog: {...}, prometheusremotewrite: {...}, jaeger: {...}}. Application code unchanged when switching backends. Standard modern instrumentation choice. (d) THE SEMANTIC CONVENTIONS. OTel specifies standard attribute names: http.request.method, http.response.status_code, db.system, db.statement, service.name, k8s.pod.name, rpc.service, etc. Every OTel-instrumented service produces signals with these attributes regardless of framework/language. Cross-service queries work identically. Standard 2023+ practice. (e) THE COLLECTOR ROUTING. OTel Collector receives from applications (OTLP protocol) and exports to any backend. Deployment: agent (DaemonSet per node) receives from local pods; gateway (Deployment, regional) aggregates + tail samples + routes. Vendor migration: change Collector exporter config; application code untouched. Multi-backend: export same data to Datadog for exec dashboards AND Prometheus for engineering + cost control. Standard modern architecture. (f) THE COST BENEFIT. Datadog metrics at $340K/year → Prometheus + VictoriaMetrics at $15-40K/year (self-hosted or managed). Similar for traces (Jaeger/Tempo) and logs (Loki). Vendor cost reduced 5-20× while retaining SaaS options for specific use cases (Datadog for exec dashboards, Honeycomb for high-dimensional trace analysis). Standard 2023+ cost optimization. (g) THE MIGRATION PATH. Existing vendor SDK codebase: (i) instrument new services with OTel from day 1; (ii) wrap existing SDK calls with OTel adapter (compatibility layer); (iii) migrate service-by-service to native OTel; (iv) once all services OTel-instrumented, swap backends at Collector level; (v) decommission vendor SDKs. 12-24 month migration for large codebases; teams that start now save millions long-term. Standard modern discipline. Understanding this — that OpenTelemetry is the vendor-neutral standard and backend choice is a config decision, not a codebase decision — is Expert-tier competence. Anti-pattern §05.v captures the failure to standardize on OTel.
The composite pattern across all five is that observability failure modes reflect specific engineering gaps in cardinality management (labels vs traces), sampling strategy (head vs tail), logging discipline (unstructured vs JSON), alerting philosophy (cause vs symptom), and instrumentation standard (vendor vs OTel). High-cardinality metric labels blow up Prometheus. 100% trace sampling blows up costs. Plain-text logs blow up query time. Cause-based alerts blow up on-call effectiveness. Vendor SDK lock-in blows up switching costs. Each has specific fixes: (a) bounded metric labels + exemplars + histograms; (b) tail-based sampling at Collector with error/slow bias; (c) structured JSON logs with OTel semantic conventions + trace_id correlation; (d) symptom-based multi-window multi-burn-rate alerts on SLO burn; (e) OpenTelemetry SDK + Collector for vendor-neutral instrumentation. Getting observability right is the specific engineering discipline that turns "our distributed system is a black box we can\'t debug and our on-call is drowning in false pages" into "MTTR under 15 minutes, 1-3 pages/week per rotation, vendor-portable stack at 10× lower cost, correlated signals across metrics/traces/logs via trace_id."
The terms that show up in every incident review, every SLO planning session, every OpenTelemetry rollout discussion.
Test the observability understanding. Click an answer; explanation drops in instantly.
Perfect. Metrics + traces + logs composed via OpenTelemetry, SLO/SLI discipline with error budgets, symptom-based multi-window alerts — the specific engineering for modern production observability. Next: M.67.
The composite understanding that turns "our distributed system is a black box that occasionally breaks and we don\'t know why" into "MTTR under 15 minutes, 1-3 pages/week per rotation, vendor-portable stack at 10× lower cost, correlated signals across metrics/traces/logs via trace_id."
Metrics tell you WHAT (aggregate: error rate up 3×). Traces tell you WHERE (payment.process spent 761ms in postgres). Logs tell you WHY (exact SQL statement + parameters). Each has specific mechanics (cardinality, sampling, structure). Correlated via trace_id, they answer any question. Continuous profiling (Pyroscope, Parca) as 4th signal.
Vendor-neutral SDK + Collector + semantic conventions. Application code depends on OTel API only; backend is Collector config. Swap Prometheus/Jaeger/Loki for Datadog/New Relic/Splunk without touching services. Semantic conventions ensure cross-service query consistency. De facto standard by 2023. Standard modern instrumentation choice.
SLIs measure user-relevant behavior. SLOs set targets with error budgets (100% - SLO). Budget spent on deploys + incidents; depleted → freeze deploys. Multi-window multi-burn-rate alerts (14.4×/1h + 6×/6h) fire only on user-visible SLO burn. Cause-based signals become warnings. Result: 1-3 pages/week, high signal-to-noise, no alert fatigue.