The second Phase I capstone. Where the five modules of Phase H — batch vs stream (M.39), windowing (M.40), stateful streams (M.41), CDC (M.42), real-time analytics (M.43) — compose into a coherent multi-tier platform. The specific teaching: the same problem produces different architectures depending on whether you optimize for freshness, cost, or correctness. Five specialized tiers, each with its own scaling axis and failure surface, and the discipline to design across all of them together.
Every real-time analytics platform faces the same first architectural fact: no single technology handles sourcing, transporting, processing, materializing, and serving continuous data — and none is coming. Postgres is excellent for transactional writes, terrible for analytical scans over 100M rows. Kafka is excellent for durable ordered streams, useless as a query interface. Flink is excellent for stateful stream processing, ill-suited as a queryable database. Materialize/Pinot/Druid are excellent for querying, unable to ingest from operational databases without help. The real-time analytics stack is inherently multi-tier — five specialized components, each doing one thing well, composed into a pipeline. The composition is what makes the stack viable. But the composition also introduces its own class of problems: back-pressure across tiers, schema evolution coordination, capacity mismatches, failure cascades. This module is about designing that composition deliberately.
The specific difficulty of platform composition is that each tier is a well-understood technology in isolation but the composition creates emergent problems. Kafka is well-understood; Flink is well-understood; Materialize is well-understood. Combining them requires answering: How many Kafka partitions should the topic have to match Flink's parallelism? What checkpoint interval works for Flink given how much state Materialize can absorb without lag? How does the platform respond when Materialize goes down — does Kafka retain enough to let Flink recover, or does Flink go into back-pressure that eventually stalls Debezium and then blocks Postgres WAL? These are composition questions that don't arise from any single component's documentation. They arise from the interactions between components, and getting them right is what separates a working prototype from a production platform.
Each earlier attempt uses one tool for everything and produces a specific failure mode — Postgres-everywhere hurts OLTP, Kafka-Streams-everywhere lacks a query layer, Snowflake-everywhere is too stale. The multi-tier composition (CDC + Kafka + Flink + Materialize/Pinot + dashboards) is the only architecture that handles all three concerns simultaneously — but only if each tier is tuned for the specific requirement profile. Freshness-first pushes toward smaller checkpoints, tighter partitions, and IVM-based materialization. Cost-first pushes toward larger checkpoints, rollup-based materialization, and cached query results. Correctness-first pushes toward exactly-once semantics and audit trails. Same architecture, different knobs — this is the specific engineering skill M.45 is trying to build.
The historical arc of real-time analytics platform design is worth naming because it converged on the current shape only after multiple abandoned patterns. 2011-2014: Lambda architecture (coined by Nathan Marz) — separate batch layer (Hadoop/Spark) for correctness and speed layer (Storm) for freshness, with a serving layer merging results. Complex, high maintenance burden, dual-code paths. 2014-2016: Kappa architecture (Jay Kreps at LinkedIn) — reject the dual-path complexity; use streams for everything; replay from Kafka to rebuild historical state. Simpler mental model, but required streaming systems capable of handling batch-scale workloads — which didn't fully exist yet. 2016-2019: Flink and Kafka Streams maturation — event-time semantics, exactly-once processing, keyed state — the tools got good enough for Kappa to be viable at scale. 2019+: The modern multi-tier stack emerged as the practical realization of Kappa: not one tool doing everything, but specialized tools per tier composed together. Materialize (2019), RisingWave (2021), Debezium's rise (2016-2020) — each new piece filled a specific composition gap. The 2020s consensus is: multi-tier composition with tunable per-tier semantics is the answer. This module is a formal statement of that consensus and a guide to designing within it.
The modern real-time analytics platform has converged on a specific five-tier reference architecture — each tier a specialized layer that does one thing well and hands off to the next. Understanding this architecture is not enough to build a platform; you also need to understand the capacity, latency, and failure characteristics of each tier so you can size and tune it for your specific workload. This section walks through the five tiers with representative capacity numbers at each. The numbers are approximate — real deployments vary based on hardware, workload shape, and configuration — but the orders of magnitude are the ones you should reason about when designing.
Operational databases (Postgres, MySQL, MongoDB) plus Debezium (M.42) to emit changes as an ordered stream. Capacity is bounded by the source DB's transaction throughput — a well-tuned Postgres primary sustains 1-10K row modifications/sec. Log-based CDC is nearly free operationally (reads happen out of the write path) but requires operational discipline around replication slots and schema evolution. The "we forgot to monitor slot lag" outage is the specific failure mode.
Kafka (or Kinesis, Pulsar) as the durable pipe between source and processing. Capacity scales with partition count and broker fleet — a modest 3-broker cluster handles 100K messages/sec; 30-broker clusters at LinkedIn/Uber handle 10M+ msg/sec. Partition count is the single most important sizing decision: too few and Flink can't parallelize; too many and offset management overhead dominates. Rule of thumb: partitions = 2-4× your target Flink parallelism per topic. Retention determines how far back you can replay.
Flink (or Kafka Streams, Spark Structured Streaming) for the stateful processing layer. Windowing (M.40), state (M.41), event-time semantics. Capacity per worker is 10K-100K events/sec depending on state size and operator complexity. Checkpoint interval is the freshness knob: 2s for freshness-first, 60s for cost-first. State backend (memory vs RocksDB) determines memory pressure and checkpoint cost. Scale horizontally by increasing parallelism (which requires enough Kafka partitions upstream).
Materialize / RisingWave (streaming SQL — M.43) OR Pinot / Druid / ClickHouse (real-time OLAP — M.43). The specific choice depends on query shape: JOINs and transactional consistency → Materialize; high-cardinality dashboards → Pinot; time-series → Druid. Serves 1K-10K queries/sec per node. This tier is where users actually query, so its latency directly shows up in dashboards. Freshness tuning here (view refresh interval, rollup granularity) is where the freshness-vs-cost tradeoff most visibly lives.
Grafana, Superset, custom React dashboards, internal APIs, alerting systems. Capacity is fan-out from the materialization tier — CDN/cache layers absorb 90%+ of read traffic in mature deployments. Query patterns are hot (most users hit the same views); caching aggressively at this tier saves the materialization tier from thundering-herd problems. The 10K-100K concurrent users number assumes proper caching; without it, the materialization tier gets overwhelmed at <1K users.
The five-tier decomposition is the composition insight — no single tier does everything, but the pipeline handles all the concerns because each tier hands off a well-defined interface to the next. Postgres knows nothing about Kafka; Kafka knows nothing about Flink's state; Flink knows nothing about Materialize's query engine; Materialize knows nothing about the dashboard's rendering. Loose coupling is the specific property that lets each tier scale independently and be replaced independently. Netflix might swap Debezium for their internal CDC system; Uber might swap Flink for their internal stream processor; the rest of the pipeline is unaffected. This modularity is what makes the pipeline maintainable at multi-year scale.
The capacity numbers matter because mismatches produce specific failure modes. A Kafka topic with 3 partitions cannot support Flink parallelism of 12 — 9 Flink workers sit idle waiting for partition assignments. A Flink deployment processing 500K events/sec cannot feed a Materialize instance that ingests 50K events/sec — Materialize falls behind, view freshness lags, dashboards go stale. A Materialize view maintaining 100K unique dimension combinations may exceed available memory. Every tier's capacity has to be sized to match the tier upstream and downstream. The most common capacity-planning mistake is treating tiers independently rather than as a coupled system with matched throughputs.
The composite pipeline behaves differently from any single component because the tiers are coupled through streams — when one tier slows down, the tier upstream fills its buffer and eventually applies back-pressure, propagating the slowdown backward through the pipeline. This is a specific class of behavior that emerges only from composition and is invisible when reasoning about each tier in isolation. Understanding back-pressure and failure cascades is what separates platforms that survive real production incidents from ones that go into unrecoverable states during peak load. This section covers the specific coupling behaviors and how to design around them.
The back-pressure story works as follows in the reference architecture. Consider what happens if the materialization tier (Materialize) slows down — GC pauses, view state grows, checkpoint duration extends. Flink can't stream events to Materialize as fast as they arrive from Kafka; Flink's output buffer fills; Flink's operators back-pressure the source operators reading Kafka; Flink stops advancing its consumer offsets; Kafka retains messages that would otherwise have been consumed. If Kafka retention is 7 days, Materialize has 7 days of slack before the pipeline fails permanently — during which the operator has time to either scale Materialize up or accept that some data will be lost. The composite pipeline degrades gracefully because each tier's buffer absorbs some slack. Without buffering — if any tier synchronously called downstream — the whole pipeline would collapse the instant any component slowed. Kafka's durability + Flink's back-pressure semantics + the source's slot retention are what makes the composite survive.
The failure cascade story is the corollary: when a tier fails completely, the tiers upstream feel it, but Kafka acts as a shock absorber. If Materialize goes down, Flink stops advancing (waits for the sink to recover) but doesn't fail — Kafka keeps buffering. If Flink goes down, Kafka doesn't care — it's not consuming; the source keeps producing. If Kafka goes down (rare but catastrophic), the source's Debezium buffers WAL locally (with limits — Postgres slot retention), and everything downstream is idle. If Postgres goes down, everything stops but nothing loses data — Kafka retains, Flink recovers from checkpoint. The specific ordering matters: the further downstream a failure, the more slack the upstream buffer provides. This is the design intent behind Kafka's central role in the pipeline — it's a shock absorber, not just a message bus. Get its retention configuration wrong and the pipeline's resilience budget shrinks accordingly.
The capacity planning discipline follows from these coupling behaviors. Each tier should be sized to handle peak inbound rate × safety factor — typically 2× or 3× the average to accommodate bursts, seasonal peaks, and the occasional replay after a failure. Kafka's partition count should be sized to allow Flink parallelism to match the target throughput (partition count = 2-4× Flink parallelism per topic). Flink's state backend should be sized for the peak state, not average state (M.41's anti-pattern §05.iv). Materialize's memory should accommodate the peak result-set size, not just steady-state. Under-provisioning any tier creates a specific failure mode — the pipeline works fine 99% of the time and fails during exactly the peak that matters most. Mature deployments run load tests that push each tier to 3-5× normal load to verify the capacity math.
The schema evolution discipline is a composition problem that spans all five tiers simultaneously. When the source Postgres schema changes (add a column to users), the change flows through Debezium (which detects it and updates the schema registry), Kafka (which stores messages with the new schema version), Flink (which needs to be able to deserialize the new format), Materialize (which needs to handle the new field in its view definitions), and dashboards (which need to know whether to show the new field). Getting one tier's schema evolution wrong breaks the whole pipeline. Mature platforms use Avro/Protobuf with strict schema-registry compatibility rules, deploy schema changes across tiers in a coordinated sequence (downstream first, source last), and have dashboards to monitor schema drift. This is anti-pattern §05.v in composite form.
The observability discipline is what makes debugging composite pipelines viable. Every tier emits metrics: source tier reports CDC lag and replication slot size; transport tier reports partition offsets and consumer lag per consumer group; processing tier reports checkpoint duration, watermark lag, and state size; materialization tier reports view freshness and query latency; presentation tier reports request rate and cache hit ratio. These metrics compose: the end-to-end freshness of a dashboard is the sum of source-to-Kafka lag + Kafka-to-Flink lag + Flink-to-Materialize lag + Materialize-to-dashboard lag. When freshness degrades, tracing which tier introduced the lag is what turns a "why is the dashboard slow?" ticket into an actionable investigation. Every mature platform builds an end-to-end lag dashboard as a first-class SLI.
Below: each of the three canonical design priorities (freshness-first · cost-first · correctness-first) evaluated against three different query shapes (interactive dashboards · analytics API · real-time detection). Watch how the same "real-time analytics platform" requirement produces different technology stacks, capacity numbers, and operational profiles across the 9 combinations. The lab illustrates the specific composition insight: platform architecture is not about picking tools — it's about matching per-tier tuning to the requirement profile of the workload.
The failure modes of real-time analytics platforms are the reason "just wire the tools together" is not an architectural answer — the composition-level decisions produce failure profiles that no individual tier's documentation warns about. Platforms that look healthy in per-tier dashboards fail catastrophically at composition boundaries when the tier interactions weren't designed deliberately. Diagnosing these requires understanding the specific failure modes of composition itself, not just of individual components.
Every tier's capacity has to match adjacent tiers, and the pipeline throughput is bounded by the smallest tier — but underinvestment in one tier is a specific pattern that shows up because teams often size tiers in isolation. Kafka partitions determine Flink parallelism (each partition can be consumed by at most one Flink worker per consumer group), so 6 partitions caps parallelism at 6 regardless of how many Flink workers are provisioned. Flink instance count doesn't help; adding more workers just adds idle workers. The fix: (a) partition count should be 2-4× target Flink parallelism per topic, so if Flink target is 24 parallelism, use 48-96 partitions; (b) plan for future scale — partitions can be added but not removed without downtime; (c) run end-to-end load tests that push through every tier, not per-tier micro-benchmarks. The general principle: tier capacities are coupled through the interfaces; sizing them independently produces the "skimped tier" failure mode. Anti-pattern §05.i is the most common composition mistake.
The materialization tier (Materialize / Pinot / Druid / ClickHouse) has different query shapes it excels at. Materialize is designed for streaming SQL with JOINs and incremental view maintenance. Pinot is designed for high-cardinality dashboards with simple aggregations. Druid is designed for time-series rollups. ClickHouse is designed for OLAP scans. Putting JOINs in Pinot is expensive; putting dashboard queries in Materialize wastes its transaction support. The fix: (a) match query shape to tier — JOIN-heavy → Materialize; high-cardinality aggregations → Pinot; time-series rollups → Druid; ad-hoc analytics → ClickHouse/Trino; (b) sometimes the right answer is multiple materialization tiers, each specialized for a query class; (c) don't retrofit — if you picked Pinot and now need JOINs, adding Materialize alongside is often better than migrating everything. The general principle: the materialization tier is not one tool; it's a category, and picking the right subcategory matters more than the aggregate performance benchmarks.
M.41's anti-pattern about unbounded Flink state reappears at platform scale with additional consequences. When state grows without eviction, checkpoint duration extends (proportional to state size), recovery time extends (must reload the entire state from checkpoint storage), and Materialize's view maintenance falls behind because it can't keep up with Flink's output rate. The platform-scale symptom is end-to-end lag creeps up over months while nobody notices. The fix: (a) define TTL on all keyed state — sessions expire after 24 hours, user profiles cached for 7 days; (b) use windowed state instead of continuously-accumulating state where semantics allow; (c) monitor state size as a first-class SLI with alerts on growth rate; (d) checkpoint incrementally (RocksDB state backend, S3 checkpoints) so growth doesn't linearly increase checkpoint cost. The general principle: any state that grows without bounds will eventually exceed available memory, checkpoint budget, and recovery SLO — plan for eviction from day one.
Recovery is when back-pressure discipline is most tested and most often violated. A tier that was slow for an hour has an hour of backlog to catch up on plus continuing live traffic — so it needs 2× normal capacity to catch up in real time. If it's provisioned for exactly normal capacity, it can never catch up and the pipeline stays in permanent back-pressure. This is the failure mode that turns a 2-hour incident into a 20-hour outage. The fix: (a) each tier should be sized to 2-3× normal capacity so it can absorb recovery bursts; (b) Kafka retention should be at least as long as the worst-case recovery time; (c) during recovery, throttle the source to give downstream tiers time to catch up (Debezium supports pause/resume); (d) have runbooks for "how to recover the platform after tier X was down for Y minutes." The general principle: back-pressure discipline is not a normal-operations concern; it's a recovery concern, and provisioning for exactly normal capacity guarantees you can't recover.
Schema changes at the source flow through all five tiers, and each tier's schema handling has to be coordinated with the deploy sequence. M.42's anti-pattern §05.v (uncoordinated CDC schema evolution) reappears at platform scale with additional consequences — Flink needs the schema at deserialization time; Materialize needs it at view-definition compile time; dashboards need it at query-formulation time. Deploying schema changes without coordination breaks tiers in the order they encounter the new format. The fix: (a) use Avro/Protobuf with schema registry that enforces backward-compatibility (new schema can be read by old consumers); (b) deploy the change from downstream to upstream — dashboards learn about the new field first (ignoring it if not present), then Materialize adds it to views (with COALESCE for missing values), then Flink updates its deserialization, then the source deploys the schema change; (c) monitor schema-registry compatibility violations as a first-class SLI. The general principle: schema evolution is a distributed transaction across all five tiers; treating it as a source-only change causes cascading breakage.
The composite pattern across all five is that real-time analytics platforms fail at composition boundaries, not within individual tiers. Each tier is well-understood and well-instrumented in isolation. The failure modes emerge from tier interactions: capacity mismatches (§05.i), workload-tier mismatches (§05.ii), unbounded state (§05.iii), recovery capacity (§05.iv), schema coordination (§05.v). Mature platform teams invest in composition-level engineering practices — end-to-end lag dashboards, cross-tier load tests, schema evolution protocols, recovery runbooks. Without these, the platform works fine at low load and fails at exactly the peak that matters most.
The terms that show up in every "should we build our own platform or buy?" architecture review, every "why is our end-to-end lag creeping up?" postmortem, every "how do we handle Black Friday scale?" design doc.
Test the vocabulary. Click an answer; explanation drops in instantly.
Perfect. Five tiers, three design priorities, tier coupling, back-pressure, schema evolution — the composite intuition that separates working prototypes from production platforms. Next up: M.46, the Intermediate track finale.
The composition discipline that makes real-time analytics platforms viable at multi-year production scale.
No single tool spans the pipeline. Source (Postgres+Debezium), transport (Kafka), processing (Flink), materialization (Materialize/Pinot/Druid), presentation (dashboards). Each tier does one thing well; each tier scales independently; each tier has well-defined interfaces to adjacent tiers. The loose coupling is what makes the pipeline maintainable at multi-year scale — you can swap any single tier without rewriting the rest.
The same "real-time analytics" problem produces different architectures depending on which axis you optimize for. Freshness-first uses Materialize with tight Kafka partitions and 2s checkpoints (target sub-500ms lag). Cost-first uses Druid with heavy rollup and 60s checkpoints (target 1min lag, $$ cost). Correctness-first uses exactly-once end-to-end with 2PC-verified sinks. Same architecture, different knobs — the composition capstone is knowing which knobs match which requirement profile.
Failure modes emerge from tier interactions, not from individual components. Skimped tiers cap throughput (partition-Flink mismatch). Wrong tier for the workload (JOINs in Pinot). Unbounded state grows checkpoint duration. No back-pressure discipline turns 2-hour incidents into 20-hour outages. Uncoordinated schema evolution breaks downstream tiers. Mature platform teams invest in composition-level practices — end-to-end lag dashboards, cross-tier load tests, schema evolution protocols, recovery runbooks.