The foundational data-processing decision that shapes everything downstream. Process data in bounded chunks after it accumulates or process it continuously as it arrives. Different latency, different throughput, different consistency guarantees, different operational shapes. Phase H opens with the choice that decides how every subsequent pipeline is built.
Phase G built services — bounded contexts, gateways, meshes, discovery, tracing, storage, communication. Phase H is about how data moves through your system. Every one of those services emits and consumes data at scale — clicks, orders, sensor readings, log lines, state changes, transactions. The services part is settled. The pipelines that carry, transform, aggregate, and materialize that data are the next layer. And the foundational question that shapes every one of them — the decision this module answers — is: when should the data be processed?
The answer is one of two shapes. Batch processing waits for data to accumulate, then processes bounded chunks at scheduled intervals: nightly rollups, weekly cohort analyses, monthly billing runs. Stream processing runs continuously, handling each event as it arrives: real-time fraud checks, live dashboards, alerting on anomalies. Neither is universally right, and choosing the wrong one is one of the more expensive mistakes in data infrastructure — batch masquerading as real-time frustrates users; stream masquerading as batch burns compute for no benefit; both errors accumulate over years of pipelines that "just have always worked this way." This module is the vocabulary that prevents them.
Of six real workloads, 2 are naturally batch, 2 are naturally stream, 1 sits comfortably in micro-batch, and 1 needs both. There is no universal answer — the choice is per-workload, driven by freshness needs (how quickly must the result be available?), throughput needs (how much data per unit time?), and correctness needs (must the result be deterministic and reproducible?). Systems that pick one mode as their default and force everything into it accumulate the specific costs each workload's mismatch produces — batch systems can't do real-time fraud; stream systems burn compute on nightly rollups. Deliberate per-workload matching is the skill.
The reason both patterns coexist and neither has won is that the tradeoffs are fundamental, not incidental. Batch trades latency for simplicity and throughput: waiting until data accumulates lets you optimize the whole job — write parallelization strategies, use columnar formats, share compute across tasks. The output is a snapshot of a bounded window; if you need to rerun (bug found, requirement changed), you rerun the whole job against the same input and get the same output. Determinism is a first-class property. Stream trades simplicity for latency: processing each event as it arrives means the system is always mid-computation, holding state across events, managing time-windows that never quite close, dealing with out-of-order arrivals and late data. The output is a continuously updating view; reruns are harder because "the same input" is a moving target. Freshness is a first-class property, and correctness gets more complicated to prove.
The interesting cases are the ones that live in between — micro-batch (Spark Structured Streaming's original design: process a small batch every N seconds, get bounded latency with batch's simplicity) and Lambda architecture (Nathan Marz's 2011 pattern: run BOTH a batch layer for accuracy and a stream layer for latency, merge results at query time). Micro-batch is now widely used for "streaming-ish" workloads where sub-second latency isn't required and stream complexity isn't wanted. Lambda has largely been out-competed by Kappa architecture (Jay Kreps, 2014: stream-only, treat everything as an unbounded event log, run any historical query as a stream replay from the start of the log) — but Lambda still shows up in older systems and in situations where batch accuracy is legally or contractually required. The rest of Phase H builds on the batch/stream foundation: §04's lab lets you feel the tradeoffs concretely, and modules M.40 (windowing), M.41 (stateful streams), M.42 (CDC), and M.43 (real-time analytics) each build depth in the stream side once this foundation is settled.
The batch model is older, simpler, and — for the workloads it fits — genuinely dominant. It descends from MapReduce (Dean & Ghemawat, Google, 2004) and its Apache open-source successor Hadoop; the current mainstream is Apache Spark, which added in-memory computation, richer APIs, and dramatic performance improvements over MapReduce's disk-round-tripping. The defining property of batch is that the input is bounded — you know when the day ends, when the file is complete, when the accumulated events are ready to process. That single constraint enables everything else the model gives you: deterministic reruns, parallelizable execution across the whole input, choice of optimal data layouts and formats, and jobs whose success or failure is a binary decision at the end.
The workloads where batch dominates share a specific signature. Historical aggregations: how much revenue by region last month, cohort retention curves, ML model training on 90 days of features — these are inherently backward-looking, want the whole input, and don't need sub-minute freshness. Regulatory and financial reporting: payroll, GAAP-compliant financial rollups, tax calculations — these require deterministic reruns and audit trails, both of which batch provides naturally. Bulk data movement: nightly warehouse loads, database backups, data migrations — these are one-shot activities where "did it succeed?" is the only question. What all three share is that latency in hours or days is fine, and the value is in the completeness and correctness of the output rather than its immediacy. The batch model's simplicity — inputs known, execution parallelizable, output binary — is the specific engineering property that makes these workloads tractable at scale.
The batch model's operational shape is schedulers-and-jobs. Airflow (the industry default, from Airbnb, 2015) organizes work as directed acyclic graphs (DAGs) of tasks, with each task being a discrete unit of work — extract from source, transform via Spark, load to warehouse. Prefect and Dagster are newer alternatives with slightly different philosophies (Prefect is Python-native and flow-oriented; Dagster is asset-oriented, treating data artifacts as first-class). The scheduler triggers jobs on cron-like schedules or on data-availability sensors ("run the rollup when yesterday's raw data has finished loading"). Failures trigger retries with configurable policies; permanent failures alert humans. The whole ecosystem is optimized around the assumption that jobs run for minutes-to-hours, produce a bounded output, and either succeed or fail cleanly. When a workload violates that assumption — needs sub-second latency, needs continuous state — the batch model can be pushed into it via micro-batch, but the fit is imperfect. The next section is where that limitation shows up.
The stream model is younger, more complex, and — for the workloads it fits — indispensable. It descended from real-time analytics engines (Apache Storm, 2011; Samza, 2013), converged in the current generation around Apache Flink and Kafka Streams, and now has a mature ecosystem including managed services (Confluent Cloud, Amazon Kinesis Data Analytics, Google Dataflow). The defining property of stream is that the input is unbounded — events arrive continuously, forever, and there's no "end of file" to trigger a final result. That single fact changes everything: results are always partial and always-updating, state must be maintained across events, time-windows must be defined explicitly, and out-of-order arrivals must be handled deliberately. Streams are more expressive than batch — they can do everything batch does plus real-time — but the complexity budget is meaningful and it applies to every workload put into it.
The workloads where stream dominates share the opposite signature from batch. Sub-second freshness needs: fraud detection, live monitoring, real-time personalization, alert generation. Continuous state: rolling counters ("requests per second in the last minute"), session windows ("user activity within a 30-min gap"), running joins (order + shipping updates). Reactive coordination: microservices that need to know about state changes as they happen (M.38's pub-sub events consumed by stream processors). What all three share is that the value is in the immediacy — a fraud check delivered 24 hours late is worthless, a dashboard showing 30-min-stale data is misleading, a personalization signal for a session that already ended is useless. Stream processing exists specifically for workloads where latency is not just preferred but foundational to the value proposition.
Run TWO parallel pipelines: a batch layer for accuracy (Spark, nightly rollups) and a speed layer for latency (stream processor, real-time approximations). A serving layer merges results at query time — recent data from stream, historical data from batch. Solves the mixed-workload problem by having both types of pipeline coexist; guarantees accuracy from batch while providing latency from stream. Historically dominant in large-scale analytics.
Use ONLY stream processing. Kafka retains the full event log; historical queries "replay" from the beginning of the log; recent data flows through the same processor. Removes the code-duplication problem: every transformation is written once, in stream form. Backfills become "start a new consumer from offset 0 and let it catch up." Since 2020 or so, Kappa has become the industry default for greenfield data platforms — the Lambda code duplication was widely regretted.
The composite guidance for 2025-era systems: Kappa is the modern default for greenfield streaming platforms, with batch retained for the specific workloads where its properties (determinism, high per-run throughput, native fit for regulated / audited processes) genuinely justify a separate execution model. Modern unified engines — Apache Flink's batch-mode execution, Apache Beam's "one API for both," Google Dataflow, Materialize — collapse the Lambda code-duplication problem by letting you write the transformation once and have it execute in either batch or streaming mode depending on the source. The result is that "batch vs stream" is increasingly a question of execution strategy rather than architectural choice. The upcoming Phase H modules build on the stream side of this foundation: M.40 (windowing) is about how to slice unbounded streams into meaningful groups; M.41 (stateful streams) is about how state is managed across events; M.42 (CDC) is about turning database changes into streams; M.43 (real-time analytics) is about making stream state queryable at low latency. The batch side is settled infrastructure at this point; the stream side is where the interesting work of the next decade lives.
Below: five real data workloads — daily sales rollup, real-time fraud check, product-view counter, customer LTV, live ops dashboard. Pick a processing mode — batch (nightly), micro-batch (5-minute), or stream (continuous) — and see how each workload fares. Watch the latency change dramatically, watch which combinations are natural fits, which are wasteful overhead, and which are outright dangerous.
The failure modes of batch/stream choice are recognizable and expensive. Each of the five patterns below has cost real teams months of work — either paying for compute they didn't need, or paying for engineering complexity that yielded no user-visible latency win. Recognizing them early — ideally at architecture review — is where mature data-engineering practice actually shows up.
A daily or hourly batch job feeding a dashboard that stakeholders check as though it's real-time is one of the most common misalignments in production data systems. The team says "it updates every hour"; users treat it as live. When incidents happen, on-call SREs make wrong decisions based on old data. The fix: match the pipeline latency to the actual consumption pattern, not the one you assumed at design time. Sometimes this means moving to stream; sometimes it means changing the dashboard's honest label and setting user expectations. Whichever, the mismatch must be resolved deliberately.
Running a continuous streaming pipeline for a workload that only needs its output monthly is architectural theater with real cost. The Flink job runs 24×7 consuming compute, memory, checkpoint storage; the same rollup as a nightly Spark job runs for 40 minutes/month at ~1% the cost. Streaming justifies its complexity budget when the freshness is used. If the output is consumed on a schedule the batch scheduler can serve, batch is the simpler answer. "Everything is a stream" is a design philosophy, not a cost-benefit analysis.
Streaming state that isn't bounded — no TTL, no session-close event, no windowed eviction — grows forever until the processor exhausts memory. This is one of the most subtle and expensive failure modes of stream processing; the pipeline works fine for months, then starts OOM-ing under load, then no amount of scaling helps because the state is the problem. The fix: every keyed state must have an eviction strategy — TTL, window close, external checkpointing to a persistent store, or explicit cleanup based on domain semantics. M.41 (stateful streams) is where this discipline lives.
Micro-batch is not streaming. It's a batch job with a very short interval — usually 1-30 seconds. For workloads where "eventually consistent within a minute" is acceptable (product-view counts, hourly analytics), it's a great fit: simpler than true streaming, faster than nightly batch. But for genuine sub-second workloads (fraud detection, real-time alerting), micro-batch's minimum latency is the interval size, which is way too slow. The fix: pick micro-batch when you actually want batch-simplicity with small windows; pick true streaming (Flink, Kafka Streams) when you need sub-second latency. Micro-batch as "streaming lite" is a category error.
Lambda architecture writes every transformation twice — once in batch, once in stream — and every business rule change touches both implementations. Over time the two implementations drift; edge cases handled by batch aren't handled by stream (or vice versa); QA time doubles; the org accumulates a "which one is right?" tax that never goes away. The fix: migrate to Kappa (§03) — one code path, one truth, historical replay via log retention. Some workloads still justify Lambda (regulatory batch, complex ML training), but greenfield systems should default to Kappa. The 2010s embraced Lambda; the 2020s learned its cost.
The composite lesson of Phase H's opening module: the batch/stream choice is per-workload, driven by consumption pattern, and expensive to change after the fact. Getting it right at design time — matching pipeline latency to actual user consumption, matching complexity budget to actual freshness value, being deliberate about state boundaries — is the specific skill data engineering has developed as the field matured. The next four modules of Phase H (M.40 through M.43) build depth on the stream side of the model because that's where the interesting operational complexity lives: how to define time-windows over an unbounded stream (M.40), how to manage stateful processing safely (M.41), how to source streams from operational databases via CDC (M.42), and how to make stream state queryable at real-time analytics scales (M.43). The batch side is stable, well-understood infrastructure; the stream side is where the field is still consolidating knowledge, and where the highest-leverage engineering decisions of the next decade will happen.
The terms every "should this be batch or streaming?" architecture doc, every "why is the dashboard stale?" postmortem, every Kappa-migration RFC assumes you already know.
Test the batch/stream intuition. Click an answer; explanation drops in instantly.
Perfect. The batch model, the stream model, Lambda vs Kappa, the five anti-patterns — all yours. Phase H opened with the foundational choice; next up is windowing (M.40) — how to slice unbounded streams into meaningful groups so the aggregations you want become computable.
The foundational data-processing distinction that shapes every pipeline in Phase H.
Batch processes a fixed input in one job — deterministic, replayable, high throughput, hours of latency. Stream processes an unbounded input continuously — stateful, event-time-aware, milliseconds of latency, meaningful complexity budget. Same problem, different shapes.
Freshness needs, throughput needs, correctness needs — these determine the fit. Historical aggregations want batch. Real-time fraud wants stream. Product counters want micro-batch. Neither model is universally right; deliberate per-workload matching is the skill.
Lambda architecture wrote every transformation twice; the tax was real. Kappa uses one stream-only path with historical replay via log retention. Modern greenfield platforms default to Kappa; batch retained only where its properties (determinism, audit, throughput) genuinely justify a separate model.