The specific engineering that turns "we process events in nightly batches" into "we react to events in 200ms end-to-end" — with exactly-once semantics, ordering guarantees, and horizontal scale. Kafka + Redpanda — partitioned commit log, distributed broker cluster with local storage, thread-per-core in Redpanda. Pulsar — segmented architecture with BookKeeper storage tier, independent scaling of compute + storage, native geo-replication. Kinesis + Pub/Sub + Event Hubs — cloud-managed, serverless, per-shard billing, deep cloud integration. On top: Flink, Kafka Streams, Materialize for stateful stream processing with checkpointing, watermarks, and windowed aggregations. Understanding these architectures — and how they compose with CDC, real-time analytics, and event-driven microservices — is Expert-tier competence for modern data infrastructure.
Streaming systems process data-in-motion, not data-at-rest — a fundamentally different architectural paradigm from OLTP, OLAP, TSDB, vector, and search. Consider what real event infrastructure does: LinkedIn ingests trillions of events per day (profile updates, connection changes, feed activity, messaging) into Kafka; each event fans out to 10-100 consumers (feed ranking, notifications, analytics, search indexing, ML feature stores, CDC-to-warehouse). Uber routes millions of rider/driver location updates per second through Flink for matching + surge pricing. Netflix processes billions of viewing events through Kafka + Flink for real-time recommendations. Traditional request-response databases would fail: (a) point-to-point integration between N producers and M consumers creates N×M coupling (impossible at scale); (b) polling databases for changes has minute-scale latency and heavy load; (c) no natural way to replay events for new consumers or bug fixes; (d) no support for stateful streaming computations (windowed aggregations, joins across streams). Purpose-built streaming: (i) partitioned commit log as the durable ordered record of events (Kafka topics with N partitions, Kinesis shards, Pulsar segments); (ii) publish-subscribe model decoupling producers from consumers (consumers subscribe independently, replay from any offset); (iii) consumer groups for parallel consumption with automatic rebalancing; (iv) exactly-once semantics via transactional producers + idempotent consumers + Chandy-Lamport checkpoints in stream processors; (v) stream processing engines (Flink, Kafka Streams, Materialize) for stateful computations over unbounded streams. Understanding these systems — and how they compose real-time analytics, event-driven microservices, and CDC pipelines — is Expert-tier competence for modern data infrastructure.
user_events) is split into N partitions (independent ordered append-only sequences); each event routed to a partition by key (for ordering) or round-robin (for parallelism); each partition replicated across brokers for durability. Consumers independently track their read position (offset) per partition — enabling replay, multiple concurrent readers, and horizontal scale via consumer groups. Consumers are independent: real-time analytics (Flink windowed aggregations), downstream microservices (event-driven architecture), search indexers, warehouse sinks (Iceberg/Snowflake CDC), ML feature stores. Each consumer scales independently; new consumers can be added later replaying from an earlier offset. The delivery semantics problem — at-most-once (fire and forget), at-least-once (retry may duplicate), exactly-once (transactional producer + read-committed consumer, or Chandy-Lamport checkpoints in stream processors) — is the specific engineering complexity that separates production streaming from toy prototypes. Kafka added exactly-once semantics in 2019; Flink has supported it since 2016 via checkpointing. Three architectural families: (a) KAFKA-FAMILY — partitioned commit log with distributed broker cluster; Apache Kafka is industry default; Redpanda is C++ reimplementation targeting Kafka wire protocol with thread-per-core architecture, no ZooKeeper (or KRaft in modern Kafka), no JVM. (b) PULSAR SEGMENTED — separates broker layer (serving) from BookKeeper storage layer; independent scale of compute + storage; native geo-replication; tiered offload to S3. (c) CLOUD-MANAGED — Kinesis Data Streams (AWS), Google Cloud Pub/Sub, Azure Event Hubs; serverless (no broker ops); per-shard/partition billing; deep integration with cloud services. Understanding when each fits — and how to compose them with stream processing engines — is Expert-tier competence for modern event-driven infrastructure.The specific engineering task M.61 addresses is understanding how streaming systems process data-in-motion, how the partitioned commit log abstraction enables decoupling and horizontal scale, how exactly-once semantics work end-to-end, and how three architectural families (Kafka-family, Pulsar segmented, cloud-managed) fit different operational contexts. The critical insight: streaming is not "just async messaging with retention." It\'s a specific architectural paradigm with four defining properties: (a) Durability + replay — events persist for retention window (hours to years); consumers can rewind to any offset. Replaces point-to-point ETL entirely. (b) Decoupling — producers write to log without knowing consumers; consumers read without knowing producers; new consumers added anytime. Standard event-driven architecture foundation. (c) Ordering + parallelism trade-off — global ordering across a topic is expensive; per-partition ordering is cheap. Partition key strategy determines ordering guarantees + parallelism ceiling. (d) Stateful processing over unbounded streams — Flink, Kafka Streams, Materialize compute windowed aggregations, streaming joins, incremental view maintenance. Checkpointing gives exactly-once. Watermarks handle late-arriving data. Standard modern stream processing discipline. Each of the three architectural families implements variants of these properties with different operational trade-offs: Kafka-family is the industry default with the largest ecosystem; Pulsar-family separates compute + storage for independent scaling; cloud-managed reduces operational burden. Understanding when each fits is Expert-tier data infrastructure competence.
Each earlier attempt fails specifically. Point-to-point HTTP creates N×M coupling that becomes unmaintainable. Traditional message brokers (RabbitMQ) lack replay + scale + stream processing. Custom "we\'ll build our own" reinvents Kafka poorly over months to years. The Expert pattern: use purpose-built streaming — Kafka-family for industry-standard ecosystem, Pulsar for independent compute + storage scaling, cloud-managed for zero-ops. Compose with stream processing engines (Flink for standalone, Kafka Streams for library-embedded, Materialize for streaming SQL). Add CDC (Debezium) + Schema Registry + connector ecosystem. Understanding the composition is the specific engineering competence for modern event-driven infrastructure. §02 covers partitioned log internals (broker architecture, replication, consumer groups). §03 covers the three architectures + stream processing engines. §04 lets you explore all three architectures across three use cases.
The historical arc of streaming architectures is specifically the story of how the partitioned commit log abstraction emerged and reshaped data infrastructure. 2011: Kafka at LinkedIn (Jay Kreps, Neha Narkhede, Jun Rao). Built to replace point-to-point ETL across their microservices. Core insight (Kreps\' famous "The Log" essay): a distributed commit log is the fundamental abstraction that decouples producers + consumers + supports replay + enables stateful processing. Opened first Apache incubator in 2011. 2013: Kafka open-sourced under Apache. Rapid adoption at Netflix, Airbnb, Uber, Spotify. 2014: Confluent founded (Kreps + Narkhede + Rao). Commercial Kafka distribution + managed cloud + ecosystem (Connect, Schema Registry, ksqlDB). 2015: Amazon Kinesis GA. AWS-managed alternative to running Kafka; simpler ops but AWS lock-in. Google Cloud Pub/Sub similar. 2015: Apache Samza (LinkedIn). Early stream processor purpose-built for Kafka. Superseded by Flink and Kafka Streams. 2016: Flink 1.0. Data Artisans (later Ververica, now Alibaba). True event-time semantics via watermarks; exactly-once via Chandy-Lamport checkpoints; stateful streaming with RocksDB. Became the standard heavy-duty stream processor. 2016: Kafka Streams library. Library (not framework) for JVM apps; exactly-once via Kafka transactional API. Popular for simpler cases + apps already on Kafka. 2018: Apache Pulsar (Yahoo → Apache). Segmented architecture — brokers separate from BookKeeper storage; native multi-tenancy, geo-replication, tiered storage. Popular where Kafka\'s tightly coupled broker + storage was operationally constraining. 2019: Kafka exactly-once semantics production-stable. Transactional producers + read-committed consumers. Major maturity milestone. 2020: Redpanda (Vectorized, now Redpanda Data). C++ reimplementation of Kafka wire protocol; thread-per-core; no JVM, no ZooKeeper (KRaft-like); 10× less hardware for equivalent throughput. Popular where Kafka ops burden matters. 2020: Materialize. Streaming SQL via differential dataflow (Frank McSherry). Incrementally maintains SQL views over event streams; PostgreSQL wire protocol; dashboard/BI-friendly streaming. New paradigm for real-time analytics. 2020: Kafka KRaft (KIP-500). Removes ZooKeeper dependency; simpler ops. Stable in 3.3+ (2022). 2021: Kafka Tiered Storage (KIP-405). Offloads old segments to S3/object storage; retention economics dramatically improved. Standard modern Kafka. 2022+: streaming becomes standard for real-time analytics, CDC, event-driven microservices. Debezium for CDC from Postgres/MySQL/MongoDB WALs; Iceberg + Kafka connectors for lakehouse ingestion; Materialize/RisingWave/Flink SQL for streaming analytics. 2024: streaming SQL matures. Flink SQL, Materialize, RisingWave all offer SQL-native streaming; ksqlDB (Confluent) part of the same convergence. Streaming increasingly accessible without deep Java expertise. The historical arc explains why streaming is its own architectural paradigm — the partitioned commit log abstraction turned out to be foundational, and modern data infrastructure composes streaming with storage, query, and analytics at every layer.
The partitioned commit log is the foundational abstraction of modern streaming — a durable, ordered, replicated sequence of events split into N partitions for parallelism. Understanding its internals is Expert-tier competence because the choice of partition key determines ordering guarantees, parallelism ceiling, hot-partition risk, and rebalancing behavior. The standard architecture: (a) Topic — logical event stream (e.g., user_events); (b) Partitions — physical shards, each an ordered append-only log; (c) Partition key — determines which partition an event lands in (usually via hash: partition = hash(key) % num_partitions); events with same key go to same partition = same partition ordering; (d) Brokers — servers that host partitions; each partition has a leader (accepts writes) + followers (replicas); (e) Replication factor — number of copies per partition (typically 3); (f) Consumer groups — logical subscribers; multiple consumers in a group share partitions (each partition assigned to exactly one consumer at a time); (g) Offsets — per-partition position tracking read progress. Kafka, Pulsar, Kinesis all implement variants of this architecture. The specific engineering choices — partition count, replication factor, key strategy, consumer group topology — determine throughput, latency, ordering, and failure behavior.
partition = hash(key) % num_partitions to route to a specific partition. Events with the same key always land in the same partition = same partition ordering. Different keys distribute across partitions for parallelism. Topic (like user_events) is logically one event stream; physically split into N partitions (typically 12-100 depending on scale); each partition is an ordered append-only log. Partition count determines maximum parallelism ceiling — a topic with 3 partitions can be consumed by at most 3 parallel consumers per consumer group. Choose partition count carefully at topic creation; changing later requires re-keying + reshuffling (expensive). Broker cluster: 3-N brokers (servers); each partition has a leader + N-1 followers (replication factor typically 3). Leader accepts writes; followers replicate. In-sync replicas (ISR) = followers up to date with leader. If leader dies, one of ISR promotes. Producers can choose acknowledgment level: acks=0 (fire and forget, fastest, no durability), acks=1 (leader ack, moderate durability), acks=all (all ISR ack, strongest durability, slower). Consumer groups: logical subscribers; each group has independent offset tracking. Multiple consumers in same group share partitions — each partition assigned to exactly one consumer at a time (partition assignment via consumer group coordinator). If a consumer dies, its partitions rebalance to remaining consumers. Multiple consumer groups on same topic = independent readers; analytics group can be at offset 12345 while warehouse sink is at offset 9821. Standard modern event processing pattern: many consumer groups read same topic for different purposes. Offsets: per-(consumer-group, partition), tracked in Kafka\'s __consumer_offsets internal topic. Consumers commit offsets after processing; on restart, resume from last committed offset. Manual offset management enables exactly-once patterns (commit offset only after side effect succeeds). Kinesis uses similar model: streams = topics, shards = partitions, sequence numbers = offsets. Pulsar uses topics + subscriptions (different subscription types: exclusive, shared, key_shared, failover) with segments managed by BookKeeper. All three implement the same fundamental abstraction with different operational trade-offs.Topic is logical event stream; split into N partitions for parallelism. Each partition is an ordered append-only log. Partition count determines maximum parallelism ceiling — choose carefully at topic creation. Standard Kafka topic pattern.
partition = hash(key) % num_partitions. Events with same key → same partition → same ordering. Key strategy critical: user_id gives per-user ordering; null key round-robins (no ordering). Hot key = hot partition. Anti-pattern §05.ii.
Each partition: 1 leader + N-1 followers (RF typically 3). Leader accepts writes; ISR replicates. Leader failure → ISR promotion (M.47 Raft-like protocol in KRaft). acks=all for strongest durability.
Consumers in same group share partitions (each partition → one consumer at a time). Group coordinator handles assignment + rebalancing. Multiple groups on same topic = independent readers. Standard parallelism pattern.
Per-(group, partition) offset tracks read position; stored in __consumer_offsets topic. Manual commit enables exactly-once (commit after side effect). Auto-commit (default) simpler but at-least-once semantics.
Time-based (delete segments older than N days) or size-based (bounded total). Log compaction (per-key latest wins) for state topics. Tiered storage (KIP-405, 2021) offloads old segments to S3 → cheap infinite retention.
The partition key strategy (mech item ii) deserves specific attention because it\'s the most common source of production streaming failures. Key choice determines: (a) Ordering guarantees — same key = same partition = ordered. Different keys = different partitions = unordered relative to each other. Order matters for: state machines per entity (order status transitions), balance updates per account, event sequences per user session. (b) Parallelism ceiling — maximum concurrent consumers per group = partition count. 12 partitions = 12 concurrent consumers max. Scale by adding partitions (but see next point). (c) Hot partition risk — if one key generates disproportionate traffic (e.g., 5% of users generate 60% of events), the partition holding those keys becomes bottleneck; other partitions idle. Standard failure mode. Fixes: (i) composite key (user_id + session_id) to shard hot users; (ii) key salting (append random suffix to hot keys); (iii) separate hot topic for known heavy hitters. (d) Rebalancing cost — adding partitions to increase parallelism doesn\'t redistribute existing data; only new events use new partitions. Consumers may need to handle mixed data during transition. Full rebalance = create new topic + consumers migrate. Understanding these specific trade-offs is Expert-tier competence. The general principle: partition key strategy is the single most important design decision when creating a Kafka topic; get it wrong and you\'ll pay for it through hot partitions, ordering violations, or expensive migrations for years.
The consumer group + offset semantics (mech items iv+v) deserve equal attention because they\'re where exactly-once semantics live. Consider three delivery modes: (a) At-most-once — consumer auto-commits offset before processing; if crash during processing, event lost. Fine for lossy telemetry where losing occasional samples is OK. (b) At-least-once (default) — consumer commits offset after processing; if crash between processing and commit, event reprocessed on restart → duplicate side effects. Fine when downstream is idempotent (upserting to database with primary key, incrementing atomic counter). Anti-pattern with non-idempotent side effects (sending emails, charging credit cards). (c) Exactly-once — either via Kafka transactional producer (produce + offset commit atomically in single transaction) + read-committed consumer, or via Flink checkpointing (Chandy-Lamport snapshots of all operator state + input offsets, atomically committed). Both approaches ensure each event affects downstream exactly once even under failures. Standard modern streaming discipline. Kafka Streams uses transactional producer approach; Flink uses checkpointing. Both are production-stable since 2019-2020. Understanding when each is required — and the specific engineering complexity — is Expert-tier competence.
Modern streaming infrastructure has three canonical messaging architectures + three canonical stream processing engines, each with specific fits. Messaging layer: (a) Kafka-family (Apache Kafka + Redpanda + Confluent Cloud) — partitioned commit log with distributed broker cluster; local disk storage; massive ecosystem. Industry default. (b) Pulsar-family — segmented architecture separating broker (compute) from BookKeeper bookies (storage); independent scaling; tiered S3; native multi-tenancy + geo-replication. (c) Cloud-managed — Kinesis Data Streams, Google Cloud Pub/Sub, Azure Event Hubs; serverless; per-shard or per-message billing; deep cloud integration. Processing layer: (i) Apache Flink — dedicated cluster; true event-time semantics via watermarks; exactly-once via Chandy-Lamport checkpoints; stateful with RocksDB; Java/Scala/Python/SQL. Standard for heavy-duty stream processing. (ii) Kafka Streams — library embedded in JVM apps; exactly-once via Kafka transactional API; simpler ops (no cluster). Standard when apps already run on JVM + Kafka. (iii) Materialize + RisingWave + ksqlDB — streaming SQL; incrementally maintain SQL views over event streams; PostgreSQL-compatible endpoints. New paradigm for real-time analytics + BI. Understanding when each fits is Expert-tier competence — because most companies at scale end up composing multiple options (Kafka + Flink for main pipeline; Kinesis for cloud-native subsystems; Materialize for real-time dashboards).
Original partitioned commit log; JVM, ZooKeeper (or KRaft), distributed broker cluster. Massive ecosystem. Industry default. LinkedIn, Netflix, Uber, Airbnb, most Fortune 500. Scales to trillions of events/day.
C++ reimpl of Kafka wire protocol. Thread-per-core (Seastar framework). No JVM, no ZK. 10× less hardware for equivalent throughput. Popular where ops burden matters — smaller teams, cost-sensitive workloads. Kafka clients work unchanged.
Segmented architecture: brokers (stateless) + BookKeeper (storage). Independent scaling. Native multi-tenancy, geo-replication, tiered S3 storage. Yahoo, Tencent, Comcast. Powerful architecture; smaller ecosystem + ops complexity vs Kafka.
Kinesis (AWS, per-shard billing, 1MB/s per shard), Pub/Sub (GCP, per-message, at-least-once, global), Event Hubs (Azure, Kafka-compatible endpoint). Zero broker ops. Deep cloud integration. Vendor lock-in + cost scale trade-off.
Flink: dedicated cluster, true event-time via watermarks, exactly-once via Chandy-Lamport checkpoints, RocksDB state, Java/Python/SQL. Standard heavy stream processing. Kafka Streams: library embedded in JVM apps; simpler ops when apps already run on Kafka.
Streaming SQL — incrementally maintain SQL views over event streams. Materialize (differential dataflow, PostgreSQL wire protocol, Frank McSherry), RisingWave (similar, Rust), ksqlDB (Kafka-native). Real-time dashboards + BI without Java stream code.
The Kafka-family architecture (mech items i+ii) is the industry default for good reason. Specifically: (a) Apache Kafka: distributed broker cluster (typically 3-30 brokers per cluster); each broker holds partitions + serves reads/writes + persists to local disk; ZooKeeper (Kafka 2.x) or KRaft (Kafka 3.x+) for metadata coordination. Data organized in topics with N partitions; partitions replicated across brokers (RF typically 3); leader per partition serves reads/writes; followers replicate. Extensive ecosystem: (i) Kafka Connect — pluggable framework for source (into Kafka) + sink (out of Kafka) connectors. 200+ community connectors: Debezium (Postgres/MySQL/MongoDB CDC), JDBC, S3, Iceberg, Elasticsearch, Snowflake, MongoDB, and hundreds more. Standard for "get data into Kafka" and "get data out of Kafka." (ii) Schema Registry (Confluent open-source) — centralized store for Avro/Protobuf/JSON Schema; producers register schemas; consumers fetch schemas by ID from message header; evolution rules (backward, forward, full compatibility) enforced. Standard for schema evolution in production Kafka. (iii) Kafka Streams — library-embedded stream processing; exactly-once via transactional API. Standard for JVM apps that want stateful stream processing without separate cluster. (iv) ksqlDB — SQL over Kafka; simple continuous queries. Convenient for basic streaming SQL. (b) Redpanda: full Kafka wire protocol compatibility (existing clients work unchanged); C++ implementation using Seastar framework (thread-per-core, shared-nothing, async I/O); no JVM (no GC pauses); no ZooKeeper (Raft-based metadata). Result: 10× less hardware for equivalent throughput; single-digit-ms latency vs 20-50ms typical Kafka; simpler ops. Popular at Vercel, Discord (some pipelines), many mid-scale companies where Kafka\'s JVM + ZK operational burden matters. Understanding the Kafka-family internals + ecosystem is Expert-tier competence — because it\'s the industry default and appears in every serious modern data platform.
The stream processing engines (mech items v+vi) deserve equal attention because they\'re where the actual event processing happens. Specifically: (a) Apache Flink: dedicated cluster (JobManager + TaskManagers); handles unbounded event streams; true event-time semantics via watermarks (marks progress of event time separately from processing time — enables correct handling of late-arriving events); stateful with RocksDB per operator (state can be terabytes); exactly-once via Chandy-Lamport checkpointing (periodic snapshots of all operator state + input offsets, atomically committed to durable storage; on failure, restore to last complete checkpoint and replay from that offset). Rich API: Java/Scala DataStream API, Python (PyFlink), SQL (Flink SQL). Standard for heavy-duty stream processing at Uber, Netflix, Alibaba, ByteDance. Handles complex event-time patterns (windowed joins, session windows, watermark-triggered emissions). Alibaba processes 4 billion events/sec at peak during Singles\' Day sale. (b) Kafka Streams: library-embedded (no cluster to manage); JVM applications include the library; exactly-once via Kafka\'s transactional producer API; state stored in RocksDB per instance + changelog topic in Kafka for recovery; scales by running multiple instances (Kafka consumer group coordinates partition assignment). Simpler than Flink for JVM apps already using Kafka; less powerful for very complex event-time patterns. Standard for embedded stream processing. (c) Materialize + RisingWave: streaming SQL — user writes SQL views, engine incrementally maintains them as new events arrive. Under the hood: Materialize uses differential dataflow (Frank McSherry\'s research at Microsoft/ETH); RisingWave uses similar techniques in Rust. Result: PostgreSQL wire protocol (existing SQL tools work); dashboards can query the latest state directly (results are always current); no Java stream code needed. New paradigm for real-time analytics + BI. Popular for use cases where "the SQL is the streaming logic" — inventory monitoring, feature stores, real-time dashboards. (d) ksqlDB (Confluent): SQL over Kafka; simpler than Flink or Materialize but less powerful. Good for basic use cases. (e) Apache Beam: unified batch + stream API; portable across runners (Flink, Spark, Dataflow). Best when you need portability. Understanding when each fits is Expert-tier competence — the choice depends on stream complexity, existing team expertise, and operational context.
The Pulsar and cloud-managed options (mech items iii+iv) address specific gaps. Pulsar: the segmented architecture separating brokers from BookKeeper storage was designed at Yahoo to solve specific problems Kafka had at their scale: (a) INDEPENDENT SCALING — need more storage but not more compute? Add bookies without adding brokers. Conversely, add brokers for more consumer throughput without duplicating storage. Powerful for irregular workloads. (b) MULTI-TENANCY — native support for isolated namespaces, quota enforcement per tenant, resource isolation. Yahoo runs Pulsar as a shared service across hundreds of internal teams. (c) NATIVE GEO-REPLICATION — cross-region replication built-in; no MirrorMaker (Kafka\'s external tool). Cleaner architecture. (d) TIERED STORAGE — offload old segments to S3 automatically; retention economics dramatically better than Kafka pre-KIP-405 (though Kafka Tiered Storage from 2021 closes this gap). Weakness: smaller ecosystem than Kafka; fewer teams have expertise; more components to operate. Real-world adoption meaningful but smaller than Kafka. Choose Pulsar when compute/storage independence + multi-tenancy + native geo-replication matter more than ecosystem breadth. Cloud-managed: (a) KINESIS DATA STREAMS — AWS-native; per-shard billing ($36/shard/month + PUT payload); 1MB/s write, 2MB/s read per shard; 24h retention default (7d configurable, up to 365d premium). Deep AWS integration (Kinesis Data Analytics for managed Flink, Lambda triggers, Firehose for automatic delivery to S3/Redshift). Choose when already deep in AWS. (b) GOOGLE CLOUD PUB/SUB — globally distributed; auto-scaling; per-message pricing; at-least-once by default (exactly-once optional in some modes). Push or pull subscriptions. Great for globally distributed workloads. (c) AZURE EVENT HUBS — Kafka-compatible endpoint (existing Kafka clients work); deep Azure integration. Choice when in Azure. General principle: cloud-managed reduces ops burden but locks you into the cloud + cost scales linearly (10× traffic = 10× bill, no economies of scale). Kafka self-hosted has higher ops burden but better economics at scale + multi-cloud portability. Choose based on team size, scale, and cloud strategy.
Below: each of three streaming architectures (Kafka-family (Apache Kafka / Redpanda) · Pulsar segmented · Cloud-managed (Kinesis / Pub-Sub / Event Hubs)) evaluated against three canonical use cases (Real-time analytics · Event-driven microservices · CDC pipelines). Watch how each architecture fits or fails each use case — Kafka-family dominates industry-standard workloads with its ecosystem, Pulsar excels at multi-tenant + independent-scaling scenarios, and cloud-managed wins on zero-ops for cloud-native systems. The off-diagonals show where the wrong choice produces measurably worse cost, latency, or ops complexity. The takeaway: match architecture to workload characteristics; most companies at scale compose multiple options.
The failure modes of streaming infrastructure are the specific mechanisms by which "our event pipeline is broken" turns into "we\'re dropping 5% of events" or "our exactly-once claims aren\'t actually exactly-once." Each anti-pattern is a real production pattern; Expert engineers avoid them by matching architecture to workload, choosing partition keys carefully, understanding delivery semantics rigorously, and knowing when Kafka is (and isn\'t) the right tool. Recognizing these saves months of "why is our streaming broken" debugging.
Kafka is a log, not a database. It excels at append-only ordered event delivery with pub-sub decoupling; it categorically fails as a queryable store. Specifically: (a) NO INDEXES — Kafka has no secondary indexes; scanning by non-key attributes = full topic scan. (b) NO JOINS — cannot efficiently join across topics; consumers must materialize joined state elsewhere. (c) NO SELECTIVE READS — must read entire partitions from committed offset. Filtering happens in consumer, not in broker (except with ksqlDB which builds new topics with filtered data). (d) NO POINT LOOKUPS — cannot fetch "event with ID X" — must scan. (e) RETENTION ECONOMICS — even with tiered storage (KIP-405), keeping all events forever costs money; Kafka isn\'t optimized for historical query access patterns. The fix: (i) USE KAFKA AS EVENT TRANSPORT + MATERIALIZE VIEWS IN QUERYABLE STORES. Consume from Kafka → maintain state in Postgres/ClickHouse/Elasticsearch → query the derived state. Standard "streaming ETL" pattern. (ii) FOR HISTORICAL EVENT SEARCH — use Iceberg or lakehouse (Kafka → Iceberg via connector; query via Trino/Spark/DuckDB). Kafka + Iceberg is standard modern architecture. (iii) FOR REAL-TIME MATERIALIZED VIEWS — use Materialize or RisingWave (streaming SQL over Kafka; PostgreSQL-compatible queries). (iv) SET APPROPRIATE RETENTION — 7-30 days typical for operational topics; longer only if replay + reprocessing matters (in which case Iceberg is better for historical). The general principle: Kafka is a durable ordered log for event transport + decoupling; databases and lakehouses are for query. Compose them via streaming ETL. Anti-pattern §05.i.
Partition key strategy determines both ordering guarantees and load distribution. Skewed key distributions create hot partitions that bottleneck the entire topic. Consider the mechanism: partition = hash(user_id) % 12. If one user_id generates 30% of traffic, its partition gets 30% of load — while other partitions get proportionally less. Broker holding that leader partition CPU/disk-saturates; consumer for that partition falls behind; downstream sees increasing lag for that partition. Meanwhile other partitions are idle. Adding partitions doesn\'t help — hot key still maps to one partition. Standard failure. The fix: (a) COMPOSITE KEY — hash(user_id + session_id) or hash(tenant_id + user_id). Spreads a single hot entity across multiple partitions. Sacrifice: no total ordering per user across sessions (may or may not matter). (b) KEY SALTING — for known hot keys, append random suffix: user123-{0..N}. Distributes hot key across N partitions. Consumer must handle N sub-keys. (c) SEPARATE TOPIC FOR HEAVY HITTERS — route known heavy hitters to dedicated topic with different partition strategy. Multi-tier routing. (d) INCREASE PARTITION COUNT + REPARTITION — adding partitions only helps for future events; existing data stays. Full rebalance requires new topic + consumer migration. Expensive. (e) MEASURE — Kafka metrics per partition: records-lag, bytes-in-rate, bytes-out-rate. Alert on skew. (f) DESIGN FOR SKEW UP FRONT — most real data is skewed (Zipfian: 20% of entities generate 80% of events). Assume skew; design partition strategy accordingly. Standard modern streaming discipline. The general principle: partition key is the most important design decision when creating a topic; skewed keys create bottlenecks that no amount of scaling fixes. Anti-pattern §05.ii.
At-least-once delivery (Kafka default) means "each event is delivered at least once" — under failures, events get re-delivered. If your side effects are non-idempotent (sending emails, charging credit cards, publishing to Twitter), duplicates cause visible problems. The specific mechanism: (a) consumer polls batch of events; (b) processes each event (sends email, charges card); (c) commits offset. If crash between (b) and (c), consumer resumes from previously committed offset → re-processes batch → sends duplicate emails. This is not a bug in your code — it\'s the expected semantics of at-least-once delivery. The fix: (a) MAKE SIDE EFFECTS IDEMPOTENT WITH DEDUPE KEY — include unique event ID in email send request; email service checks (has email with dedupe_key X already been sent to user Y in last 24h?) before sending. Same pattern for payment: unique idempotency key per payment request. Standard modern side-effect design. (b) USE KAFKA EXACTLY-ONCE SEMANTICS (EOS) — transactional producer + read-committed consumer + atomic offset commit within transaction. All-in-Kafka side effects (produce to output topic) become exactly-once. Doesn\'t help for external side effects (email service is not part of transaction). Fine when the "side effect" is producing to another Kafka topic. (c) USE FLINK EXACTLY-ONCE VIA CHECKPOINTING — Flink checkpoints all operator state + input offsets atomically; on failure, restore to checkpoint + replay from checkpoint offset. Works for external systems via two-phase commit sink connectors (which coordinate with external system\'s transaction). More complex but works for external side effects. (d) TRACK PROCESSED EVENT IDS IN DEDUP TABLE — consumer maintains "processed event IDs" in Postgres/Redis; skips already-processed IDs. Standard pattern when EOS isn\'t available. (e) DESIGN FOR IDEMPOTENCE FROM DAY ONE — every event has unique ID; every side effect uses idempotency key or dedupe table. Standard modern streaming discipline. The general principle: at-least-once is not exactly-once; non-idempotent side effects require explicit dedupe or Flink checkpointing; understanding delivery semantics rigorously is Expert-tier competence. Anti-pattern §05.iii.
orders. Sometimes it processes an update before the create. We assumed Kafka events arrive in order — apparently not?"Kafka guarantees ordering per partition, not per topic. Events with different partition keys may be processed in any order relative to each other. Specifically: (a) MECHANISM — a topic with 12 partitions has 12 independent ordered logs. Two events with different keys land in different partitions; consumers process each partition independently and in parallel. No ordering guarantee across partitions. If your ordering is `order_created(id=A)` from producer 1 → then `order_updated(id=A)` from producer 2, and both events use `order_id` as key, they land in the same partition = ordered. But if events use different keys (say create uses `customer_id`, update uses `order_id`), they land in different partitions = potentially processed out of order. Common mistake. (b) OTHER FAILURE MODES: producer retries reorder events (mitigated by enable.idempotence=true which uses producer sequence numbers); consumer parallelism within partition (rare — usually one thread per partition); replaying from earlier offset while processing current. The fix: (a) USE CONSISTENT PARTITION KEY per entity — all events for `order_id=X` use `order_id` as key. Same partition → ordered. Standard pattern. (b) ENABLE IDEMPOTENT PRODUCER — enable.idempotence=true — Kafka assigns producer sequence numbers; broker rejects duplicates + reorders. Enabled by default in Kafka 3.x+. (c) DESIGN EVENT SCHEMA WITH VERSIONING — every event has version number; consumer keeps latest version per entity; older versions are ignored. Handles out-of-order gracefully. Standard event-sourcing pattern. (d) EVENT SOURCING — store all events; project current state deterministically. If you replay in different order, some events "no-op" (e.g., "update to state X" when already at state Y > X). Requires careful design but eliminates the ordering problem. (e) MEASURE — track out-of-order event rate; investigate + fix. Standard modern streaming discipline. The general principle: Kafka provides per-partition ordering only; total ordering across topic is expensive and rarely necessary; design consumers to handle per-partition ordering with idempotence or event-sourcing. Anti-pattern §05.iv.
Building a distributed event log with exactly-once semantics, consumer groups, replication, and stream processing ecosystem is PhD-level distributed systems work. Kafka encapsulates decade+ of engineering; reinventing it typically takes years and yields worse results. The specific limitations of "Kafka on Postgres" (or "Kafka on DynamoDB", or "Kafka on S3"): (a) POLLING LATENCY — you can\'t match Kafka\'s push semantics with polling. Latency floor is polling interval. (b) NO EXACTLY-ONCE — must build transactional coordination manually. Getting exactly-once right is famously hard (LinkedIn/Confluent spent years). (c) NO CONSUMER GROUPS — must build partition assignment + rebalancing + heartbeat protocol. Non-trivial distributed systems work. (d) NO STREAM PROCESSING ECOSYSTEM — Flink, Kafka Streams, Materialize all expect Kafka-like semantics. Roll-your-own means rebuilding them too. (e) NO CONNECTORS — Debezium (CDC), S3 sink, Iceberg sink, and 200+ others all target Kafka. Roll-your-own = write connectors. (f) NO SCHEMA REGISTRY — need centralized schema management. Kafka has Confluent Schema Registry. (g) OPPORTUNITY COST — 8+ months of engineering time that could have been product work. Standard "not-invented-here" trap. The fix: (a) USE KAFKA (or Redpanda or Pulsar or Kinesis) — proven, batteries-included, massive ecosystem. Weeks to production vs years for custom. (b) IF CUSTOM IS TRULY NEEDED — only justified at truly extreme scale (LinkedIn, Meta, Google) where cost + specific constraints matter. Even then, borrow architecture from Kafka; don\'t reinvent from scratch. (c) IF EXISTING INVESTMENT — extract the log semantics into a well-defined interface; over time, replace with Kafka behind the interface. Gradual migration. (d) MEASURE the ROI of custom vs Kafka — engineering cost + operational cost + lost feature velocity. Almost always Kafka wins. The general principle: log-based streaming is a solved problem; use existing systems; only reinvent when truly justified (rare). Anti-pattern §05.v.
The composite pattern across all five is that streaming failure modes reflect specific engineering understanding gaps that Expert-tier competence addresses. Kafka-as-database misses that Kafka is transport, not query — compose with materialized views + lakehouse. Hot partitions from bad key strategy miss that skewed data needs composite keys or salting. At-least-once with non-idempotent side effects misses that delivery semantics + side effects must be jointly designed. Global ordering assumption misses that Kafka guarantees per-partition ordering only. Reinventing Kafka poorly misses that it\'s a decade+ of engineering that\'s hard to match. Each has specific fixes: (a) Kafka + Iceberg + Materialize for query; (b) composite keys + salting + measurement; (c) idempotent design + EOS + Flink checkpointing; (d) consistent partition keys + idempotent producer + event sourcing; (e) use Kafka, don\'t reinvent. Getting streaming architecture right is the specific engineering discipline that turns "our event pipeline is fragile" into "we ingest 10M events/sec exactly-once with per-key ordering guarantees and materialize dashboards in 200ms."
The terms that show up in every streaming design review, every Kafka capacity plan, every event architecture discussion.
__consumer_offsets topic in Kafka. Manual commit enables exactly-once (commit after side effect). Auto-commit is at-least-once. Standard for replay + progress tracking.Test the streaming understanding. Click an answer; explanation drops in instantly.
Perfect. Partitioned log internals, exactly-once semantics, Flink checkpointing, Kafka vs Pulsar vs cloud-managed — the specific engineering discipline for modern streaming infrastructure. Next: M.62.
The composite understanding that turns "our event pipeline is broken" into "we chose Kafka for the industry-standard ecosystem, Debezium for CDC, Flink for stateful processing, and Materialize for real-time dashboards — with exactly-once semantics end-to-end."
Not RabbitMQ (no replay, scale ceiling). Not point-to-point HTTP (N×M coupling). Not custom-built (years of PhD-level distributed systems work). The partitioned commit log (Kafka topics + partitions, Pulsar segments, Kinesis shards) is the foundational modern abstraction — durable, ordered, replicated, replayable, decoupled.
Partition key determines ordering + parallelism + hot-partition risk. Choose wisely: composite for skewed data, salting for known heavy hitters, event_id or entity_id typically. Delivery semantics: at-least-once with idempotent consumers is default; exactly-once via Kafka EOS or Flink checkpoints when non-idempotent side effects. Design these together from day one.
Kafka (or Pulsar or Kinesis) for transport. Flink or Kafka Streams for stateful processing. Materialize/RisingWave for streaming SQL. Debezium for CDC. Iceberg/lakehouse for historical query. Standard modern data platform composes all layers. Kafka alone is transport; the value comes from the composition.