Impossible in the strict theoretical sense — Two Generals proved that in 1975. Achievable in the pragmatic sense — at-least-once delivery plus idempotent consumers plus the transactional outbox. The stack of patterns that has been quietly holding up every guarantee Phase F promised.
Two armies on hilltops on either side of a valley. The enemy is in the middle. If both armies attack at dawn, they win. If only one attacks, that one is destroyed. The generals can only communicate via messengers who must cross the valley — and messengers can be captured. General A sends: "attack at dawn." Did General B receive it? B must send an acknowledgment. Did A receive the acknowledgment? A must ack the ack. Did B receive that? The regress is infinite. No finite exchange of messages can give both generals certainty that the other will attack.
This is a 1975 result by Jim Gray at IBM, formalized by others in the years after. It's an impossibility proof, and it's what every distributed system runs into the moment a network is involved. Substitute "General A" for "producer" and "General B" for "consumer" and the picture is exact. The producer sends a message; the consumer processes it; the consumer sends an ack; the producer receives it. Any of these steps can fail. No protocol using an unreliable network can guarantee both "the message was delivered" AND "the message was delivered exactly once."
If exactly-once is impossible, why does every message broker vendor claim to support it? Because "exactly-once" has been quietly redefined. The pragmatic definition — the one that actually ships in production systems — is "at-least-once delivery + idempotent consumers = effectively-once effects." Duplicates still happen in the network layer; the consumer detects and discards them; the observable business effect is exactly-once even though the delivery mechanism is at-least-once. This is not a workaround. It's the only correct engineering response to the impossibility result.
Everything in Phase F has been quietly assuming this stack works. Kafka's at-least-once delivery (M.26) plus consumer group offsets — assumes consumers are idempotent. Event sourcing (M.28) replaying events into projections — assumes projections are idempotent. CQRS command handlers (M.29) receiving retried commands — assume idempotent handlers. Saga compensations (M.30) that might run twice on network blips — assume idempotent compensations, or they corrupt state. This module makes explicit what has been implicit: every event-driven pattern rests on idempotency. Once you understand this layer, everything else in Phase F snaps into a coherent picture.
The three delivery semantics — introduced in M.26 as "the three promises brokers make" — deserve a second look now that we've named the impossibility. Only two of them are real; the third is the sleight of hand that makes production systems work.
Producer sends and doesn't wait for ack. Zero retries. Message might be lost if the network drops it. Never duplicated because there's no resend logic. UDP-style delivery for the messaging layer.
Producer resends on ack timeout. Consumer might process the same message twice if the ack was slow rather than lost. This is Kafka's default and the default of every serious broker. Never lost, sometimes duplicated.
The broker still delivers at-least-once; the consumer detects duplicates via an idempotency check and discards them. Observable business effect is exactly-once; delivery mechanism is at-least-once. This is what "exactly-once" means in every production system.
The equation that runs the industry: at-least-once delivery + idempotent consumer = effectively-once effect. This is not a compromise or a workaround; it's the mathematically-correct engineering response to the Two Generals impossibility. Duplicates will happen in the delivery layer — the network is unreliable, retries are necessary, so duplicates are unavoidable. Rather than trying to prevent them (impossible), we make them harmless. If applying an operation N times produces the same result as applying it once, duplicates stop mattering.
Some operations are already safe to repeat. SET x = 5 can run a thousand times; the state is the same. UPDATE users SET last_seen = '2024-...' where the value is deterministic. Recognizing which operations are naturally idempotent is free correctness; the trap is assuming they're rarer than they actually are.
Every request carries a unique ID — client-generated UUID, or a hash of the request payload. The handler stores processed IDs in a processed_requests table and checks it before acting. Stripe's API uses this pattern openly via the Idempotency-Key header. Storage grows unbounded unless you TTL it; the TTL must exceed max realistic retry window (usually 24h+).
Use the business primary key as the idempotency key. INSERT INTO orders (id, ...) VALUES (?, ...) ON CONFLICT (id) DO NOTHING. If the same order_id arrives twice, the second insert is a no-op. Cheapest possible dedup; works only when you have a natural unique identifier that's assigned before the message is sent (not database-generated).
Model the domain as a state machine with explicit transitions. An order goes pending → paid → shipped → delivered. Only pending → paid is allowed; a duplicate "mark paid" message on an already-paid order is a no-op (or a check that raises "already paid, ignoring"). The state machine itself enforces idempotency; belongs naturally with event sourcing (M.28).
One implementation trap worth calling out: the idempotency check and the state write must be atomic. If the check ("have I seen this key?") and the effect ("apply this operation") are separate operations, two duplicate messages can arrive concurrently, both fail the check, and both apply their effect. This is a classic race condition that turns your "idempotent" handler into a duplicated-effect one under load. The fix: do both in one transaction (SQL upserts with unique constraints; Redis SETNX; DynamoDB conditional writes). Idempotency is a property of the atomic operation, not just the intent.
Here's the problem that plagued event-driven systems for a decade: you need to atomically (1) update your database AND (2) publish an event about the update. In a monolith, both happen inside one SQL transaction. In a service that owns a database and publishes to Kafka, they're two independent systems with no shared transaction manager. Two things must both happen — but they can't happen together. Every naive resolution corrupts state in a specific failure mode.
Write to database; then send to Kafka. If the service crashes between the two, or Kafka is unreachable during publish, the DB has the state change but no event is emitted. Downstream services never learn. The system is silently inconsistent — this order exists in the orders table, but the shipping service was never notified. Data loss disguised as a happy DB write.
Send to Kafka; then write to database. If Kafka acks but the DB write fails, an event has been emitted about state that doesn't exist. Downstream services react to a ghost. Bills get generated for orders that were never actually created. Data corruption is worse than data loss.
The transactional outbox pattern solves this with a startlingly simple idea: use only one system to write to, then have a separate process forward the events. Inside a single database transaction, write both the business state (to the orders table) AND the event you want to publish (to an outbox table). Because both writes are in one transaction, they're atomic — either both commit or neither does. A separate poller then reads new rows from the outbox table and publishes them to Kafka, marking each row as sent after successful publish. If the poller crashes mid-work, the outbox row remains unsent and gets retried on the next poll. Delivery becomes at-least-once (retries can duplicate), so consumers must be idempotent — but atomicity between state and event is now guaranteed.
sent_at=NULL, gets picked up on next poll. Retries can produce duplicates in Kafka.event_id that flows through Kafka into the consumer. Consumers use it as the idempotency key. Duplicates from retries get discarded.The outbox pattern is the single most-implemented pattern in modern event-driven services — and it should be. It solves a genuinely hard problem with a trick that seems almost too simple: use the ACID guarantees you already have (one DB, one transaction) to bootstrap consistency across systems you don't control (Kafka, downstream consumers). The poller is separate infrastructure — a cron job, a change-data-capture stream (Debezium reading the outbox table as WAL entries), or a dedicated goroutine — but its correctness requirements are minimal: read pending rows, publish, mark sent, repeat. Frameworks that implement this pattern out of the box: Debezium's outbox event router, MassTransit's transactional outbox, Eventuous, and the Temporal SDK's activity model all essentially adopt it.
Pick a failure scenario. See how the naive "write DB then publish to Kafka" pattern handles it (usually badly) versus how the transactional outbox pattern handles it (usually correctly). The first scenario is a tie — both work on the happy path. The other three are where the outbox pattern earns its complexity.
Since Kafka 0.11 (2017), the broker has claimed to support exactly-once semantics — EOS. The claim is technically true, but heavily qualified. Understanding what EOS delivers and where it stops applying is essential — because most teams that adopt Kafka EOS without reading the fine print end up with a system that appears to work but silently fails in specific scenarios where EOS was never designed to help.
Every producer gets a unique Producer ID (PID) and a per-partition sequence number. The broker tracks the highest sequence number seen per PID/partition; if a producer retries and sends a duplicate, the broker sees the same sequence number and discards it. Duplicate detection at the broker layer, within a single producer session, for retries of the same message. Prevents producer-retry duplicates but not app-level duplicates.
Producers can wrap multiple writes across multiple partitions in a "transaction." Either all writes commit atomically or none do. Consumers with isolation.level=read_committed only see committed writes; aborted transactions are invisible. Enables atomic multi-partition writes — useful for read-process-write patterns that update several output topics. Doesn't extend beyond Kafka — a Kafka transaction can't include a write to your external DB.
Kafka Streams applications get "exactly-once processing" of the read-process-write cycle within Kafka. Consumer offsets and output writes are committed in the same transaction. If the app crashes mid-processing, the offset isn't advanced and no partial output is visible. Full EOS — for stream processing that reads from Kafka, transforms, and writes back to Kafka. The scope is everything-inside-Kafka; nothing else.
The moment your consumer writes to an external system — your PostgreSQL, your Redis, a third-party API, an email sender — Kafka's EOS guarantees stop cold. That external write isn't part of the Kafka transaction. If your consumer crashes between "receive event from Kafka" and "write to Postgres," EOS doesn't help you; the offset might get advanced (event skipped) or not advanced (event processed twice). Kafka's fine print says "exactly-once processing" — and the print gets finer if you keep reading: within Kafka. For anything outside Kafka, you're back to at-least-once delivery + idempotent consumer. Which is what the industry runs on anyway.
The pragmatic advice that emerges from a decade of Kafka in production: use idempotent producers unconditionally (they cost nothing and prevent a real class of bugs); use transactional producers when you have Kafka-only read-process-write pipelines (Kafka Streams applications benefit hugely); don't rely on transactional producers for consistency between Kafka and your database — that's what the outbox pattern is for. "Kafka EOS" is a real feature. It's just not the feature most people think it is.
The other lesson: every Phase F pattern converges on the same requirement. Brokers deliver at-least-once (M.26); consumers must be idempotent. Pub/sub can duplicate under redelivery (M.27); subscribers must be idempotent. Event sourcing replays events (M.28); projections must be idempotent. CQRS handlers get retried commands (M.29); handlers must be idempotent. Saga compensations can re-run (M.30); compensations must be idempotent. Six modules of distinct patterns, one underlying obligation. Now you can see why this module was the finale: everything in Phase F needs the ideas in this module to work.
The terms every Kafka doc, every "we implemented the outbox pattern" post-mortem, every Two Generals citation assumes you already know.
Idempotency-Key header is the canonical example.Test the exactly-once intuition. Click an answer; explanation drops in instantly.
Perfect. Two Generals, idempotency, outboxes, Kafka EOS — all yours. And with this, Phase F is done. Six modules of event-driven mastery.
The layer that makes every other event-driven pattern actually work in production.
Strictly impossible per Two Generals. Practically achievable as at-least-once + idempotent consumers = effectively-once. That equation runs the industry.
Atomically write business state + event to one DB in one transaction. Separate poller forwards to broker. Consumers dedup on event ID. The canonical solution to atomic state-and-event updates.
Idempotent producers, transactional producers, Streams EOS all work — inside Kafka's boundary. External writes are still at-least-once. The outbox pattern covers what EOS can't.