Five services need to do a thing together. None of them share a database. Any of them can fail. ACID can't help you here. The pattern that does — by accepting that "rollback" across services isn't undo, it's apologizing in business terms.
A customer hits "place order." Five services need to act in concert: Order Service creates the order, Payment Service charges the card, Inventory Service reserves stock, Shipping Service creates a label, Notification Service sends the email. In a monolith with one database, this is a single transaction: BEGIN, do five things, COMMIT. If any step fails, ROLLBACK undoes the rest. ACID handles it. We've trusted databases to do this for forty years.
But these five services don't share a database. Each owns its own. Each is deployed independently. The payment service uses Postgres; inventory is on DynamoDB; shipping calls a vendor API; notifications go through SendGrid. There is no single transaction that spans them. ACID gives you correctness within one database; it gives you nothing across them. And the moment you have more than one database in your business flow, ACID's guarantees stop where the database does. This is the central problem of distributed transactions — and it's the problem sagas solve.
The classic answer from the database community is two-phase commit (2PC): a coordinator asks every participant "can you commit?" (phase 1: prepare), and if all say yes, tells them all to commit (phase 2: commit). If any participant says no, all roll back. It's a beautiful protocol. It works perfectly when everyone is healthy. But it has two problems that make it unsuitable for service-based systems: it locks all participants during the protocol (no other transactions can touch the same data, halting throughput), and it blocks indefinitely if the coordinator crashes after phase 1 — participants don't know whether to commit or roll back, so they wait. 2PC is correct but slow and fragile. It works for tightly-coupled XA-enabled databases; it falls apart across loosely-coupled services.
Sagas take a different approach: give up on atomicity, embrace compensation. Split the long transaction into local transactions, each fully committed in its own service. If a later step fails, run compensating actions against the earlier services that already committed — "refund the payment", "release the inventory", "cancel the shipment." The system passes through inconsistent intermediate states, but every state is either valid or being actively corrected. The trade is real: you give up isolation and atomicity, you get availability and scalability. For most service-based systems, that trade is worth it.
The saga pattern comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem at Princeton, titled simply "Sagas." Their original concern wasn't microservices — it was long-running database transactions that held locks for hours, blocking other work. Their proposal: break the long transaction into a sequence of shorter local transactions, each immediately committed; if any step fails, run a corresponding compensating transaction against each completed step. The system gives up serializability between sagas in exchange for not holding locks across the whole flow.
The shape of the idea translates perfectly to services. Each step is a local transaction in one service: charge the card (Payment), reserve the stock (Inventory), schedule the shipment (Shipping). Each one fully commits before the next one starts. If step N fails, the saga runs compensating actions for steps 1 through N-1, in reverse order. The system is briefly inconsistent during execution — but eventually all in one of two valid states: completed or fully compensated.
Each step is a fully-committed transaction in one service's database. T₁ in Payment's Postgres; T₂ in Inventory's DynamoDB. No locks held across services. Once committed, the change is immediately visible to anyone reading that service.
For each Tᵢ there's a corresponding Cᵢ that semantically undoes it. Not a rollback — a new transaction that produces the opposite business effect. Refund the payment. Release the inventory. Cancel the shipment. The original commit isn't erased; a new compensating commit balances it.
Instead of compensating backward, some sagas retry forward indefinitely. Useful when a step is "almost always recoverable" (transient network errors). Combined with compensation: retry N times forward; only after N failures, give up and compensate. Modern saga frameworks (Temporal, Cadence) make this easy.
While the saga runs, the system is briefly inconsistent — payment has been taken but the order isn't yet "done"; stock is reserved but not yet shipped. Reads during this window see truth-in-progress. Sagas accept this in exchange for not locking everyone out. The UX has to account for it.
The crucial distinction from rollback is worth saying twice. A saga compensation isn't undo. It's a new fact about the world: "refund issued", "stock released back to pool", "shipment canceled." The original event still happened; you can't erase it from the audit log. Compensations have to be designed as semantically meaningful business actions, not technical reverse-deltas. Some operations have no clean compensation — you can't "un-send" an email or "un-call" a wire transfer. Those steps need to go later in the saga, or be designed to fail rarely, because once they execute, there's no going back.
Sagas come in two implementation styles. Choreography: each service listens for events from others and decides what to do, with no central authority. Orchestration: a single saga coordinator drives the flow, telling each service when to act. Both achieve the same end-state; they differ wildly in operability, debuggability, and where complexity lives. Most teams converge on orchestration as systems mature — but choreography has real strengths in narrow contexts.
Services publish events to a shared bus (M.26-M.27). Each service subscribes to events it cares about and decides locally when to act. The saga flow lives in the chain of reactions; there's no central place that owns the workflow definition.
A saga coordinator (often a stateful service, sometimes a workflow engine like Temporal or AWS Step Functions) holds the workflow definition. It sends commands to each service in sequence, waits for responses, and decides the next step based on results.
The honest verdict from production experience: orchestration wins for non-trivial sagas. The "workflow lives in the coordinator" property is hugely valuable: you can look at the code and see what the saga does. With choreography, you can't — the saga emerges from the union of subscriptions across N services, and that emergence is often where the bugs hide. "Why did this saga get stuck at step 3?" takes minutes to answer with orchestration (read the coordinator's state); it takes hours with choreography (trace events across logs). The systems people choose to use to build sagas — Temporal, Cadence, AWS Step Functions, Camunda, Azure Durable Functions — are all orchestration-based. That convergence isn't an accident.
Choreography earns its keep in a specific case: small sagas where the participating services have natural event-emitting reasons to exist anyway. If you already have an event-sourced architecture (M.28) with services publishing to a shared log, adding a 2-step saga via two subscriptions is genuinely simpler than introducing a coordinator. The trap is letting that pattern grow. A 2-step choreographed saga becomes a 4-step one, then 7-step; eventually you have a workflow whose state is scattered across seven event subscriptions, and now refactoring to orchestration is a quarterly project.
Below: an order-placement saga across four services — Payment, Inventory, Shipping, Notification. Toggle between orchestration mode (a coordinator drives the flow) and choreography mode (services react to events). Pick a scenario: happy path, payment failure, inventory failure, or shipping failure. Step forward and watch the saga complete — or fail mid-flow and compensate backward. The saga log on the right records every command and event in order.
The concept is elegant. The implementation is not. Most saga bugs aren't in the happy path — they're in compensation logic, retries, idempotency, and the edge cases that emerge when services fail in unexpected combinations. Teams who've shipped sagas in production all share the same set of war stories. The pitfalls below come up so consistently that they're essentially required reading before you write your first one.
When step 3 fails, the saga compensates. Then the compensation message gets retried (network blip), and the refund runs again. Now you've refunded $99 twice. Compensating actions MUST be idempotent — check "have I already refunded this transaction?" before issuing the refund. Most production saga incidents trace back here.
You can refund a payment. You can release inventory. You cannot un-send an email or un-call an external wire transfer. Those actions are irreversible. The fix: order the saga so irreversible actions happen LAST — by the time you've sent the email, everything before has already committed and won't be compensated. If shipping the notification first is unavoidable, design the compensation as a follow-up action ("we sent the wrong notification, please ignore"), not a true reversal.
The forward path failed; you're compensating; the compensation itself fails. Now what? Two choices: (1) retry the compensation indefinitely with exponential backoff (the right answer most of the time), or (2) escalate to a human dead-letter queue for manual resolution. Production sagas need both — automated retries for transient failures, human review for persistent ones. A saga can be "stuck compensating" for days; the system must surface this clearly.
Sagas give up isolation. While yours is in progress, another process might read the half-completed state — paid but not yet inventory-reserved — and start its own flow against it. The mitigation: semantic locks. When the order is created, mark it state="pending"; only allow other operations once state advances. Real distributed locks (M.21) aren't usually necessary; "marker columns" enforced at the application layer typically suffice.
In orchestration, the coordinator's state IS the answer. In choreography, there's no single place that knows the saga's progress — it's spread across N service logs. The first feature most teams build after their first production saga is a "saga inspector" — a UI showing which step each in-flight saga is on. Workflow engines (Temporal, Step Functions) give you this for free; rolling your own means building it.
A saga might span days — wait for payment confirmation, wait for warehouse pick, wait for shipping carrier. The coordinator can't hold this in memory; deploys and restarts are routine. State must persist somewhere durable — an event store, a workflow engine's database, a saga log table. Without this, every deploy risks orphaning in-flight sagas. This is the main reason teams reach for Temporal or AWS Step Functions instead of writing their own coordinator.
The composite advice that emerges across all six: sagas are not a thing you "just implement". They're a coordination protocol with real correctness obligations, real failure modes, and real operational requirements. The reason workflow engines (Temporal, Cadence, Step Functions, Camunda, Azure Durable Functions) all exist is that everyone who tries to build sagas from scratch hits the same six pitfalls, and after a few rounds of war-room incidents, decides that buying or borrowing the saga infrastructure is cheaper than building it. "Should we use a workflow engine?" is the right question — and the answer is usually yes once your saga has more than three steps.
The terms every Temporal doc, every "we use sagas" architecture review, every Garcia-Molina paper citation assumes you already know.
state="pending") that prevents other operations from acting on a saga's intermediate state. Substitute for real isolation; cheaper than distributed locks.Test the saga intuition. Click an answer; explanation drops in instantly.
Perfect. ACID's limits, sagas, compensations, orchestration vs choreography — all yours. Next: the one delivery guarantee we've been dancing around since M.26.
The coordination pattern that lets services collaborate on long-running business flows without the false promise of cross-service ACID.
Cross-service transactions need a different model. 2PC is correct but doesn't scale; sagas give up atomicity and isolation for the availability that actually ships.
They're new business transactions: refund, release, cancel. They must be idempotent. They can fail. Some actions have no clean compensation — those go last, or never.
Orchestration via Temporal, Step Functions, or Camunda almost always beats rolling your own coordinator. Three steps is the rough threshold; beyond it, build-vs-buy reliably points to buy.