The specific mechanisms for atomic commit when data lives on multiple nodes. Two-Phase Commit (Gray 1978), Percolator (Google 2010), Spanner\'s 2PC-over-Paxos (2012), Calvin\'s deterministic ordering (2012), and sagas (Garcia-Molina 1987) — five approaches to the same problem with fundamentally different tradeoffs. Understanding which protocol fits which system is Expert-tier competence for anyone building distributed data systems, cross-service workflows, or microservice architectures.
Single-node atomicity is a solved problem. Every database engine handles it: write-ahead log, commit record, fsync, done. Either the commit record is on disk (transaction committed) or it isn\'t (transaction aborted). The specific mechanism has been well-understood since the 1970s — journaling filesystems use it, every SQL database uses it, event-sourcing systems use it. Local ACID is easy. What breaks the moment your transaction touches data on multiple nodes: local ACID doesn\'t compose. If node A commits and node B fails to commit (crash, network partition, disk full), you have an inconsistent state — data on A that references data on B which doesn\'t exist. The transaction is partially applied. There\'s no way to undo A\'s commit after the fact (its clients have already seen the committed state), and no way to force B to commit (B has already durably declined). This partial-commit problem is what distributed transaction protocols exist to solve — and every protocol makes specific tradeoffs about which failure modes it handles gracefully and which it blocks on.
The specific engineering challenge is that all the failure modes of a distributed system happen during commit. A single-node crash before commit is fine (recovery aborts the incomplete transaction). A single-node crash after commit is fine (the commit record is durable). But in a distributed transaction, "after commit" is ambiguous: after node A commits but before node B commits, we\'re in a partial state that no protocol can undo. Network partitions during commit are worse — a participant may have voted "yes" (agreed to commit if told) and then lost contact with the coordinator; it can\'t safely commit (maybe the coordinator decided abort) and can\'t safely abort (maybe the coordinator decided commit). This is the specific "blocking problem" of 2PC — participants can be stuck in the "prepared" state indefinitely if the coordinator dies at the wrong moment. Understanding this failure mode precisely, and knowing which protocols address it and how, is the specific Expert-tier competence this module builds.
Each earlier attempt fails in a specific way. Independent commits give no atomicity. Assume reliability ignores partitions. Classic 2PC is correct but blocks on coordinator failure. Modern approaches all solve the blocking problem through different mechanisms: Spanner replicates the coordinator via Paxos (still 2PC, but the coordinator can\'t die); Calvin uses consensus to agree on transaction order in advance (no 2PC needed since order is already agreed); Sagas give up atomicity and use compensating actions. Which fits your system depends on whether you need strong consistency (Spanner), high throughput with deterministic workloads (Calvin), or long-running cross-service workflows (Sagas). Modern distributed data systems combine these — Spanner and CockroachDB use 2PC-over-Paxos for OLTP transactions; microservice architectures use Sagas for business processes; specialized OLTP systems like FaunaDB use deterministic ordering. §02 walks through 2PC mechanically; §03 walks through Percolator, Spanner-style, Calvin, and Sagas.
The historical arc of distributed transactions is a specific case where the theoretical framework was established early and practical variants have emerged incrementally over four decades. 1978: Jim Gray publishes "Notes on Database Operating Systems." Introduces 2PC as the standard protocol for atomic commit across sites. Framed as an internal IBM document but widely adopted; every subsequent distributed transaction protocol builds on Gray\'s framework. 1981: Skeen publishes Three-Phase Commit (3PC). Adds a "pre-commit" phase to make the protocol non-blocking under certain synchrony assumptions. Rarely used in practice — the synchrony assumptions don\'t hold in real networks, and the added phase increases latency. Interesting theoretically; rarely deployed. 1987: Garcia-Molina and Salem publish "Sagas." Recognizes that long-running transactions can\'t hold locks for hours; proposes decomposing them into a sequence of local transactions each with a compensating action. If any step fails, execute the compensations in reverse to logically undo the work. Foundational for modern microservice architectures. 1990s-2000s: XA (X/Open) standardizes 2PC in enterprise systems. Databases, message queues, and app servers all implement XA. Widely deployed for cross-database transactions in enterprise Java applications. Also widely criticized for performance issues and specific blocking failure modes. 2007: Dynamo paper. Amazon\'s specific choice: give up distributed transactions entirely. Use eventually-consistent K-V storage with application-level conflict resolution. This becomes the model for "NoSQL" era — throughput and availability at the cost of atomicity. 2010: Google Percolator paper. Snapshot isolation transactions over Bigtable using a global Timestamp Oracle (TSO) and per-row locks. Enables cross-row transactions on massively distributed storage at reasonable throughput. Basis for TiDB\'s transaction layer. 2012: Two major papers arrive simultaneously. Google Spanner: 2PC over Paxos groups with TrueTime for external consistency. Yale Calvin: deterministic ordering via consensus, then parallel execution — no 2PC needed. Both solve the coordinator-blocking problem, but through fundamentally different mechanisms. Spanner is the industrial reference for strong-consistency multi-region OLTP; Calvin is the academic reference for deterministic transaction execution. 2014-present: Modern distributed SQL databases (CockroachDB, YugabyteDB, TiDB, FoundationDB) all use variants of the Spanner or Percolator patterns. Distributed transactions become a commodity — you can deploy CockroachDB and get global ACID transactions at Cassandra-like scale. 2015+: Microservice architectures adopt Sagas at scale. Netflix\'s Conductor, Uber\'s Cadence (now Temporal), various event-driven architectures — Sagas become the standard pattern for long-running cross-service workflows. 2020+: The choice of transaction protocol is a first-class architectural decision. Understanding when each fits — 2PC-over-Paxos for OLTP, Sagas for microservices, Percolator for massive K-V transactions — is standard senior engineer competence. The arc explains why "distributed transactions are hard" turned into "distributed transactions are a menu of specific protocols with specific tradeoffs".
Two-Phase Commit (2PC), introduced by Jim Gray in 1978, is the foundational distributed transaction protocol. Every subsequent atomic-commit protocol either implements 2PC directly (with variations), uses 2PC as a subroutine (Spanner\'s cross-Paxos-group commits), or explicitly rejects 2PC and provides an alternative (Calvin, Sagas). Understanding 2PC precisely — what messages get sent, what state each participant maintains, and specifically where it blocks — is the specific competence that turns "distributed transactions are hard" into "distributed transactions have a specific failure mode that specific protocols solve in specific ways."
Phase 1 (Prepare): coordinator asks all participants "can you commit?" Each votes YES (durably persisting its intent) or NO. Phase 2 (Commit): coordinator broadcasts the decision (COMMIT if all YES, ABORT otherwise). Two phases are the minimum needed for atomic commit — one phase gives no chance to detect participant refusal before commit.
Every state transition is durably logged. Participants persist their vote before responding. Coordinator persists the decision before broadcasting. This is what makes 2PC crash-tolerant for participant failures: a crashed participant recovers, checks its WAL for the prepare record, and asks the coordinator for the decision. Recovery is deterministic.
After voting YES, a participant is in the "prepared" state — it has committed to either commit or abort but doesn\'t know which yet. It must hold all locks and cannot process other transactions on the affected data. This is what makes 2PC expensive: prepared participants block. If the coordinator takes 5 seconds to decide, participants hold locks for 5 seconds.
The classic 2PC failure mode. If the coordinator crashes between Phase 1 and Phase 2, participants are stuck. They can\'t safely make a unilateral decision — abort might contradict the coordinator\'s decision to commit; commit might contradict the coordinator\'s decision to abort. They wait for coordinator recovery, holding locks. This is why classic 2PC is called "blocking."
Common optimization: if participants don\'t receive a decision within a timeout, they can query the coordinator or presume abort (only safe if they haven\'t received COMMIT). Reduces blocking window at the cost of correctness in edge cases. Real implementations combine timeouts, backups, and administrator intervention to unstick participants.
X/Open XA (1991) standardizes 2PC across databases and message queues. Widely deployed in enterprise Java (JTA/JTS). Enables cross-database transactions in application servers. Also widely criticized for performance and specific failure modes — many teams avoid XA in favor of application-level compensating patterns (Sagas).
The prepared state (iii) is the specific source of 2PC\'s cost profile. When a participant votes YES in Phase 1, it\'s making a durable commitment: "if you tell me to commit, I can commit; if you tell me to abort, I can abort." To maintain this promise, the participant must hold all locks acquired during the transaction. It can\'t release the locks (a concurrent transaction might see uncommitted state) and can\'t downgrade them (the participant may still need to modify the data at commit time). This means every millisecond between Phase 1 vote and Phase 2 decision is a millisecond of held locks. On a busy database, held locks translate directly to reduced throughput — other transactions wait, timeout, or deadlock. This is why 2PC is expensive: not the message count (2N messages per participant) but the lock-holding duration. The specific engineering implication: 2PC works well when the coordinator can decide quickly (single-datacenter latency, ~10ms round-trip); it fails badly across WAN (cross-region latency, 100-300ms) because locks are held for the full round-trip time. Anti-pattern §05.i: 2PC across WAN is a known specific antipattern.
The coordinator single point of failure (iv) is the specific and severe blocking failure that motivates every modern alternative. Consider the sequence: coordinator sends PREPARE, all N participants durably vote YES. Coordinator receives all votes and durably decides COMMIT. Coordinator crashes before broadcasting COMMIT. Now: each participant knows it voted YES but doesn\'t know the coordinator\'s decision. If it aborts, it might contradict a "commit" decision the coordinator persisted before crashing (other participants might have received COMMIT before the crash and already applied — abort would leave inconsistent state). If it commits, it might contradict an "abort" decision (if the coordinator hadn\'t actually decided yet and would have aborted upon recovery). The safe action is to wait — hold locks, do nothing, poll the coordinator. In real deployments, coordinator recovery can take minutes to hours depending on the failure mode (disk failure requires restore from backup; network partition requires manual intervention). Participants hold locks the whole time, other transactions pile up, cascading timeouts and deadlocks. This specific failure mode has taken down entire enterprise database clusters — and it\'s not a rare occurrence, it happens every time a coordinator crashes at the wrong instant. Every serious 2PC deployment has monitoring, automated failover, and playbooks for exactly this scenario.
The presumed abort optimization (v) is the specific mitigation most 2PC implementations use to reduce the blocking window without breaking correctness. The idea: if the coordinator crashes before deciding, no participant should have applied commit — they were all still in the prepared state. If a participant times out waiting for the decision, it can safely presume abort if it can prove the coordinator hadn\'t yet decided. Real implementations do this via specific checks: (a) query the coordinator directly if reachable; (b) query other participants to see if any received COMMIT; (c) fall back to blocking if uncertainty remains. Presumed commit is the dual: if you can prove the coordinator decided commit (e.g., by querying other participants who received COMMIT), you can commit safely. Both optimizations reduce but don\'t eliminate the blocking problem. The fundamental issue is that 2PC\'s safety proof requires the coordinator to be non-faulty at a specific moment. Fixing this requires replicating the coordinator (Spanner-style 2PC over Paxos, §03) or eliminating the coordinator entirely (Calvin\'s deterministic ordering, §03).
Modern distributed transaction protocols solve 2PC\'s coordinator-blocking problem in three fundamentally different ways. Each represents a specific engineering choice with a specific tradeoff profile. Percolator (Google 2010) uses snapshot isolation with a global Timestamp Oracle (TSO) and per-row locks — no traditional coordinator, transactions coordinate via a centralized lock table. Spanner-style 2PC-over-Paxos (Google 2012) replicates the coordinator as a Paxos group — the coordinator can\'t "die" without a majority of Paxos replicas failing simultaneously. Sagas (Garcia-Molina 1987, revived for microservices ~2015) give up atomicity entirely — long-running transactions are decomposed into local transactions with programmer-supplied compensating actions. Understanding which fits which system is the specific competence for architecting distributed data or microservices.
Each data shard is a Paxos group (§ M.47). Cross-shard transactions run 2PC where the coordinator is itself a Paxos group. Since Paxos survives minority failures, the coordinator can\'t "crash" in the blocking sense — a majority must fail simultaneously. Preserves 2PC\'s strong consistency without the blocking problem.
Snapshot isolation via a global Timestamp Oracle. Each transaction gets start_ts and commit_ts from TSO. Per-row locks with a designated "primary" row whose commit state determines the entire transaction\'s outcome. No traditional coordinator — the primary row IS the commit point. Async cleanup rolls forward the other rows.
Sequence of local transactions T1, T2, ..., Tn, each with a corresponding compensation C1, C2, ..., Cn. Forward flow: execute T1, T2, ..., Tn. On failure at step k: execute C(k-1), C(k-2), ..., C1 to logically undo. No atomicity — other observers can see intermediate states. Programmer must write correct compensations for every step.
Yale 2012 alternative. All replicas agree on a total order of transactions via consensus BEFORE execution. Then each replica executes deterministically — same input, same output. No 2PC needed because order is already agreed. High throughput; requires deterministic execution (no external side effects during transaction). Basis for FaunaDB.
Sagas provide no isolation — during saga execution, other transactions can see partial state. If Saga A is halfway through (T1 committed, T2 committed, T3 pending), Saga B can see T1 and T2 committed even though the overall saga hasn\'t. Semantic locking, semantic isolation, or careful application design must handle this. Anti-pattern §05.v: ignoring saga isolation.
OLTP with strong consistency + moderate latency: Spanner-style (Spanner, CockroachDB). Massive K-V transactions with snapshot isolation: Percolator (TiDB). Cross-service microservice workflows: Sagas (Temporal). Deterministic OLTP with high throughput: Calvin (FaunaDB). Legacy XA where you need cross-database transactions: classical 2PC with monitoring.
The Spanner-style 2PC-over-Paxos (i) is the specific engineering pattern that made distributed OLTP practical at global scale. The insight: 2PC\'s only failure mode is coordinator failure, and consensus (Paxos/Raft) exists specifically to make replicated state machines tolerant to failures. So replicate the coordinator. Each data shard is a Paxos group (say, 5 replicas across zones). A transaction that touches shards A, B, C picks one shard\'s Paxos group (typically the shard with the most participants) as the "coordinator group." 2PC runs between the coordinator group and each participating shard\'s Paxos group. The coordinator group can\'t "die" without a majority of its Paxos replicas failing simultaneously — extremely rare in properly-designed deployments. Prepare records are written to Paxos logs, so they\'re durably persisted across replicas. Commit decisions are made by the coordinator group (via Paxos) and broadcast to participants. The composite result: strong consistency (2PC gives atomicity), high availability (Paxos gives failure tolerance), and no blocking under normal operations. Spanner adds TrueTime for external consistency across regions. CockroachDB uses this pattern with HLCs. YugabyteDB uses it with Raft groups. This is what modern distributed OLTP looks like — 2PC that survived the coordinator problem by consensus-replicating everything.
The Percolator TSO-based approach (ii) is Google\'s specific innovation for cross-row transactions on Bigtable, and it\'s the pattern TiDB inherited. The key insight: instead of a traditional 2PC coordinator, use a global Timestamp Oracle (TSO) that issues monotonically increasing timestamps, and use one "primary" row per transaction whose commit state serves as the transaction\'s ground truth. Protocol: (a) client gets start_ts from TSO; (b) reads see the snapshot at start_ts; (c) writes acquire per-row locks with a "lock column" that points to the primary row; (d) once all writes are locked, client gets commit_ts from TSO; (e) client updates the primary row atomically (write commit_ts to the primary\'s special column); (f) at this moment, the transaction is committed — the primary row\'s state is the commit point; (g) async cleanup rolls the other rows forward (removes locks, writes commit_ts). If the client crashes before step (e), other transactions can detect the abandoned lock and clean up (they see the primary row not committed → abort the abandoned transaction). If the client crashes after step (e), other transactions can see the primary row committed and finish the async cleanup themselves. No traditional coordinator is needed because the primary row IS the coordinator — its atomic state determines everything. Cost: 2 round-trips to the TSO per transaction. Works well for massive K-V stores where consistency matters but where you don\'t want to run Paxos on every row.
The Saga pattern (iii) is the specific approach for long-running distributed transactions, particularly across microservices where holding locks for the transaction duration is infeasible. Original 1987 formulation was for long-running database transactions (e.g., booking a multi-leg vacation with hotel + flight + rental car reservations, where each step takes seconds and holding locks across the whole process would kill throughput). Modern revival is for microservice orchestration: a business process spans multiple services (order-service, payment-service, inventory-service, shipping-service), each with its own database, and you need "either the order succeeds fully or nothing sticks." Saga structure: sequence of steps T1, T2, ..., Tn, each a local transaction in one service. Each step Ti has a compensation Ci that logically undoes it. Forward execution: T1 (charge card) → T2 (reserve inventory) → T3 (schedule shipment). If T3 fails: execute C2 (release inventory) then C1 (refund card). The specific tradeoff Sagas make: no atomicity, no isolation, but works for long-running distributed workflows. Other observers can see partial state during saga execution (order is charged but not yet shipped). Compensation must be idempotent (Ci might run multiple times due to retries). Compensation must be semantically correct (refund isn\'t always the exact inverse of charge — original charge might have generated audit trails, notifications, etc.). Modern implementations: Temporal (Uber\'s Cadence rewritten and open-sourced), Camunda (BPMN-based workflow engine), Netflix Conductor, and various homegrown orchestrators. Understanding Sagas is prerequisite for anyone architecting microservices — they\'re the specific pattern for cross-service consistency.
Below: each of three canonical distributed transaction approaches (2PC · Spanner-style · Sagas) evaluated against three failure scenarios (Coordinator crash after prepare · Network partition mid-transaction · Participant timeout). Watch how each protocol handles each failure — the specific failure modes show exactly why modern protocols exist and where each fits. The 9 cells illustrate the specific tradeoffs that drive protocol selection.
The failure modes of distributed transactions are the specific mechanisms by which "our system says the transaction committed" turns into "some of it did, some didn\'t, and we have no idea what state the data is in." Each of these anti-patterns is a real pattern from real production incidents, and each has a specific mitigation that Expert engineers implement by default. Recognizing them at architecture time saves the postmortem.
Classic 2PC across wide-area networks pays the full network round-trip in held locks. Prepare phase requires one round-trip; commit phase requires another. Between prepare-ack and commit-ack, every participant holds locks. Cross-continent latency of 100-300ms means locks are held 200-600ms per transaction. On a busy system, held locks cascade into contention, timeouts, and cascading failures. The fix: (a) prefer single-region strong-consistency with async cross-region replication for reads; (b) use Spanner-style 2PC-over-Paxos which can pipeline prepare and commit phases; (c) redesign to avoid cross-region transactions — partition data so each transaction stays in one region; (d) accept eventual consistency across regions with sagas for cross-region workflows. The general principle: classical 2PC across WAN is a specific antipattern with known throughput consequences; modern deployments use consensus-replicated coordinators or eliminate cross-region transactions entirely.
Classic 2PC has a specific blocking failure mode when the coordinator crashes between prepare and commit phases. Participants who voted YES are stuck in "prepared" — they can\'t abort (maybe the coordinator decided commit) and can\'t commit (maybe the coordinator decided abort). They hold all locks. In real deployments this can take out entire database clusters. The fix: (a) use Spanner-style 2PC-over-Paxos where the coordinator is itself a Paxos group — no single-VM failure; (b) implement operator playbooks for manual coordinator recovery — force-abort prepared participants after long timeouts (with data-integrity risk); (c) implement 3PC (rare in practice due to complexity); (d) use Percolator-style pattern where the primary row eliminates the coordinator; (e) use Sagas for workflows where classical 2PC coordinator failure is unacceptable. The general principle: classical 2PC without coordinator replication is a specific antipattern; every serious production deployment either replicates the coordinator or has documented manual recovery procedures.
Sagas require a compensating action for every step, but real-world side effects (emails, notifications, external API calls) often can\'t be truly undone. Semantic compensation is required — the compensation must produce an outcome that is equivalent to as-if-not-executed, not literally undo the action. For emails: send a follow-up "your order was cancelled" email. For external API calls: use the API\'s cancellation endpoint if available, otherwise mark the transaction as "conditionally executed pending confirmation." The fix: (a) design compensations semantically, not literally — think about business meaning; (b) sequence steps so irreversible actions come last (send email as final step, so if it succeeds the whole saga succeeded); (c) use semantic locking to prevent other transactions from seeing intermediate states that could become invalid; (d) accept some level of "compensation debt" that must be resolved manually or via customer support. The general principle: sagas require thoughtful compensation design; missing or incorrect compensations produce specific classes of production bugs.
Applying 2PC (or any strong-consistency distributed transaction protocol) to workloads that don\'t require it pays the coordination cost for no correctness benefit. Most microservice interactions don\'t need cross-service atomicity — they can accept eventual consistency with retries and idempotency. Wrapping them in distributed transactions makes every service depend on every other service being available, which produces cascading failures. The fix: (a) identify which workflows actually need cross-service atomicity (money movement, inventory-affecting operations, ordering-sensitive ops) — these use Sagas or 2PC; (b) use eventual consistency with retries + idempotency for everything else; (c) use event-driven architectures for loose coupling; (d) accept that "some things are eventually consistent" is normal, not a bug. The general principle: distributed transactions are expensive; use them only where correctness requires them; default to eventual consistency for microservice interactions.
Sagas provide no isolation between concurrent transactions. Unlike ACID transactions, saga steps commit locally and are immediately visible to other transactions. If two sagas concurrently modify the same data, they can see each other\'s intermediate states and make inconsistent decisions. Classic example: overselling in inventory systems. The fix: (a) use semantic locks — reserve inventory as a pending status during saga execution, only convert to committed after saga completes; (b) use commutative operations — design saga steps so their effects commute (order doesn\'t matter); (c) use version numbers and detect conflicts at commit time; (d) accept some level of business-logic-level reconciliation (oversold orders trigger customer service workflow). The general principle: sagas are ACD (atomic-consistent-durable at each step) but lack isolation; concurrent sagas can produce anomalies that ACID transactions prevent.
The composite pattern across all five is that distributed transactions are a specific engineering tool with specific failure modes and specific costs. Not every operation needs distributed atomicity — most operations can accept eventual consistency with retries. The operations that DO need distributed atomicity have specific protocol options (2PC for legacy, Spanner-style for modern OLTP, Sagas for cross-service workflows), and each protocol has specific failure modes that must be planned for. Choosing the right protocol for the right operation is the specific Expert-tier judgment. Getting it wrong produces: (a) unnecessary latency and throughput loss (2PC where eventual would suffice), (b) unplanned outages (classical 2PC without coordinator replication), (c) subtle correctness bugs (sagas without proper compensation or isolation). Recognizing these patterns at architecture time is what makes distributed data systems reliable in production.
The terms that show up in every distributed database design review, every microservices architecture discussion, every "how do we handle this cross-service transaction?" conversation.
Test the tradeoffs. Click an answer; explanation drops in instantly.
Perfect. 2PC prepare-and-commit, the coordinator blocking problem, Spanner\'s 2PC-over-Paxos, Percolator\'s TSO + primary row, sagas\' compensating actions — the tools for atomic commit across nodes with specific tradeoffs. Next up: M.52, geo-distributed transactions.
The composite understanding that turns "distributed transactions are hard" into "distributed transactions are a menu of specific protocols with specific tradeoffs."
Single-node atomicity (WAL + commit record) is solved. Multi-node atomicity is fundamentally harder because every participant and the coordinator can fail at any point. Every distributed transaction protocol exists to solve this specific problem: get all participants to agree on commit-or-abort before any of them makes the commit visible.
Jim Gray\'s 1978 protocol is correct in the absence of failures but has a specific severe blocking failure mode: if the coordinator crashes between prepare and commit phases, participants are stuck in the prepared state holding locks. Every modern variant solves this specific problem — Spanner replicates the coordinator via Paxos, Percolator eliminates it via primary row, Sagas give up atomicity for compensation.
OLTP with strong consistency: Spanner-style (Spanner, CockroachDB, YugabyteDB). Massive K-V with SI: Percolator (TiDB). Cross-service microservice workflows: Sagas (Temporal, Camunda). Deterministic OLTP at scale: Calvin (FaunaDB). Legacy cross-database: classical 2PC (XA). Each protocol has specific failure modes to plan for; using the wrong one produces specific classes of production incidents.