Module 23 / 46 · Phase E — Distributed Reality · 45 min

Time, order,
and Lamport
clocks.

Distributed systems have no shared clock. Wall time disagrees by milliseconds at best, hours at worst. So how do nodes agree on what happened first? Lamport's elegant 1978 insight: stop trying to measure time. Measure causality instead.

// What you'll know by the end

  • Why physical clocks can't order distributed events
  • The happens-before relation, precisely
  • Lamport timestamps — and where they fail
  • Vector clocks, and why Riak/Dynamo use them
§ 01 — Why time is the wrong question

Two writes.
Whose came first?

Two database replicas in different datacenters each accept a write at "the same moment." Both record their write with a timestamp from their local clock. Now you replay the writes elsewhere and have to decide which one to apply first. You look at the timestamps. Replica A says 14:32:01.847. Replica B says 14:32:01.831. So B came first, right? Maybe. Or maybe A's clock is fast by 20ms. Or B's NTP daemon hasn't synchronized in an hour. Or there's a clock-skew shock from the last leap second. The numbers are real. The ordering they imply may not be.

// WHAT ACTUALLY HAPPENS · WHEN MACHINES TRY TO AGREE ON TIME
SOURCE
CLOCK BEHAVIOR
WHY IT'S NOT TRUSTWORTHY
CPU quartz
drifts ~10ppm from true time
Two identical servers drift ~1 second per day apart from each other. Across a fleet of thousands, you're guaranteed disagreement.
NTP synchronization
accuracy: ~1-50ms over WAN
Best case ~1ms in a datacenter; much worse over the public internet. NTP can also step backward, breaking the assumption that time moves forward.
PTP (datacenter-grade)
accuracy: ~100µs in well-tuned LAN
Better, but still orders of magnitude worse than CPU cycle times (~ns). Events happening in the same millisecond are unorderable even with PTP.
GPS / atomic clocks
accuracy: ~7ns (Google TrueTime)
Extraordinarily expensive infrastructure. Used by Spanner. Out of reach for almost every other system.

The escape hatch is to stop asking what time it is. The thing we really care about isn't "what's the timestamp," it's "if event B's outcome depends on event A, did A actually happen before B?" That question has an answer even when clocks are unreliable — because causality leaves its own trail. If A sent a message and B received it, A must have happened before B. No clock needed. Lamport's 1978 paper turned this intuition into formal mathematics, and the entire field of distributed systems has been building on it ever since.

Forget wall clocks. Track causality. The order that matters is the one you can prove.

The rest of this module unpacks how. We'll define the happens-before relation (the foundational primitive), build Lamport timestamps (a clever scheme that captures part of it), see where Lamport's elegance falls short, and upgrade to vector clocks — which capture causality completely. The lab in §04 lets you watch all of this play out across three processes exchanging messages.

§ 02 — The happens-before relation

Three rules.
That's all.

Lamport's first move was definitional: he gave us a precise mathematical relation, written A → B ("A happens-before B"), that captures the only ordering that's both verifiable and meaningful in a distributed system. Three rules generate the whole relation. From these rules, everything else follows.

// THE HAPPENS-BEFORE RELATION · A → B

i
Program order
if A and B are in the same process and A precedes B → A → B

Within a single process, events happen in the order the program executes them. This is the only kind of ordering you can take for granted — a single CPU, a single thread, a single sequence of instructions. The bedrock everything else builds on.

ii
Message order
if A is "send msg m" and B is "receive msg m" → A → B

A message can't be received before it was sent. Send always happens-before receive. This is the bridge between processes — the only causal link you can establish without a shared clock. If A sent something B holds, A was first.

iii
Transitivity
if A → B and B → C, then A → C

Causality chains. If your message reached me, and I sent it on, the original sender happens-before the eventual recipient. Transitivity is what makes "happens-before" a partial order — it lets you reason about causes far down the chain from effects.

Two events are concurrent (written A || B) if neither happens-before the other. This is the new, distinctly distributed-systems concept: some events have no inherent order. They may have been "really" simultaneous, or one may have "actually" come first by physical wall time — but if no causal chain connects them, the system has no way to know, and (more importantly) their order doesn't matter to any computation. Concurrent events can be applied in any order; the result is the same.

Concurrent doesn't mean "at the same moment." It means "no causal chain between them."

This is the single most important conceptual upgrade in distributed systems. In a single-machine world, all events have a total order (the program executes them one at a time). In a distributed world, events have only a partial order. Some events are clearly ordered (causal chains link them); others are genuinely incomparable. Trying to force a total order on partially-ordered events is the root cause of half the bugs in distributed databases. Causal consistency from M.22 is precisely the model that respects this partial order without trying to fake a total one.

§ 03 — Lamport & vector clocks

Two ways to
track causality.

Knowing happens-before is one thing; computing it efficiently is another. You can't ship the entire event history with every message. So Lamport (and later, Mattern & Fidge) invented compact representations — logical clocks — that summarize causality in a few numbers. Two designs matter: Lamport timestamps (one integer per event) and vector clocks (one integer per process per event). They make different trade-offs.

// THE TWO LOGICAL CLOCK ALGORITHMS · SIDE BY SIDE

// SIMPLER · WEAKER
Lamport Timestamp
// Each process keeps one counter L on local event: L = L + 1 on send msg m: L = L + 1 attach L to m on receive msg m: L = max(L, m.L) + 1

One integer per event. Cheap to store, cheap to ship in every message. Captures the forward direction of happens-before perfectly: if A → B, then L(A) < L(B). Guaranteed.

The catch: the reverse isn't true. L(A) < L(B) does NOT imply A → B. They might be concurrent. Lamport timestamps can't distinguish "causally before" from "happens to have a smaller number."
// RICHER · COSTLIER
Vector Clock
// Each process keeps vector V[N] of N counters on local event: V[me] = V[me] + 1 on send msg m: V[me] = V[me] + 1 attach V to m on receive msg m: V = elementwise_max(V, m.V) V[me] = V[me] + 1

N integers per event, where N = number of processes. Tracks how much each process has "seen" from every other. Larger, but encodes the full happens-before relation.

The win: A → B if and only if V(A) < V(B) component-wise (every entry ≤, at least one strictly <). Otherwise: concurrent. Vector clocks detect concurrency; Lamport hides it.

The trade-off is exactly what it looks like: Lamport gives you cheap, one-integer timestamps that get the easy cases right but can't tell you when events are concurrent. Vector clocks pay N integers per event for the ability to detect concurrency precisely — which turns out to be exactly what distributed databases need for conflict resolution (M.22 territory). If two replicas accepted writes whose vector clocks are incomparable, the writes are concurrent, and the system needs to merge them. Lamport timestamps would give you two different numbers and lie about whether one preceded the other.

Both algorithms share a common shape: local events bump your own counter; messages carry causality forward; receivers merge by taking max. That last move — max — is the heart of it. When you receive a message, you absorb everything the sender knew about the past. The max ensures you never go backward, and the +1 ensures your next event is strictly after every event in your causal past.

Time to make this physical. The lab below runs three processes exchanging messages across three scenarios. Toggle between Lamport and vector mode; step through events one at a time; watch the counters update via exactly these rules. By the end, you'll be able to compute either by hand.

§ 04 — The clock simulator · interactive lab

Watch causality
get a number.

Below: three processes (P₁, P₂, P₃) exchanging messages. Pick a scenario. Toggle between Lamport and vector clocks. Step through one event at a time and watch the timestamps update according to the rules in §03. The right panel shows the exact computation for each step — and the key insight that step is meant to teach.

CLOCK.SIM // m.23 lab
Step 0 / 0
// EVENT TIMELINE · three processes
SCENARIO READY
Press Next to start
"Step through the events to see clocks update one at a time."
Each scenario walks through a sequence of events on three processes. Pick a clock mode (Lamport or Vector) above. As you advance, watch the timestamps update at each step using the algorithm rules.
§ 05 — Time in real systems

Where this
actually lives.

Lamport's 1978 paper introduced logical clocks. Almost fifty years later, every distributed database in production uses some descendant. The specific designs vary — pure Lamport is rare; vector clocks have known scaling issues; modern systems mostly use hybrid approaches — but the foundational ideas are the same: track causality with a logical structure, optionally augmented with physical time bounds. Below: four concrete examples.

// THE LOGICAL CLOCK · IN PRODUCTION

Vector Clocks// Riak · early Dynamo · CouchDB
Pure vector clocks in AP systems. Every write carries a vector clock. On reads, the system returns all "incomparable" (concurrent) versions to the client — letting the application decide how to merge. Famous from the original Dynamo paper. Cost: vectors grow with cluster size; pruning is tricky.
Hybrid Logical Clocks// CockroachDB · MongoDB · YugabyteDB · Kulkarni et al, 2014
HLC: (physical_time, logical_counter). Combines wall-clock with logical advancement. Physical part stays close to NTP; logical part handles ties and out-of-order receives. Best of both: readable timestamps that respect causality. The modern default for new distributed databases.
TrueTime// Google Spanner · requires GPS + atomic clocks
Physical clocks with bounded uncertainty. Each call returns an interval [earliest, latest] rather than a single timestamp. Spanner waits out the uncertainty before committing, ensuring external consistency — wall-clock-ordered transactions across continents. Cost: custom hardware in every datacenter. Out of reach for non-Google systems.
Lamport Timestamps// Kafka log offsets · Raft log indices · most consensus protocols
Single-leader systems use a single counter for total ordering — effectively a Lamport timestamp where there's only one process that increments. Simple, correct, fast. The reason Kafka topics have monotonically-increasing offsets, and the reason Raft has log indices. The CP world's favorite primitive.

Two things to internalize. First, the choice of clock mirrors the consistency choice from M.21 and M.22. AP systems with eventual consistency tend to use vector clocks (to detect conflicts for merging). CP systems with linearizable consistency lean on single-counter Lamport-like indices (because there's a single leader, so total order is achievable). HLC sits in between, useful for systems that want causal consistency with readable timestamps. The clock you reach for is determined by the consistency model you committed to.

Second, most application engineers will never touch clocks directly — they're implementation details of the databases and message brokers below them. But understanding them is what lets you read the docs, debug the weird cases, and choose intelligently when buying a database. The vendor's marketing copy will not say "we use HLC." The architecture diagram in their docs will. Now you can read it.

§ 06 — Eight words for the clock conversation

Vocabulary,
for the partial order.

The terms that appear in every distributed systems paper from 1978 onward. These are foundational.

Happens-Before
/ˈhæp.ənz/
Lamport's partial order on events: A → B if same process & A first, or A is a send and B is the matching receive, or transitivity. The only ordering verifiable in a distributed system without a shared clock.
Concurrent
/kənˈkʌr.ənt/
Two events A and B are concurrent (written A || B) if neither happens-before the other. Doesn't mean "at the same wall-clock instant" — means no causal chain links them.
Logical Clock
/ˈlɒdʒ.ɪ.kəl/
A counter that advances on events instead of measuring wall time. Tracks causality rather than physical time. Lamport timestamps and vector clocks are two well-known variants.
Lamport Timestamp
/ˈlæm.pɔrt/
Single integer per process. max(local, received) + 1 on receive. Guarantees: if A → B then L(A) < L(B). Does not detect concurrency.
Vector Clock
/ˈvek.tə/
Vector of N counters per process, where N is the number of processes. Detects concurrency precisely — vectors are comparable iff events are causally related. Used by Riak, Dynamo, CouchDB for conflict detection.
Hybrid Logical Clock
/ˈhaɪ.brɪd/
"HLC" — combines physical time (NTP) with a logical counter. Form: (wall_time, logical). Used by CockroachDB, MongoDB, YugabyteDB. The modern default for new distributed databases.
External Consistency
/ɪkˈstɜːr.nəl/
Stronger than linearizability: operations are ordered according to real wall-clock time, system-wide. Achievable only with bounded clock uncertainty (Spanner's TrueTime). Required for cross-region ACID transactions.
Clock Skew
/skjuː/
The difference between two physical clocks reading "the same moment." Can be milliseconds, seconds, even hours in poorly maintained systems. The reason wall-clock timestamps can't safely order distributed events.
§ 07 — Knowledge check

Five questions.
Order the events.

Test the clocks intuition. Click an answer; explanation drops in instantly.

QUESTION 1 OF 5
Loading question...
Score: 0 / 5
5 / 5

Ordered.

Perfect. Happens-before, Lamport, vector — yours. Next: how do nodes actually agree on a single ordering, even with all this clock uncertainty? Consensus & Raft.

§ 08 — The recap

Three ideas to
carry forward.

The mathematical foundation under every distributed database. These return in every module ahead.

i

Causality, not wall time

Distributed events have only a partial order. The order that matters isn't "what time it was" but "what caused what." Lamport's reframing 50 years ago is still the foundation.

ii

Lamport < Vector

Lamport gives one number per event — captures forward causality. Vector gives N numbers — captures concurrency too. The cost vs precision trade-off is the choice.

iii

Real systems are hybrid

HLC, TrueTime, single-leader Lamport. Pure schemes are rare in production. But every one is built on the foundations in this module. Recognize them in the docs.

↓ UP NEXT

M.24 — Consensus
& the Raft protocol.

You can detect concurrency and order causal events. But how do replicas agree on a single sequence to apply, even when nodes fail and the network drops messages? Time for the algorithm that powers etcd, Consul, CockroachDB — and most of the modern distributed world. Raft.

Continue to Module 24 →