Streams have to come from somewhere. For most enterprises, that somewhere is an operational database that was never designed to be streamed. Trigger-based CDC hurts write latency. Polling misses changes between polls. Log-based CDC — reading Postgres WAL, MySQL binlog, MongoDB oplog directly — is how modern data pipelines source themselves. Debezium, Kafka Connect, and the specific discipline of transactional ordering, snapshot initialization, and schema evolution.
Modules 39–41 assumed events arrived at Kafka on their own — from web servers, from mobile SDKs, from IoT devices publishing directly to a stream. That's the greenfield model, and it works for pipelines that begin at "click event" or "page view." But most enterprise data has always lived in operational databases: users, orders, invoices, inventory, customer records. These systems long predate the streaming platforms that now want to consume them, and they were designed for a completely different access pattern — transactional queries from application code, indexed by primary key, with strict ACID guarantees. Getting a stream of "everything that changed in the users table" out of Postgres or MySQL is not what these databases were built to do. The mechanism you use — polling, triggers, or log-based CDC — determines whether the resulting stream is correct, complete, ordered, and operationally sustainable. It's the "how do we source the pipeline?" problem that every real streaming platform has to solve.
users table to Kafka, in commit order, without missing any, without slowing the database. The four requirements — complete, ordered, low-impact, recoverable — are the specific hurdles every CDC mechanism has to clear. Trigger-based and polling approaches fail at least one of these; log-based CDC is the mechanism that satisfies all four, which is why it's now the industry default.The stakes are high because the change stream is upstream of everything downstream stateful streaming does. If M.41's pipeline is running exactly-once semantics with perfectly-tuned checkpoints and RocksDB state, but the source stream is silently missing 3% of DELETE events, the downstream state is systematically wrong regardless of how correctly the streaming machinery processes it. Correctness starts at the source. Every "why don't our numbers match production?" incident that gets escalated eventually traces back to either an event-time / watermark issue (M.40) or a CDC completeness / ordering issue (M.42). The two together — event semantics and source semantics — are where streaming pipelines earn or lose their trustworthiness. Get one wrong, and no amount of downstream engineering will save you.
SELECT * FROM users, diff against last snapshot, emit changes." Works for tiny tables and dev environments. For a 10M-row table, each scan is a full sequential scan — minutes of I/O, high memory to hold two snapshots, catastrophic impact on OLTP write performance during the scan. Misses DELETE entirely unless you also track which rows disappeared. Misses intra-poll changes — a row updated twice in 5 minutes shows up as one update. No transactional grouping — related changes to different tables are emitted separately with no way to correlate.// FAIL MODE: catastrophic DB impact + misses changes + no orderingupdated_at column with a trigger to update it; poll WHERE updated_at > :last_seen every 30 seconds; track high-watermark cursor." Better — the query hits an index and the volume is bounded by change rate. But: misses DELETE entirely (deleted rows don't have updated_at anymore, they're gone). Race conditions around updated_at equality — two updates in the same millisecond can be missed or double-emitted. Requires schema modification. And the polling interval defines the freshness ceiling — 30-second CDC pipelines can never do sub-30-second downstream work.// FAIL MODE: misses DELETE + polling latency + can't detect DDLcdc_events audit table; a consumer polls the audit table." Captures DELETE correctly (trigger fires before the row is actually gone). Captures transactional grouping via txid_current(). But: every write to the source table now writes to two tables — write latency roughly doubles, throughput drops. Under high write load, the audit table becomes a hot contention point. Trigger code is DB-specific and fragile. And the audit table must be aggressively purged or it becomes larger than the source. The pattern from 2005-2015; still works, but it's the wrong shape for modern high-throughput systems.// FAIL MODE: write amplification + operational fragility + trigger burdenEach attempt fails at a specific requirement — completeness (1 and 2 miss changes), low-impact (1 catastrophic, 3 doubles write cost), recoverability (2 has cursor races). Only attempt 4 — log-based CDC reading the transaction log directly — satisfies all four requirements simultaneously. This is why the entire industry has converged on log-based CDC as the sourcing mechanism: the database already produces the exact stream you need as part of its normal operation; CDC just needs to consume that stream.
The historical arc mirrors the evolution of thinking. The 2005-2010 era was dominated by trigger-based patterns — Oracle GoldenGate, Informatica, IBM InfoSphere. These worked at moderate scale, but write-amplification became prohibitive as workloads grew. Around 2012, LinkedIn's Databus paper described one of the first industrial log-based CDC systems — reading MySQL binlog directly to feed downstream services. Kafka Connect (2015) and Debezium (2016) made log-based CDC accessible outside FAANG — an open-source Kafka Connect plugin that speaks Postgres logical replication, MySQL binlog, MongoDB oplog, and SQL Server change tracking. Debezium is now the mainstream default — the "if you're building a CDC pipeline today, start with Debezium" answer that every senior data engineer converges on. Managed services (AWS DMS, GCP Datastream, Azure Change Data Capture) have proliferated for teams that don't want to operate Kafka Connect. The 2020s pattern is "streams sourced from CDC, processed by Flink/Materialize, materialized to read-optimized stores" — end-to-end streaming from database changes to user-facing dashboards.
The specific insight that makes log-based CDC work is that operational databases are already producing exactly the stream you need — the transaction log — for their own purposes. Postgres writes a Write-Ahead Log (WAL) for durability and replication; MySQL writes a binary log (binlog) for the same reasons; MongoDB writes an operation log (oplog) so replica set members can catch up. These logs contain every INSERT, UPDATE, and DELETE in the order they were committed, with transaction boundaries preserved, in a durable and replayable format. CDC's job is not to invent a new source of change events — it's to consume the log that already exists. This is what makes log-based CDC low-impact (reads happen after the fact, out of the write path), complete (the log has every change by construction), and ordered (the log is a total order over committed transactions).
pgoutput plugin, and publishes to per-table Kafka topics. On restart, Debezium reads the last committed LSN from Kafka Connect's offset topic and resumes from that point.Postgres LSN (Log Sequence Number) is the WAL's monotonically-increasing byte offset — format like 0/1A4F82C0. Every WAL record has an LSN; the consumer tracks "last successfully processed LSN" and asks Postgres for records after it. LSN is the primary key of the log. MySQL uses binlog file + position; MongoDB uses oplog timestamp.
A Postgres construct that reserves WAL from being cleaned up while a consumer hasn't yet processed it. WAL between the slot's confirmed LSN and the current LSN is retained on disk; deleting the slot releases it. Failing to delete unused slots is anti-pattern §05.iv — disk fills up.
The WAL is binary, format-versioned, and complex. A logical decoding plugin (Postgres's pgoutput, older wal2json) transforms binary WAL records into structured messages. Plugin runs inside Postgres; the output is what's sent to the consumer over the replication protocol.
Every WAL record includes the transaction ID; Debezium can preserve transaction boundaries in the output stream via BEGIN and COMMIT marker events. Downstream consumers can apply all changes in a transaction atomically. This is how CDC preserves the source database's ACID guarantees downstream.
The LSN cursor mechanism (i) is what makes CDC recoverable. Debezium stores the last committed LSN in Kafka Connect's offset topic. On restart — whether from a crash, a redeployment, or a Kafka Connect rebalance — Debezium reads its last LSN and asks Postgres "give me all WAL records after this LSN." Because Postgres's replication slot has been retaining WAL from that point forward, the records are available; because LSNs are monotonic, there's no ambiguity about ordering. No events are lost; duplicates are possible only within the last unconfirmed batch (which downstream systems can dedupe by LSN if needed).
The replication slot (ii) is the operational surface that most often causes production incidents. Postgres cannot garbage-collect WAL that hasn't been consumed by all active slots — if Debezium is down, falling behind, or misconfigured such that its LSN cursor doesn't advance, Postgres accumulates WAL on disk indefinitely. A busy database can produce gigabytes of WAL per hour; disks fill up in days. When the disk fills, Postgres stops accepting writes — a full outage caused by an unrelated CDC consumer. Monitoring "replication slot lag" as a first-class SLI is non-negotiable, and Debezium/Postgres deployments should include alerts on slot lag beyond a threshold (10GB of retained WAL, 30 minutes of processing lag). Anti-pattern §05.iv is specifically about this failure mode.
The equivalent mechanisms in other databases follow the same pattern with different specifics. MySQL binlog: enable ROW format binary logging (not STATEMENT, which is old and lossy); Debezium reads via the MySQL replication protocol, tracks position as (binlog file, byte position). MongoDB oplog: a capped collection in the local database that records every operation on the primary; Debezium tails it via a change stream cursor. SQL Server: has native "Change Data Capture" features. Oracle: LogMiner reads redo logs; GoldenGate also targets this. Debezium supports all major databases through a unified connector abstraction — the operator writes essentially the same configuration regardless of source DB, and the output messages have a consistent schema.
The mechanism in §02 works perfectly for capturing changes that happen AFTER CDC is set up. But the source database usually already contains data — a users table with 10M rows, an orders table with 500M rows, invoices going back years. How do you initialize a CDC pipeline on a table that already has content? The naive answers all fail: start streaming from now (missing all historical rows), pause writes and copy the full table (unacceptable downtime), do a full table scan while streaming (race conditions). The correct answer — consistent snapshot at a known LSN, then stream from that LSN forward — is subtle enough that most CDC implementations get it wrong on the first attempt.
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ and pg_export_snapshot(), the snapshot reads the table as it existed at exactly that LSN, ignoring any concurrent commits. Debezium records LSN N BEFORE the snapshot starts, streams the snapshot rows as synthetic op=r events, then begins streaming the WAL from LSN N+1. The op-code discipline is what lets downstream state machines apply snapshot and stream events identically without needing to know which phase they came from.Acquire LSN, then bulk-read the table using a transaction isolation snapshot pinned to that LSN. Every row emitted as an event with op=r. May take hours for large tables; Debezium's incremental snapshot mode chunks this so it interleaves with streaming (avoids blocking).
Stream WAL from LSN_snap forward at maximum rate to catch up to current LSN. During snapshot, writes continued — the WAL between LSN_snap and now contains those changes. This phase replays them in commit order. Ends when consumer LSN is close to current WAL LSN.
Continuous WAL tail at sub-second latency. Debezium reads new WAL records as they're written; publishes to Kafka. This is the mode CDC pipelines run in for their entire operational lifetime — years — punctuated only by restarts (which resume from the last committed LSN) and rare re-snapshot events.
The op-code convention (r/c/u/d) is the specific engineering discipline that makes snapshot and stream uniformly consumable. From the downstream perspective, an event with op=r (a snapshot row) and an event with op=c (a live INSERT) both represent "this row exists with these values" — the downstream state machine applies them identically. The consumer doesn't need to know or care which phase it's in. It just processes events in order, maintaining a running materialized view of the source table's current state. This uniformity is what makes CDC downstream code simple — the same operator that ingests steady-state changes handles the bootstrapping snapshot with no special logic.
The incremental snapshot mechanism deserves specific mention because it addresses a real production problem. The naive snapshot phase is a bulk SELECT * FROM users that streams the whole table before any streaming begins. For a 500M-row table, this can take hours; during those hours, WAL accumulates on the replication slot; if the snapshot fails and restarts, all that WAL retention was wasted. Incremental snapshots (Debezium 1.6+) interleave snapshot chunks with streaming — take a chunk of rows (say, 10K rows keyed by primary key range), stream them, then process any incoming WAL changes, then take the next chunk. Debezium's implementation is based on the DBLog paper (Netflix, 2019), which formalizes the incremental-snapshot approach. For any large-table CDC pipeline, incremental snapshots are essentially mandatory.
The failover story is where the snapshot-plus-stream discipline gets tested. When Postgres fails over from primary A to primary B, the replication slot on A doesn't automatically exist on B — it needs to be created on the new primary, and its starting LSN needs to match (or exceed) the last consumer LSN. If the new primary's WAL doesn't cover the range from the consumer's last LSN to the current LSN, there's a gap: some changes committed on A shortly before failover may not have been shipped to B. Modern Postgres deployments use synchronous replication or failover slots (Postgres 17+) to avoid this failure mode. Debezium's Postgres connector has specific failover handling: fail with manual intervention (safe) or reinitialize from a fresh snapshot (fast, but loses ordering across failover). Anti-pattern §05.iv addresses the "no failover story" version.
Below: each of the three CDC strategies (trigger-based · polling · log-based) evaluated against three different challenge scenarios (schema evolution · database failover · backfill from empty). Watch how each strategy handles the specific engineering tests that separate production-grade CDC from prototype-grade. The 9 combinations illustrate why "which CDC strategy?" is a decision about which failure modes you can tolerate.
The failure modes of CDC are especially treacherous because they surface downstream, in the pipelines that consume the CDC stream, not in the CDC layer itself. The Debezium metrics look green; the Kafka topic has messages; the downstream aggregations don't match production numbers. Tracing the discrepancy back to a specific CDC configuration issue — wrong strategy, missed schema change, unhandled failover, stale replication slot — is what separates senior data engineers from those still learning the discipline.
Trigger-based CDC doubles the write cost of every transaction because every INSERT/UPDATE/DELETE now writes to two tables. Under normal load, the overhead is 15-30%; under peak load, contention on the audit table (hot rows, index updates, lock waits) can drop throughput by 40-60%. Trigger code is DB-specific and fragile — schema changes require trigger migration; new tables require trigger installation; DBA operations (bulk updates, schema migrations) run 2× slower. The fix: migrate to log-based CDC (Debezium) which has zero write-path overhead. This is one of the specific migrations that pays for itself in the first month at scale. Trigger-based CDC was appropriate in 2005; it's an anti-pattern in 2025.
WHERE updated_at > :last_seen every 30 seconds. Downstream reconciliation shows we're missing 2% of DELETEs. The rest of the data looks fine."Timestamp-column polling has three fundamental correctness problems. (1) Cannot detect DELETE — deleted rows don't have updated_at anymore. (2) Race conditions at cursor boundary: two updates with the same updated_at can be missed (excluded by >) or double-emitted (included by >=); high-throughput databases produce these regularly. (3) Polling latency is the freshness ceiling: 30-second polls mean downstream can never see changes faster than 30 seconds. The fix: for anything more than a proof-of-concept, switch to log-based CDC. If you must use polling (managed DB doesn't expose WAL), use soft deletes (deleted_at column, DELETE becomes UPDATE) so the "missing DELETE" problem doesn't exist. Polling is fine for warehouses; it's not fine for real-time downstreams.
CDC emits per-table topics, but real-world transactions modify multiple tables atomically. When downstream applies changes to different tables in the order they arrive at each topic (rather than in transactional order across topics), it can observe intermediate states that never existed in the source. The fix has multiple approaches: (a) Use Debezium's outbox pattern — application writes to a single outbox table within its transaction; CDC emits from just that table; downstream sees atomic events. (b) Use Debezium's transaction metadata topic + downstream state machine that buffers changes until BEGIN/COMMIT boundary. (c) Accept eventual consistency and design downstream to handle intermediate states. Transactional grouping is preserved in the WAL — CDC must preserve it end-to-end for downstream to observe consistent states.
Two compounding failures: (1) Debezium didn't have a failover plan when the primary changed; (2) the abandoned replication slot on the old primary prevented WAL cleanup. Postgres cannot garbage-collect WAL that hasn't been consumed by all active slots; the abandoned slot retained WAL indefinitely; the disk filled; the database refused new writes. This is one of the most damaging CDC failure modes because it turns a routine failover into a full outage. The fix: (a) monitor replication slot lag as a primary SLI with page-level alerts; (b) use Postgres 17+ failover slots that migrate to standbys automatically; (c) have runbooks for "drop abandoned slot and reinitialize CDC" that DBAs can execute within minutes; (d) test failover in staging with CDC connected — most teams discover the gap only when it hits production. Replication slot rot is anti-pattern §05.iv's specific form; the outage it causes is disproportionate to the underlying config error.
CDC captures schema changes in the WAL, but downstream consumers must be able to evolve their schemas in sync. Debezium uses schema registries (Confluent Schema Registry, Apicurio) to publish schema evolution events; downstream consumers must respect compatibility rules (backward-compatible: add optional fields; forward-compatible: never remove required fields). A required column added to the source without downstream preparation breaks every consumer. The fix: (a) treat schema changes as a coordinated deploy — announce in advance, deploy downstream compatibility first, then apply source schema change; (b) use Avro or Protobuf with strict compatibility rules enforced at registry level; (c) test schema evolution in staging before production; (d) monitor "downstream deserialization errors" as a first-class SLI. CDC pipelines have their own release-engineering discipline that mirrors the state-migration discipline from M.41 — savepoints on the streaming side, schema-registry coordination on the sourcing side.
The composite pattern across all five is that CDC correctness is a whole-pipeline concern, not a component concern. Getting Debezium configured correctly is the easy part; the failure modes come from the interactions between the source database's operational reality (failovers, schema changes, capacity events), the CDC layer's assumptions (LSN continuity, slot retention, schema evolution), and the downstream consumers' expectations (transactional consistency, ordered events, schema stability). Mature CDC deployments treat all three layers as a coordinated system with shared SLIs: replication slot lag, downstream schema deserialization errors, per-topic freshness, downstream reconciliation gaps. Each metric maps to a specific class of failure mode; monitoring them together is what makes CDC pipelines reliable at multi-year time scales.
The Phase H roadmap ends with the module that closes the loop. M.43 (Real-Time Analytics) covers how the streams produced by CDC pipelines like the ones in this module — and processed by Flink/M.41 with correct windowing/M.40 — get materialized into read-optimized stores that user-facing dashboards and applications can query. The end-to-end pattern is: Postgres → Debezium → Kafka → Flink → Materialize/RisingWave/Pinot → user dashboards. Each stage has its own discipline; M.42 established the source discipline, and M.43 will establish the sink discipline that makes the streaming data actually queryable at analytical latencies.
The terms every "why aren't our downstream numbers matching?" incident review, every Debezium connector RFC, every "is our replication slot leaking WAL?" postmortem assumes you already know.
binlog; MongoDB's is the oplog.0/1A4F82C0). The primary key of the WAL. Every CDC consumer tracks its last-processed LSN; on restart, asks for records after that LSN.pgoutput, wal2json) transform binary WAL records into structured messages consumable by external readers. The mechanism Debezium uses to read WAL without parsing binary internals.r (READ — snapshot row), c (CREATE — INSERT), u (UPDATE), d (DELETE). Downstream applies snapshot and stream events uniformly by treating r and c as equivalent "row exists" signals.outbox table within its transaction; CDC emits from just that table. Solves the transactional-boundary problem: downstream sees atomic multi-table changes as coherent events. Preserves ACID guarantees end-to-end.Test the vocabulary. Click an answer; explanation drops in instantly.
Perfect. WAL, LSN, replication slots, Debezium, outbox pattern, incremental snapshots — the specific vocabulary that makes CDC pipelines survive real production. Next up: real-time analytics (M.43), the Phase H finale, where the CDC-sourced streams meet queryable state and end-to-end streaming as a data platform.
The discipline that makes streams sourced from operational databases trustworthy.
Every operational database writes a total-ordered log of committed changes for its own durability and replication purposes: Postgres WAL, MySQL binlog, MongoDB oplog. CDC's job is to consume that log, not to invent a new source of change events. Log-based CDC (Debezium) satisfies completeness, ordering, low-impact, and recoverability simultaneously — trigger-based and polling approaches all fail at least one.
Three-phase initialization — consistent snapshot at LSN N, catch-up from LSN N to current, steady-state WAL tail — produces a contiguous downstream stream that includes all historical rows plus every subsequent change. Op-code discipline (r for read/snapshot, c/u/d for stream) makes the join seamless: downstream consumers apply snapshot and stream events identically without needing to know which phase they came from.
CDC correctness lives in the interactions between source-DB reality (failovers, schema changes), CDC layer assumptions (LSN continuity, slot retention), and downstream expectations (transactional consistency, schema stability). Mature deployments monitor all three layers with shared SLIs — replication slot lag, downstream deserialization errors, per-topic freshness. The failures show up downstream; the diagnostics require the whole pipeline.