Expert Track · Phase J · 7 of 26
Every database is a storage engine dressed as a query layer. The engine\'s structure determines throughput, latency, and space — before any query optimizer runs.
Module 53 · Expert 7 / 26 · 95 min

Storage engine
internals.

Below every SQL API, below every transaction manager, sits a storage engine. B+ trees for read-heavy OLTP (PostgreSQL, MySQL), LSM trees for write-heavy workloads (RocksDB, Cassandra, TiKV), and columnar formats for analytics (Parquet, ClickHouse, DuckDB) — three fundamentally different structures with three fundamentally different performance profiles. Understanding which fits which workload — and why — is the specific competence that turns "our database is slow" into "our storage engine mismatches our workload; here\'s the specific fix."

// What you'll know by the end

  • B+ tree structure and page management
  • LSM trees, MemTables, compaction
  • Columnar layout and vectorized execution
  • WAL, write amplification, and buffer pools
§ 01 — Every database is a storage engine dressed as a query layer

The structure
on disk determines
throughput before
a query runs.

Every database — SQL, NoSQL, analytics, time-series, graph — has a storage engine underneath its query API. The storage engine is the specific data structure that organizes bytes on disk (or in memory), handles reads and writes, manages durability, and provides indexes. PostgreSQL uses B+ trees. MySQL InnoDB uses B+ trees. SQL Server uses B+ trees. Cassandra uses LSM trees. RocksDB uses LSM trees. TiKV, YugabyteDB, and CockroachDB use RocksDB underneath. ClickHouse and DuckDB use columnar formats. Snowflake uses columnar over object storage. Each storage engine has a specific write amplification (how many bytes actually get written per byte of user data), read amplification (how many bytes must be read to serve a user read), and space amplification (how much disk space is used per byte of user data). These three amplification factors — plus the compaction/checkpoint model — determine the storage engine\'s workload fit: a B+ tree is efficient for random reads but suffers under heavy random writes; an LSM tree is efficient for high write throughput but suffers under range scans with many overwrites; a columnar format is efficient for analytical scans but wasteful for OLTP row lookups. Choosing the wrong engine for a workload produces production issues that no amount of query tuning can fix.

// STORAGE ENGINE STRUCTURE · DETERMINES PERFORMANCE PROFILE
DATABASE STACK · STORAGE ENGINE IS THE FOUNDATION QUERY LAYER · SQL parser · query planner · execution engine TRANSACTION LAYER · isolation · concurrency control · MVCC STORAGE ENGINE ← M.53 lives here B+ tree (PostgreSQL) · LSM tree (RocksDB, Cassandra) · Columnar (Parquet, ClickHouse) determines: write amp · read amp · space amp · compaction cost · durability OS · FILESYSTEM · PAGE CACHE · SSD / HDD hardware // Storage engine sits between transaction layer and OS · every byte flows through it · structure matters
The storage engine layer. Between the transaction layer (which handles isolation, MVCC, concurrency control) and the OS/filesystem sits the storage engine — the specific data structure that organizes bytes on disk. Every read and every write passes through this layer. Its structure determines: write amplification (bytes written per byte of user data — 1× for pure append; 10-30× for LSM with heavy compaction; 2-5× for B+ trees with page splits); read amplification (bytes read per byte of user data — logN for B+ trees; K×logN for LSM where K is the number of levels; near-1 for columnar with prefetch); space amplification (disk space per byte of user data — 1.05-1.3× for B+ trees with fill factor; 1.1-2× for LSM depending on compaction strategy; 0.1-0.5× for columnar with compression). Each engine trades one factor for another. Choosing wrong for a workload produces performance issues that no query tuning can fix — the bottleneck is below the query layer.

The specific engineering task M.53 addresses is understanding these three storage engines precisely enough to (a) diagnose which one your existing database uses, (b) predict how it will behave under your workload, and (c) know when to choose a different one. This isn\'t rare knowledge for senior engineers — it\'s standard competence. When a PostgreSQL database hits a write throughput ceiling, the specific mechanism is B+ tree page contention plus buffer pool eviction plus WAL fsync latency. When a Cassandra cluster shows read latency spikes, the specific mechanism is LSM compaction stalling reads plus tombstone accumulation plus SSTable-count explosion. When a ClickHouse query is 100× faster than the equivalent PostgreSQL query for a specific analytical pattern, the specific mechanism is columnar layout plus vectorized execution plus SIMD-friendly compression. Each observation has a specific storage-engine mechanism as its explanation, and knowing these mechanisms is what turns "the database is slow" into "here\'s the specific fix and here\'s why it works".

// FOUR APPROACHES TO STORAGE ENGINE SELECTION · WHERE EACH FAILS OR FITS
Attempt 1: "use whatever the framework defaults to"// PostgreSQL for everything · no engine analysis
"Our framework says PostgreSQL, so we use PostgreSQL. It\'s the industry standard." Works for a large class of read-heavy OLTP workloads — that\'s exactly what B+ trees are designed for. Fails specifically when: (a) write throughput is high — B+ tree page splits under concurrent writes produce lock contention and write amplification; PostgreSQL will throttle; (b) analytical scans are common — B+ tree row-oriented layout is inefficient for scans reading few columns of many rows; queries that would be 100ms in ClickHouse take 30 seconds in PostgreSQL; (c) time-series data with high ingestion rate — B+ trees can\'t sustain millions of writes per second per node; specialized time-series engines (TimescaleDB\'s hypertables, InfluxDB\'s TSM) are 10-100× faster. Not wrong, but doesn\'t match every workload. Understanding when to reach for a different engine is Expert-tier judgment.// FAIL MODE: mismatched workload · unnecessary throughput cost
MISSED
LEVERAGE
Attempt 2: "the query layer is what matters"// spend all time tuning queries and indexes
"If the query is slow, we\'ll tune the query — add indexes, rewrite as JOINs, use CTEs. The storage engine is a black box." Works for surface-level optimization but hits ceilings quickly. A B+ tree with terrible workload fit can be tuned only so far; the fundamental structure is the constraint. Common symptoms: (a) <50% CPU utilization during peak load — bottleneck isn\'t compute, it\'s the storage engine\'s throughput ceiling; (b) tuning one query slows another — index additions increase B+ tree write amplification; (c) vacuum/compaction storms take out the system — background maintenance is the storage engine\'s way of paying for its structure; can\'t tune away. Understanding the storage engine gives you leverage the query layer can\'t provide. Every senior database engineer eventually hits the "we tuned everything and it\'s still slow" wall; the answer is usually storage-engine-level.// FAIL MODE: query tuning ceiling · engine is the constraint
SHALLOW
LEVERAGE
Attempt 3: "roll our own storage engine"// custom implementation for our specific workload
"Our workload is unique. Existing engines don\'t fit. We\'ll build our own — that\'s how the big platforms do it." Sometimes correct (Google built LevelDB and Bigtable; Facebook built RocksDB; Uber built Docstore; TimescaleDB built time-series-specific extensions). Usually wrong. Storage engines are deceptively complex — thousands of person-years have gone into RocksDB\'s tuning; the specific failure modes (write cliffs, compaction stalls, tombstone accumulation, WAL corruption edge cases) take years to encounter and fix. Rolling your own means: (a) reinventing every optimization that\'s been done; (b) hitting every failure mode that\'s been fixed; (c) supporting it forever with a specialized team. The specific rule: roll your own only when (i) you have a truly novel workload that no existing engine fits, (ii) you have the team and time to maintain it for years, (iii) the leverage is worth the cost. For virtually all teams, adapting an existing engine or choosing a different one is the right answer.// FAIL MODE: reinvent the wheel · years to production quality
USUALLY
WRONG
Attempt 4: match engine to workload characteristics// B+ tree, LSM, columnar per workload · Expert pattern
"Analyze workload characteristics (read/write ratio, access pattern, size, latency requirements, retention). Match to the storage engine designed for that profile. Use different engines for different workloads." The Expert pattern. Specifically: (a) read-heavy OLTP with random access → B+ tree (PostgreSQL, MySQL, SQL Server); (b) write-heavy workloads, sequential ingestion, key-value patterns → LSM tree (RocksDB, Cassandra, TiKV); (c) analytical scans, wide-table aggregation → columnar (ClickHouse, DuckDB, Snowflake, Parquet-on-S3); (d) time-series with high ingestion + retention → specialized time-series engines (TimescaleDB, InfluxDB, ClickHouse for large-scale). Modern polyglot persistence: use PostgreSQL for OLTP + ClickHouse for analytics + RocksDB (or Redis) for state + S3+Parquet for archived data. Each optimized for its specific workload. This is what mature production data architectures look like — different engines for different data types, coordinated through common APIs.// FIT: match engine to workload · polyglot persistence
EXPERT
PATTERN
// THE COMPOSITE PATTERN

Each earlier attempt fails specifically. Framework defaults miss workload-specific leverage. Query-layer-only tuning hits ceilings the storage engine imposes. Rolling your own underestimates the specialized engineering already done. The Expert pattern: analyze workload characteristics, match to the storage engine designed for that profile, use different engines for different workloads, understand the specific tradeoffs each engine makes. §02 walks through B+ trees vs LSM trees mechanically. §03 covers columnar plus the universal WAL mechanism. §04 lets you explore all three across three workload types.

The historical arc of storage engines is specifically about accumulating engineering wisdom for specific workloads. 1972: Bayer & McCreight publish B-tree paper. The fundamental self-balancing tree structure that dominates database indexes for the next fifty years. Later evolved into B+ trees (all data in leaves, internal nodes contain only keys) which are the specific variant used by all modern OLTP databases. Simple, elegant, well-understood, extensively optimized. 1990s: B+ trees dominate OLTP. PostgreSQL, MySQL InnoDB, SQL Server, Oracle — all use B+ trees as their primary index structure. Buffer pool management, write-ahead logging, and MVCC are all built on top. Assumption: reads dominate writes, random access is common, data fits (mostly) in memory. Works well for a large class of workloads. 1996: O\'Neil, Cheng, Gawlick, and O\'Neil publish "The Log-Structured Merge-Tree" (LSM tree paper). Recognizing that B+ trees are inefficient for write-heavy workloads (in-place updates require random writes, page splits cause write amplification), the LSM tree proposes append-only writes to sorted runs, merged in background. Specific insight: convert random writes into sequential writes at the cost of read amplification, then compact in the background to keep read amplification bounded. 2004: Google publishes Bigtable paper. Uses LSM at massive scale (Google\'s specific workload: web crawl, indexing, analytics on trillions of rows). Basis for HBase (Facebook rewrite), Cassandra (Facebook then Apache), and subsequent NoSQL systems. LSM proves out in production at unprecedented scale. 2006: Cassandra released. LSM-based distributed database, borrows from Bigtable + Dynamo. Popularizes LSM in open source. Write throughput becomes a defining feature — Cassandra can sustain millions of writes per second per cluster while B+ tree databases plateau at tens of thousands. 2011: LevelDB open-sourced by Google. Extracted LSM engine from Chrome. Small, embeddable, high-quality reference implementation. Immediately adopted for embedded storage in many systems. 2013: Facebook releases RocksDB. Fork of LevelDB, optimized for server workloads. Adds column families, transactions, checkpointing, tunable compaction. Becomes the reference LSM engine — used by TiKV, YugabyteDB, CockroachDB, ArangoDB, MongoDB\'s WiredTiger competitor, and countless embedded uses. RocksDB is the specific storage engine underneath more modern distributed databases than any other single project. 2013: Apache Parquet released. Columnar file format for Hadoop ecosystem. Inspired by Google\'s Dremel paper (2010). Enables analytical scans over massive datasets on cheap storage. Basis for modern data lakes. 2016: Apache Arrow. In-memory columnar format. Enables zero-copy sharing between systems and vectorized execution across languages. Basis for DuckDB, Polars, modern analytical engines. 2020s: Storage engine as a first-class architectural component. Distributed SQL databases explicitly declare their storage engines (CockroachDB uses Pebble — Go rewrite of RocksDB — since 2020). Analytical databases explicitly use columnar (ClickHouse, DuckDB, Snowflake). Specialized workloads use specialized engines (TimescaleDB\'s hypertables for time-series, DuckDB\'s vectorized-in-process for analytics). The specific competence: understand which engine your database uses, why, and when to choose differently. The arc explains why "storage engines are one thing" turned into "storage engines are a menu of specific structures with specific workload fits, and choosing the right one is Expert-tier data engineering."

Below every query API sits a specific data structure — B+ tree, LSM, or columnar. Its structure determines throughput before any query runs. The Expert task is matching the structure to the workload.
§ 02 — B+ trees vs LSM trees · the write vs read tradeoff

Sorted pages vs
sorted runs.
In-place vs append.

B+ trees and LSM trees are the two dominant storage engine structures for transactional workloads, and they represent fundamentally opposed answers to the same question: how do you keep data sorted on disk while still handling writes efficiently? B+ trees answer with in-place updates in sorted pages — the tree stays balanced and sorted through all operations, at the cost of expensive writes (random I/O, page splits, write amplification through the tree). LSM trees answer with append-only sorted runs plus background compaction — writes go to memory then flush sequentially to disk, sorted runs merge periodically to bound read amplification. Understanding the specific mechanics of each — where each has its performance ceiling, where each amplifies costs — is the specific competence that turns "we chose PostgreSQL" or "we chose Cassandra" from a defaults-driven decision into a workload-driven decision.

// B+ TREE vs LSM TREE · SIDE BY SIDE STRUCTURE AND WRITE PATH

B+ TREE (LEFT) vs LSM TREE (RIGHT) · WRITE PATH COMPARISON B+ TREE · sorted pages + in-place 50 | 100 | 150 20 | 35 70 | 85 120|130 10 15 22 40 55 65 75 82 88 125 140 WRITE(45): random I/O · page 40-65 loaded, updated in-place, flushed 1. Traverse root → internal → leaf (3 page reads) 2. Load leaf page 40 55 65 (random I/O) 3. Insert 45 → page becomes 40 45 55 65 4. If page full → split into 2 pages, propagate up 5. WAL append + eventual page flush READS: fast (logN) WRITES: random I/O + splits PostgreSQL · MySQL · SQL Server LSM TREE · MemTable + sorted runs MemTable (in-memory sorted) 10 22 45 55 65 82 88 140 flush L0 SST 1 SST 2 SST 3 compact L1 SST 1-50 SST 51-100 L2 SST 1-500 (larger, older data) WRITE(45): sequential · append to MemTable + WAL 1. Append 45 to WAL (sequential write) 2. Insert 45 into MemTable (in-memory) 3. Client ACK (fast!) 4. Background: MemTable full → flush to L0 SST WRITES: fast (sequential) READS: check multiple SSTs RocksDB · Cassandra · TiKV
The fundamental structural difference. B+ tree: balanced tree of sorted pages, updated in place. Writes traverse the tree (logN), load the target leaf page (random I/O), update it in place, and potentially split pages up the tree. Reads are fast (logN page reads) because everything is sorted at the page level. But writes cause random I/O and write amplification through the tree. LSM tree: writes go to an in-memory MemTable (sorted) plus a sequential WAL append. When the MemTable is full, it flushes to disk as an SSTable (Sorted String Table) — immutable, sorted, sequentially written. Multiple SSTables accumulate; background compaction merges them into larger, fewer SSTables at deeper levels. Writes are extremely fast (sequential I/O + memory insert). But reads must check multiple SSTables (plus MemTable) until they find the key, causing read amplification. The specific tradeoff: LSM trades read cost for write cost; B+ trees trade write cost for read cost. Which fits your workload depends on the read/write ratio and access patterns.
i
B+ tree write path.

Traverse tree (root → internal → leaf, ~3-4 page reads). Load leaf page (random I/O, ~4-16KB read). Insert key. If page full, split into two pages, potentially propagating splits up. Log to WAL for durability. Eventually flush dirty page to disk. Cost: 1 leaf read + 1 leaf write + possible splits + WAL append. Typical write amplification: 2-5×.

ii
LSM tree write path.

Append to WAL (sequential write, ~1KB). Insert into MemTable (in-memory sorted structure, O(logN) memory op). Client ACK — done. In background: when MemTable is full, flush to disk as a new L0 SSTable (sequential write). Compaction later merges SSTables. Cost per write: 1 WAL append + memory op. Extremely fast writes.

iii
B+ tree read path.

Traverse tree (root → internal → leaf). Return the key from the leaf. Cost: logN page reads. If pages are in buffer pool (memory), essentially free. If pages must come from disk, ~logN random I/Os. Read amplification: near 1× (only the requested key is read from the leaf page, plus tree traversal). Fast for point lookups and range scans.

iv
LSM tree read path.

Check MemTable. Check L0 SSTables (unsorted, must check all — bloom filters help). Check L1, L2, ... SSTables (sorted by key, binary search or index lookup within each). Cost: 1 MemTable check + N L0 checks + logN L1+ checks. Read amplification: 5-20× depending on levels. Bloom filters reduce false positives dramatically.

v
B+ tree amplification.

Write amp: 2-5× (leaf update + tree splits + WAL). Read amp: ~1× (single leaf page read for point lookup). Space amp: 1.05-1.3× (fill factor of ~70% typical). Optimal for read-heavy workloads with moderate write rates. Ceiling: random I/O throughput of underlying storage; concurrent writes to same page cause lock contention.

vi
LSM tree amplification.

Write amp: 10-30× (compaction rewrites data multiple times). Read amp: 5-20× (check multiple SSTables). Space amp: 1.1-2× (obsolete versions before compaction). Optimal for write-heavy workloads. Ceiling: compaction throughput; when compaction can\'t keep up with writes, "write stalls" pause the database entirely until compaction catches up.

The B+ tree write amplification (v) is worth understanding precisely because it\'s the specific mechanism that limits B+ tree databases under write-heavy workloads. Consider inserting a new key into a leaf page that\'s already 70% full. First, the page must be read into the buffer pool (if not already there — random I/O from disk). Then the key is inserted and the page is dirty. If the page fits, we\'re done — the dirty page is later flushed (another I/O). If the page is full, it splits: allocate a new page, redistribute keys, update the parent internal node to reference the new page, potentially cascading splits up to the root. Each split is a page allocation plus multiple page writes. Under sustained write load, splits become common and the write amplification climbs. Typical measured B+ tree write amplification: 2-5× under moderate load, 5-10× under heavy write load with high key entropy (random keys). Concurrent writes to the same page require exclusive locks, producing contention that further limits throughput. The specific ceiling: a well-tuned PostgreSQL B+ tree on modern NVMe SSD sustains ~50,000-100,000 random writes per second per node; beyond this, throughput degrades due to WAL fsync latency, buffer pool contention, and vacuum/autovacuum overhead. For workloads exceeding this, LSM engines like RocksDB are 10-100× faster.

The LSM tree compaction (vi) is the specific and defining feature of LSM engines — the background process that keeps read amplification bounded by merging SSTables. Two dominant compaction strategies. Leveled compaction (RocksDB default, HBase): SSTables organized into levels L0, L1, L2, ..., LN with each level ~10× larger than the previous. When L1 exceeds its size limit, one SSTable from L1 is merged with all overlapping SSTables in L2, producing new L2 SSTables. Aggressive read optimization — each level has non-overlapping SSTables (except L0), so lookups check at most one SSTable per level. Cost: high write amplification (each key can be rewritten O(logN) times as it moves through levels). Best for read-heavy LSM workloads. Tiered/size-tiered compaction (Cassandra default): SSTables grouped by size; when N SSTables of similar size accumulate, they\'re merged into one larger SSTable. Less write amplification (each key rewritten fewer times) but higher read amplification (more SSTables overlap per lookup). Best for write-heavy LSM workloads. The specific tuning discipline: choose compaction strategy based on read/write ratio, tune level count and size ratios, monitor compaction throughput vs write rate, alert on compaction lag. Compaction that can\'t keep up produces "write stalls" — RocksDB will slow client writes to prevent SSTable count from exploding. Real production LSM incidents almost always involve compaction: either misconfigured (wrong strategy), under-resourced (insufficient background thread count), or over-triggered (too-small SSTables create compaction storms).

The read amplification of LSM (iv) deserves specific attention because it\'s counterintuitive — LSM engines can serve reads reasonably fast despite checking multiple SSTables. The specific techniques that make this work: Bloom filters. Each SSTable has a bloom filter — a probabilistic data structure that answers "does this SSTable definitely NOT contain key X?" with certainty (no false negatives) but "might contain key X" with configurable false-positive rate (typically 1%). For a point lookup, a well-tuned bloom filter means checking only 1-2 SSTables on average despite having 5-20 levels. Block cache. Recently-accessed SSTable blocks are cached in memory. Hot data effectively lives in memory; cold data pays disk I/O. Similar to B+ tree buffer pool but for SSTable blocks. Compaction-driven consolidation. Compaction merges overlapping SSTables, reducing the effective read amplification for hot key ranges. Frequently-updated ranges get compacted more, so lookups on them check fewer SSTables. Level-aware indexing. Each SSTable has a sparse index (typically 1 entry per ~64KB block) allowing binary search within the SSTable. Combined with bloom filter negative-check, a "hit" typically requires one binary search + one block read. Real-world LSM read latency: ~50-200μs for uncached hot data on modern NVMe, competitive with B+ trees for random access. Range scans are slower than B+ trees because you must merge streams from multiple SSTables, but tolerable for most workloads. Understanding these techniques is what turns "LSM has high read amp" into "LSM read amp is bounded and mitigated through specific engineering; here\'s what to monitor and tune".

B+ tree pays for writes to keep reads cheap. LSM pays for reads (mitigated by bloom filters + compaction) to keep writes cheap. The workload picks the winner.
§ 03 — Columnar storage + WAL · the analytical layout and universal durability

Rows vs columns.
Random vs vectorized.
WAL: the universal.

Columnar storage is the third dominant storage engine structure, optimized for a specific workload class distinct from OLTP: analytical scans that aggregate over many rows but only touch a few columns. Where B+ trees and LSM trees store rows contiguously (row-oriented layout), columnar formats store each column contiguously (column-oriented layout). This structural difference produces order-of-magnitude performance differences on analytical queries — a query like SELECT AVG(amount) FROM sales WHERE year = 2024 reads only 2 of 50 columns in a columnar format vs all 50 columns in a row-oriented format. Combined with column-specific compression (dictionary encoding, RLE, delta encoding) and vectorized execution (SIMD operations on column batches), columnar engines like ClickHouse, DuckDB, Snowflake, and Parquet-on-Spark achieve 10-100× faster analytical query performance than row-oriented databases on the same hardware. Meanwhile, WAL (Write-Ahead Log) is the universal durability mechanism across all storage engines — the specific append-only log that ensures crash recovery is possible for every write.

// COLUMNAR LAYOUT vs ROW-ORIENTED · AND WAL AS UNIVERSAL DURABILITY

ROW-ORIENTED vs COLUMNAR · STORAGE LAYOUT + WAL UNIVERSAL DURABILITY ROW-ORIENTED (B+ tree, LSM) id:1 name:Alice amt:$50 year:2024 id:2 name:Bob amt:$75 year:2023 id:3 name:Cara amt:$40 year:2024 SELECT AVG(amt) WHERE year=2024 → read ALL columns of every row · wasteful COLUMNAR (Parquet, ClickHouse) id 1 2 3 name Alice Bob Cara amt $50 $75 $40 year 2024 2023 2024 SELECT AVG(amt) WHERE year=2024 → read ONLY amt + year columns · 100× less I/O WRITE-AHEAD LOG (WAL) · UNIVERSAL DURABILITY MECHANISM CLIENT write X WAL APPEND fsync ACK CLIENT later apply to engine flush to disk // ON CRASH: replay WAL from last checkpoint · deterministic recovery WAL is sequential (fast fsync), append-only (no locking), single source of durability truth. Every storage engine — B+ tree, LSM, columnar — uses WAL for crash-safe writes. group commit: batch multiple WAL entries per fsync → 10-100× throughput improvement
Two orthogonal structural choices. Row vs columnar layout: row-oriented (B+ tree, LSM) stores whole rows contiguously — fast for retrieving all columns of a row (typical OLTP), slow for scans reading few columns of many rows. Columnar (Parquet, ClickHouse) stores each column contiguously — 10-100× faster for analytical scans that read few columns of many rows, slower for OLTP row lookups (must assemble row from separate column locations). WAL as universal durability: every storage engine uses a write-ahead log for crash safety. Client write → append to WAL (sequential) → fsync (~100μs on NVMe) → ACK client. Later, apply the write to the actual storage engine (B+ tree page, LSM MemTable, columnar column). On crash, replay WAL from last checkpoint to recover. WAL is fast because it\'s sequential and append-only; group commit batches multiple writes per fsync for 10-100× throughput improvement (100 concurrent writes → 1 fsync instead of 100).
i
Columnar layout.

Data organized column-by-column instead of row-by-row. Reading one column doesn\'t require reading others. For queries touching few columns of many rows (analytical scans), reads only the needed columns from disk — often 10-100× less I/O than row-oriented. Enables SIMD-friendly column-wise compression and vectorized execution over batches of column values.

ii
Column encoding.

Each column stored with type-specific compression. Dictionary encoding: for low-cardinality columns (country, status), replace values with dictionary indices — often 10-50× compression. Run-length encoding (RLE): for repeated values, store (value, count) pairs. Delta encoding: for sorted/near-sorted columns, store differences from previous value. Combined, columnar achieves 5-30× smaller storage vs row-oriented.

iii
Vectorized execution.

Query operators process batches of column values (typically 1024-8192 at a time) using SIMD CPU instructions. Instead of processing one row at a time (row-oriented), process a vector of column values. Modern CPU SIMD (SSE, AVX-512) applies operations to 4-16 values in parallel per cycle. Vectorization + columnar layout produces 10-100× analytical query speedup on same hardware.

iv
Where columnar fails.

Row assembly for OLTP lookups is expensive — reading one row means fetching one value from each column\'s storage. Writes are slow (must update multiple column files). Point lookups are slow. Small transactions with mixed operations are inefficient. Columnar is a specific tool for a specific workload — analytical scans — and misapplying it to OLTP produces catastrophic performance.

v
Write-Ahead Log (WAL).

Universal durability mechanism across all storage engines. Sequential append-only log persisted before any change to the actual storage. Client write → WAL append → fsync → ACK. On crash: replay WAL from last checkpoint. Fast because sequential I/O + no locking. Every write in every storage engine (B+ tree, LSM, columnar) goes through WAL first.

vi
Group commit.

WAL optimization: instead of fsync per write (100 writes → 100 fsyncs), batch multiple writes into single fsync (100 writes → 1 fsync). Each fsync ~100μs on NVMe; batching produces 10-100× throughput improvement. Trade-off: individual write latency slightly higher (waits for batch). Standard in PostgreSQL, MySQL, RocksDB, and every serious storage engine.

The columnar layout advantage (i) is worth understanding precisely because it drives virtually all modern analytical databases. Consider a sales table with 50 columns and 100 million rows, and the query SELECT AVG(amount) FROM sales WHERE year = 2024. In a row-oriented engine, executing this query requires reading each row (all 50 columns) from disk, extracting the year and amount columns, filtering by year, and averaging. Total I/O: 100M rows × 50 columns × ~20 bytes/column ≈ 100GB read. Even with indexes, the leaf pages contain full rows and must all be scanned. In a columnar engine, only two columns are read: year (100M × 4 bytes = 400MB) and amount (100M × 8 bytes = 800MB) — total 1.2GB, ~80× less I/O. Additionally, both columns are compressed — year (integer, likely low cardinality — 5-10 distinct values) compresses to ~10MB via dictionary encoding; amount compresses to ~200MB. Actual disk I/O: ~210MB — nearly 500× less than row-oriented. Add vectorized execution using SIMD instructions and the query runs 100× faster than PostgreSQL on the same hardware. This 100-1000× speedup for analytical queries is why every modern data warehouse (Snowflake, BigQuery, Redshift, ClickHouse, DuckDB) uses columnar storage. It\'s not a marginal optimization; it\'s a fundamental structural advantage for the analytical workload class.

The WAL mechanism (v) is the specific engineering trick that makes every serious storage engine crash-safe without sacrificing write throughput. The problem WAL solves: to ensure a write is durable, it must be on disk before the client is told "committed." But writing to the actual storage structure (updating a B+ tree page, updating a MemTable + eventually flushing) involves random I/O and complex data structures that are slow to persist synchronously. WAL\'s solution: write a sequential append-only log entry first (fast — sequential I/O + fsync), then update the actual storage lazily later. Client sees: write → WAL append → fsync → ACK. The actual storage update happens in background. On crash, recovery replays WAL from the last checkpoint, reconstructing any writes that weren\'t yet applied to the storage. This gives durability (write survives crash) at the cost of one sequential I/O + fsync per write. Modern NVMe fsync: ~100μs. So per-write durability cost: ~100μs. But group commit batches multiple concurrent writes into a single fsync — 100 concurrent writes = 1 fsync at ~100μs instead of 100 fsyncs at ~10ms total. This is why PostgreSQL, MySQL, and RocksDB can all sustain tens of thousands of writes per second per node while maintaining crash safety. WAL + group commit is the specific engineering pattern; understanding it precisely is what distinguishes "we use PostgreSQL" from "we understand what \'synchronous_commit=on\' actually costs and when to relax it for throughput."

The vectorized execution (iii) is the specific execution model that combines with columnar layout to produce the analytical speedup. Traditional row-at-a-time execution (Volcano model, used by PostgreSQL and most OLTP engines) processes one row through the query operators one at a time — next_row = filter.next(); result = aggregate.process(next_row); return next_row. Each row incurs function call overhead, branch prediction misses, and cache thrashing. Modern vectorized execution (used by ClickHouse, DuckDB, Snowflake) processes batches of column values through operators — batch = filter.next_batch(); aggregate.process_batch(batch); return batch. A batch is typically 1024-8192 column values. This produces specific efficiencies: (a) function call overhead amortized over 1000+ values; (b) tight loops enable auto-vectorization by the compiler using SIMD (AVX2/AVX-512 for x86, NEON for ARM); (c) cache-friendly access patterns — sequential column values fit in L1/L2 cache; (d) branch prediction becomes trivial because branches are batch-wide, not per-value. Combined result: modern vectorized columnar engines like DuckDB and ClickHouse achieve close to memory bandwidth throughput for aggregations — often 1-10 billion values processed per second per core, compared to ~10 million rows per second for a well-tuned PostgreSQL query. This is why analytical workloads should use analytical engines, not OLTP engines — the structural difference is a two-order-of-magnitude speedup that no query tuning can overcome.

Row-oriented for OLTP. Columnar for analytics. WAL for durability everywhere. Group commit for throughput. Each is a specific tool with a specific workload fit.
§ 04 — Storage engine explorer

Three engines.
Three workloads.

Below: each of three storage engines (B+ tree · LSM tree · Columnar) evaluated against three workload profiles (Write-heavy OLTP · Read-heavy OLTP · Analytical scans). Watch how each engine fits or fails each workload — the sharp diagonals show exactly which engine suits which workload, and the off-diagonals show the specific performance costs. This is the matrix Expert engineers implicitly consult when choosing databases.

STORAGE.SIM // m.53 lab
Workload →
// STORAGE ENGINE BEHAVIOR · under current workload
// METRICS · PERFORMANCE / EFFICIENCY PROFILE
Write throughput-
Read latency-
Write amplification-
Space usage-
Compaction cost-
Overall fit-
// VERDICT
Loading...
...
§ 05 — Where storage engine choices decay

Every storage-engine
bug is a workload
mismatch.

The failure modes of storage engines are the specific mechanisms by which "we chose PostgreSQL" turns into "PostgreSQL can\'t sustain our write rate" or "our Cassandra cluster is unusable due to compaction storms" or "our analytical query takes 30 minutes instead of 300 milliseconds." Each of these anti-patterns is a real pattern with a specific mitigation that Expert engineers deploy by default. Recognizing them at architecture time avoids the migration under time pressure.

// FIVE STORAGE-ENGINE ANTI-PATTERNS

i
The LSM for read-latency-critical
"We chose Cassandra for our user session store because ‘NoSQL scales.\rsquo; Point lookup latency is 20ms p50 and 300ms p99 due to compaction happening. Users complain about slow logins. Read amplification is 15× because we have 20 SSTables per level."

LSM engines have inherently variable read latency due to compaction and multi-SSTable lookups. For workloads where p99 read latency matters more than write throughput, LSM is the wrong choice. The specific mechanism: reads check MemTable + multiple SSTables. Bloom filters help but don\'t eliminate the cost. During compaction, read latency spikes as SSTables are being rewritten. In steady state, well-tuned LSM p50 is ~1ms; but p99 during compaction can be 100-500ms. For a session store where consistent low latency matters more than 100K writes/sec, a B+ tree engine (PostgreSQL) or in-memory store (Redis) is the correct choice. The fix: (a) migrate to PostgreSQL/MySQL for read-latency-critical OLTP; (b) use Redis for hot session data with periodic snapshots to persistent storage; (c) if staying on LSM, tune compaction aggressively (leveled compaction, smaller L0, more compaction threads) — helps but doesn\'t eliminate the p99 tail; (d) accept the p99 latency if write throughput requirements dominate. The general principle: LSM optimizes write throughput at the cost of read latency variance; use it only when write throughput is the constraint.

ii
The B+ tree for high write throughput
"Our PostgreSQL cluster is at 40K writes/sec and can\'t go higher. We\'ve added indexes, tuned autovacuum, upgraded to NVMe. Buffer pool contention is the bottleneck. Any single hot table saturates lock contention on the same B+ tree pages."

B+ trees have specific write throughput ceilings driven by page contention, write amplification, and vacuum overhead. Beyond these ceilings, no tuning helps — the structural bottleneck is the tree itself. Random writes cause page splits, which propagate up the tree and require exclusive locks. Concurrent writes to the same page block. Vacuum reclaims dead tuples but adds background load. Well-tuned PostgreSQL on modern NVMe sustains ~50-100K random writes/sec per node; beyond this, structural limits dominate. The fix: (a) for time-series or append-only workloads, use LSM engines (RocksDB, TimescaleDB\'s hypertables) — 5-20× higher write throughput; (b) partition data across multiple B+ trees (sharding) — throughput scales linearly with shards; (c) use write-optimized index types (BRIN for time-series columns, hash for point lookups); (d) batch writes to reduce lock contention; (e) migrate write-heavy tables to a different engine while keeping OLTP on B+ tree. The general principle: B+ trees have hard write throughput ceilings around 100K/sec/node; for higher throughput per node, choose LSM.

iii
The columnar for OLTP
"We tried using ClickHouse for our order processing system because ‘analytics is fast.\rsquo; Placing a single order requires reading and writing 30 columns — costs 300ms per operation because columnar layout requires assembling the row from separate column storage. We migrated back to PostgreSQL."

Columnar formats are specifically optimized for analytical scans (few columns, many rows) and are catastrophically inefficient for OLTP (many columns, single row). Row assembly requires random access to each column\'s storage — the opposite of what columnar layouts optimize for. Point lookups can be 100-1000× slower than a B+ tree. Writes are equally bad (must update multiple column files, disrupt compression). The fix: (a) use OLTP engines (PostgreSQL, MySQL) for transactional workloads with single-row reads/writes; (b) use columnar engines (ClickHouse, DuckDB, Snowflake) only for analytical workloads with aggregation over many rows; (c) use ETL/CDC to replicate OLTP data to columnar analytics store — best of both; (d) modern databases like DuckDB support both by allowing OLTP-friendly single-row operations in-memory with columnar disk layout, but this is a special case. The general principle: match engine class to workload class — OLTP goes to row-oriented, analytics goes to columnar; mixing produces catastrophic performance.

iv
The ignored compaction tuning
"Our RocksDB deployment worked fine for months. Now we have 500 SSTables in L0, write stalls happen every few minutes, and compaction is 10 hours behind. Nothing works. Adding more storage doesn\'t help; the compaction throughput is the bottleneck."

LSM engines require ongoing compaction tuning; when write rate exceeds compaction throughput, SSTable counts grow and reads slow catastrophically. Specific mechanism: writes create L0 SSTables faster than compaction can merge them. RocksDB has configurable stall thresholds (e.g., stop writes if L0 has >30 SSTables) — beyond these, client writes are throttled or paused entirely. Root cause is usually: (a) too few compaction threads; (b) compaction strategy mismatch (using tiered when leveled is needed, or vice versa); (c) I/O throughput ceiling (SSD saturated by compaction reads/writes); (d) memory pressure limiting MemTable size. The fix: (a) increase compaction thread count (RocksDB max_background_compactions parameter); (b) switch to leveled compaction if reads are latency-sensitive; (c) increase L0 SSTable size to reduce compaction frequency; (d) provision more I/O bandwidth (bigger NVMe, more disks); (e) monitor compaction lag as a first-class metric — alert when SSTable count exceeds thresholds. The general principle: LSM compaction is a background process that must keep pace with writes; monitor and tune compaction as a first-class operational concern.

v
The undersized buffer pool
"Our PostgreSQL query latency has degraded from 5ms to 200ms over the past month. Data has grown; buffer pool hit ratio has dropped from 99% to 40%. Every query now hits disk. We\'d assumed the default shared_buffers was enough."

B+ tree databases depend heavily on the buffer pool (cache of hot pages in memory) — when working set exceeds buffer pool, every query becomes disk-bound and latency degrades by 10-100×. Specific mechanism: PostgreSQL\'s shared_buffers holds cached B+ tree pages; if a query needs a page not in cache, it triggers disk I/O (~100μs on NVMe vs ~10ns memory access — 10000× slower). Buffer pool hit ratio dropping from 99% to 40% means 60% of page accesses now hit disk instead of memory — catastrophic latency degradation. The fix: (a) size buffer pool to fit working set — typically 25-50% of total system memory (PostgreSQL shared_buffers) or 50-75% (MySQL InnoDB buffer_pool_size); (b) monitor buffer pool hit ratio as first-class metric — alert when it drops below 95%; (c) if working set exceeds memory, consider vertical scaling (more RAM) or partitioning (shard data across nodes so each node\'s working set fits); (d) use SSD/NVMe for storage to reduce the disk-hit cost; (e) for very large working sets, consider LSM engines (which handle disk-bound workloads better due to sequential I/O). The general principle: B+ tree databases assume the working set fits in memory; when it doesn\'t, provision more memory or partition to keep working set per-node in memory.

The composite pattern across all five is that storage engine performance is dominated by structural fit to workload, not by hardware or configuration alone. A B+ tree on premium NVMe with abundant RAM will still hit its write throughput ceiling on write-heavy workloads. An LSM with unlimited compaction threads will still have p99 read latency spikes. A columnar engine with fastest CPU will still be slow for OLTP row lookups. The specific engineering discipline: analyze workload characteristics (read/write ratio, access patterns, latency requirements, data size, growth rate) at architecture time; choose the storage engine designed for that profile; monitor for workload drift over time; be willing to migrate engines when workloads change fundamentally. Modern polyglot persistence — PostgreSQL for OLTP + ClickHouse for analytics + Redis for hot state + object store + Parquet for archival — reflects this discipline in practice.

Every storage engine bug is a workload mismatch. Match engine to workload at architecture time; monitor for drift; migrate when workloads change fundamentally. This is Expert-tier data engineering.
§ 06 — Eight words for the storage engine conversation

Vocabulary,
for the structural case.

The terms that show up in every database architecture review, every "why is our database slow?" investigation, every storage engine benchmark comparison.

B+ Tree
/biː plʌs triː/
Self-balancing tree of sorted pages with data only in leaves. Root and internal nodes contain keys for navigation. Reads: logN. Writes: in-place update, potentially page splits. Foundation of PostgreSQL, MySQL InnoDB, SQL Server, Oracle. Optimal for read-heavy OLTP.
LSM Tree
/ɛl-ɛs-ɛm triː/
Log-Structured Merge tree (O\'Neil 1996): writes to in-memory MemTable + WAL, flushed to disk as immutable sorted SSTables, merged via background compaction. Optimized for write throughput. RocksDB, Cassandra, HBase, TiKV.
MemTable
/mɛm-teɪbəl/
In-memory sorted structure for LSM writes, typically skiplist or B-tree. Recent writes live here. When full, flushed to disk as immutable SSTable. Multiple MemTables can exist (one active + others being flushed).
SSTable
/ɛs-ɛs-teɪbəl/
Sorted String Table: immutable on-disk file of sorted key-value pairs. Written sequentially from MemTable flush or compaction. Contains bloom filter + index for fast lookups. Basis for LSM engines. Origin: Google Bigtable.
Compaction
/kəmˈpækʃən/
Background LSM process that merges SSTables to bound read amplification. Leveled (RocksDB): SSTables in size-based levels, one-per-level structure. Tiered (Cassandra): SSTables grouped by size, merged when N accumulate. Different write/read amplification tradeoffs.
Write Amplification
/raɪt ˌæmplɪfɪˈkeɪʃən/
Bytes written to disk per byte of user data. B+ tree: 2-5× (page updates + splits). LSM: 10-30× (compaction rewrites). Determines SSD wear and write throughput. Key metric for storage engine efficiency.
Buffer Pool
/ˈbʌfər puːl/
B+ tree in-memory cache of hot pages. PostgreSQL: shared_buffers. MySQL InnoDB: buffer_pool_size. When queries hit cached pages, memory speed; when they miss, disk I/O. Hit ratio should be >95% for good OLTP performance.
WAL
/wɔːl/
Write-Ahead Log: sequential append-only log persisted before actual storage update. Universal across all storage engines. Enables crash recovery via replay from last checkpoint. Group commit batches writes for throughput.
§ 07 — Knowledge check

Five questions.
The structural intuition.

Test the engines. Click an answer; explanation drops in instantly.

QUESTION 1 OF 5
Loading question...
Score: 0 / 5
5 / 5

Storage engines earned.

Perfect. B+ tree sorted pages, LSM MemTable + SSTables + compaction, columnar layout + vectorized execution, WAL + group commit — the specific structures determining every database\'s throughput and latency. Next up: M.54, hardware-aware system design.

§ 08 — The recap

Three ideas to
carry forward.

The composite understanding that turns "we use database X" into "we understand the specific storage engine mechanisms determining our database\'s performance profile."

i

Structure determines throughput

Every database sits on a storage engine — B+ tree, LSM, or columnar. Each has specific write amplification, read amplification, space amplification, and compaction characteristics. These determine throughput and latency BEFORE any query optimizer runs. Choosing the wrong engine for a workload produces performance issues that no query tuning can fix.

ii

Read vs write tradeoff

B+ trees pay for writes (in-place updates + page splits) to keep reads cheap (logN, ~1× read amp). LSM trees pay for reads (multiple SSTables + bloom filter mitigation) to keep writes cheap (sequential MemTable + WAL). Columnar pays for OLTP (row assembly) to keep analytics fast (100× less I/O + vectorized). The workload\'s read/write ratio and access pattern picks the winner.

iii

Polyglot persistence in practice

Real production data architectures use different engines for different workloads. PostgreSQL for OLTP + ClickHouse for analytics + Redis for hot state + Parquet on S3 for archival. Each optimized for its specific workload class. Understanding which engine fits which workload — and using them together — is the specific Expert-tier competence for modern data engineering.

↓ UP NEXT · PHASE J CONTINUES

M.54 — Hardware-
aware system design.

The next Expert module. Below the storage engine sits the hardware — CPUs, caches, NUMA topology, NIC offload, kernel bypass. Understanding hardware-aware design is what turns "our system is fast" into "we know exactly why this operation takes 100ns instead of 10μs."

Continue to Module 54 →