page_cache_bench

Microbenchmarks comparing 4 KB random-read latency across four Linux I/O paths on a large (64 GB) file:

1. pread, page-cache warm — data already resident in the page cache.
2. pread, cold — page cache dropped before each measurement, forcing a real block-device read.
3. mmap + MADV_RANDOM — reads become page faults; MADV_RANDOM disables readahead so the kernel doesn't prefetch neighboring pages.
4. O_DIRECT — bypasses the page cache entirely; buffer, offset, and size are 4 KB-aligned as required.

Built on Google Benchmark, fetched automatically via CMake FetchContent.

Architecture

flowchart TD
    B[Benchmark Driver]
    B --> P1[pwrite64: 64 GB rand file]
    B --> P2[sync + drop_caches]
    B --> P3[1M random 4 KB offsets]

    subgraph Paths
        W[pred, warm<br/>page cache hit]
        C[pred, cold<br/>block device read]
        M[mmap + MADV_RANDOM<br/>page fault]
        D[O_DIRECT<br/>bypass cache]
    end

    P3 --> W
    P3 --> C
    P3 --> M
    P3 --> D

    W --> KC[Kernel: page cache]
    C --> BD[Kernel: block layer]
    M --> PF[Kernel: page fault handler]
    D --> BD2[Kernel: direct I/O]

    KC --> R[Google Benchmark<br/>SetBytesProcessed]
    BD --> R
    PF --> R
    BD2 --> R

Loading

Why

The four paths hit different layers of the kernel I/O stack (cache lookup, page fault handling, direct block I/O), so their latency profiles diverge a lot under random 4 KB access. Running them side by side on the same file and machine makes those differences directly comparable instead of theoretical.

Requirements

• Linux (uses O_DIRECT, /proc/sys/vm/drop_caches, madvise)
• CMake >= 3.16, a C++17 compiler
• sudo access, to drop page caches between cold-path runs
• Enough free disk space for the test file (64 GB by default)

Setup

Gather machine info relevant to interpreting results:

free -h                          # RAM size
lsblk -d -o NAME,MODEL,SIZE      # SSD model
uname -r                         # kernel version
lscpu | grep "Model name"        # CPU
findmnt -no FSTYPE /tmp          # filesystem of the test location

Create the test file (64 GB of random data, so it can't be effectively compressed by any layer of the stack):

mkdir -p bench
dd if=/dev/urandom of=bench/bench.dat bs=1M count=65536 status=progress
ls -lh bench/bench.dat   # should show 64G

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build

Run

chmod +x run_bench.sh
./run_bench.sh

run_bench.sh drives build/io_bench through all four scenarios in order, dropping the page cache (sync; echo 3 > /proc/sys/vm/drop_caches, via sudo) before each cold measurement. Set BENCH_FILE to point at a different test file if needed (defaults to bench/bench.dat).

Design Decisions

Why 4 KB pages. The x86-64 page size is 4 KB. Using a different size would measure a different phenomenon (compound page faults, partial-page IO, or multiple-page reads). 4 KB isolates the per-page cost.

Why a 64 GB file on a machine with less RAM. A file larger than available RAM guarantees that cold-path reads actually go to the block device — the page cache cannot hold the entire file. Without this, cold reads might find their page still resident from a prior benchmark pass, producing artificially low (warm) numbers for the cold case.

Why random data instead of zeros. /dev/urandom produces incompressible data. Filesystems and block devices sometimes compress or deduplicate all-zero pages, which would make block-device reads faster than they would be for real data.

Why MADV_RANDOM for mmap. Without it, the kernel performs readahead on page faults, fetching neighboring pages speculatively. For truly random access, readahead wastes I/O bandwidth and pollutes the page cache with pages that will never be touched. MADV_RANDOM disables readahead, making the mmap benchmark a fair comparison against the other paths.

Why only one byte per page for mmap. A page fault is triggered by the first access to any byte on a page. Reading the full 4 KB after the fault would add a memcpy cost that doesn't exist for pread/O_DIRECT (which read into user buffers directly). Touching one byte isolates the page-fault latency — the thing being measured.

Why shared offsets across all four paths. The 1M random offsets are generated once with seed 42 and reused. This eliminates offset-generation as a confounding variable: any difference between paths is due to the I/O mechanism, not because one path hit a different set of pages.

Why no mmap + MAP_POPULATE variant. MAP_POPULATE pre-faults all pages at mmap time, combining the fault cost into a single upfront operation. It's a useful optimization in practice, but it measures setup cost, not per-access cost — it doesn't fit the per-iteration benchmark model used here.

Benchmark Internals

• 1,000,000 uniformly random 4 KB-aligned offsets are generated once (seed 42) and reused across all four benchmarks, so each path is measured against the same access pattern.
• pread/O_DIRECT read a full 4 KB page per iteration; mmap only touches one byte per page, since the page fault — not a memcpy — is what's being measured.
• Throughput is reported via SetBytesProcessed (4 KB × iterations).