Deduplication numbers from real pipelines

The front page quotes 95.5% deduplication. That number is real but it comes from the friendliest possible case, so here is the fuller picture: four build trees, thirty consecutive builds each, one shared store per tree.

Setup

Each tree was built thirty times from thirty consecutive commits on its main branch. After every build the output directory was packed with haven create --store into the same store. The figure reported below is the mean bytes written per archive after the first, divided by the mean uncompressed tree size.

for sha in $(git log -30 --format=%H --reverse); do
  git checkout -q "$sha" && make -s build
  haven create --store ./store -o "arc/$sha.hv" ./dist
done

Results

tree                    size     written/build   dedup
frontend bundle       412 MiB        18.4 MiB      95.5%
go service              47 MiB        31.2 MiB      33.6%
python app + venv      308 MiB         6.1 MiB      98.0%
container rootfs       1.2 GiB        91.0 MiB      92.6%

Why the Go service is the outlier

A statically linked Go binary is one large file that changes throughout on almost every build. Symbol addresses shift, the build ID changes, inlining decisions move code around. Content-defined chunking finds few stable boundaries because very little of the byte stream is actually stable.

Lowering the target chunk size to 16 KiB pushed it to 41%, at the cost of four times the manifest size and noticeably slower verification. Not worth it for the general case, which is why 64 KiB remains the default.

Why the Python tree looks so good

Almost all of those 308 MiB is a virtualenv that changes only when dependencies change — twice in thirty builds. The application code is a few hundred kilobytes. This is the shape most people expect deduplication to have, and it flatters the tool.

The honest summary

If your artifact is mostly vendored dependencies, expect 90% or better. If it is a single compiled binary, expect a third and consider whether an archive format is the right layer to be solving that at. Haven does not help with the second case much, and pretending otherwise would waste your afternoon.

Raw logs for these runs are in bench/2026-07/ in the repository.

← All notes