Making output byte-identical

The goal was simple to state: packing the same tree twice, on two machines, should produce the same archive bytes. Getting there took longer than expected, and four things had to change.

Directory iteration order

The obvious one. readdir returns entries in filesystem order, which differs between ext4 and XFS and changes as directories are modified. Entries are now sorted by their byte representation, not by locale-aware collation — sorting Z before a looks wrong to humans but is the same everywhere.

Timestamps

Three fields were being stored: mtime, atime and ctime. Only mtime is meaningful to preserve, atime changes just from reading a file, and ctime cannot be set on extraction anyway. Both were dropped from the format.

mtime stayed but is now truncated to whole seconds. Filesystems disagree about sub-second precision, and a tree copied with cp -a between two of them would otherwise produce different archives.

Sparse files

A hole and a run of zero bytes are indistinguishable after reading, but SEEK_HOLE reports them differently depending on filesystem and on whether the file has been fsynced. Detection was removed entirely; holes are read as zeros and chunked normally. Runs of zeros dedupe to a single chunk regardless, so the storage cost is negligible and the behaviour is now identical everywhere.

Uid, gid and umask

Numeric owners were being recorded, which meant the same tree checked out by two different users produced different archives. Ownership is no longer stored. Extraction writes files owned by whoever runs it. Permission bits are kept, but masked to 0777 — setuid and setgid are deliberately discarded.

Verifying it holds

CI packs the same fixture tree on Ubuntu with ext4, Alpine with overlayfs, and macOS with APFS, then compares digests:

$ haven create -o /tmp/a.hv ./testdata/fixture
$ b3sum /tmp/a.hv
41d9e0c7f2b8a35e9c4d1f6072ba8e13c5a90d4f8e2b1c7a6d3f0e9b8c5a2d17  /tmp/a.hv

That digest is committed to the repository. It has changed twice since, both times deliberately, both times with a format version bump.

← All notes