Format version 2

Version 1 put the manifest at the end of the file, the way zip does. That is the natural choice when writing, because you do not know the chunk offsets until you have written the chunks. It also makes the format unreadable from a pipe, because you cannot seek backwards on a pipe.

The layout change

v1:  [header][chunk data ...][manifest][manifest offset]
v2:  [header][manifest][chunk data ...]

Writing v2 needs the manifest before the chunk data exists, which means a full scan of the tree to compute digests first, then a second pass to write chunk bodies. Two passes over the input instead of one.

Why that was acceptable

The first pass reads and hashes; the second mostly writes chunks that the store already contains and therefore skips. On the four benchmark trees the second pass touched between 2% and 66% of the data. Wall-clock time went up by 9% on the worst case and down slightly on the best, because the write path no longer had to backpatch offsets.

What it enabled

Reading from stdin, which was the point:

ssh build@host 'haven create -o - ./dist' | haven extract --into ./release

Also cheap list and inspect over HTTP range requests — the manifest sits in the first few kilobytes, so listing a 900 MB remote archive no longer downloads all of it.

What it broke

Anything that constructed archives by concatenation. One internal tool was appending chunk data from two archives and rewriting the trailing manifest, which is no longer expressible. It was rewritten to use a shared store instead, which is what it should have done originally.

Reading v1 archives still works and will keep working. Writing them does not. There was no deprecation period because the only known users were us.

← All notes