Sunsetting gzip substitutes availability

Your location:

Sunsetting gzip substitutes availability

Maxim Cournoyer — February 9, 2022

Starting next month (2022/03/01), the build farm known as ci.guix.gnu.org will no longer offer gzip-compressed binary substitutes. The Guix daemon has known to use lzip for substitutes since 2019; unless you are running a very outdated daemon, you have no need to worry about this change.

This idea was first discussed about a year ago, when it was found that gzip-compressed substitutes accounted for about only 1% of the downloaded substitutes. Since then, the daemon has gained support for zstd on top of gzip and lzip, and the build farm has happily generated compressed substitutes for all of these compression schemes.

While migrating the storage array of the Berlin-hosted build farm to faster electronic storage, it was discovered that the cached substitutes use about 16 TiB of storage space, which means they occupy most of the new 22 TiB array, of which gzip substitutes account for 6.5 TiB:

$ du -sh /var/cache/guix/publish/{gzip,lzip,zstd}
6.5T    /var/cache/guix/publish/gzip
4.8T    /var/cache/guix/publish/lzip
4.0T    /var/cache/guix/publish/zstd

Letting go of gzip substitutes will allow such storage space to be re-purposed to more pressing needs (such as a growing /gnu/store). In the future, we may consider offering only zstd-compressed substitutes, as zstd offers a competitive compression ratio compared to lzip (as can be glimpsed above) while being faster to both compress and decompress.