Skip to content

Support for Gentoo Linux package caching?

Gentoo "documents" something that passes for "support", here: https://wiki.gentoo.org/wiki/Local_distfiles_cache#Using_net-misc.2Fapt-cacher-ng

It's not very good, it uses PfilePatternEx: .* to work around "refusing to download non-debian packages" and then excludes https traffic so that gpg can fetch keys. I figure that it may be best to design proper support, so here goes:

The mirror system can be browsed at https://distfiles.gentoo.org/ and you will immediately see three important folders:

  • distfiles/ contains package sources e.g. upstream source tarballs. This is governed by GLEP 75: https://www.gentoo.org/glep/glep-0075.html

    The file layout.conf is unstable and mirror-specific and should be downloaded by the package manager to read the current rules for how to fetch distfiles by subdirectory. The popular default is filename-hash BLAKE2B 8 which says to take the filename being fetched, get its b2sum (printf FILENAME | b2sum, no trailing newline) and take 8/4 -> 2 hexdigest characters and expect to find the file in that directory, so e.g. 0a/MooseX-OneArgNew-0.007.tar.gz.

  • releases/ contains architecture-specific subdirectories, if we look at for example releases/amd64 we see:

    • autobuilds/ contains install ISOs and tar.xz's of writable system images, we don't need a cacher for that probably?
    • binpackages/ contains various yearly profiles and glibc/musl/llvm variant directories, but let's use binpackages/23.0/x86-64/ as the variant directory we want. It contains:
      • Packages and Packages.gz -- an index of all binary packages available in this variant directory, will be updated periodically (Gentoo official repos usually ~once per day)
      • */*/*.gpkg.tar -- these are binary packages equivalent to a .deb or a .rpm (there will never be a compression extension, because like .deb, a Gentoo binary package is an uncompressed archive containing a metadata file as well as a compressed payload image.tar.xz)

  • snapshots/ contains compressed tar archives of the repository source tree in the form {name}-YYMMDD.tar.{compress} plus .gpgsig / .md5sum files. Also {name}-latest.tar.{compress} change once a day. deltas/snapshot-YYMMDD-YYMMDD.patch.{compress} is used by some people as well.

Ignore the experimental/ directory, those are random and don't need caching. 😜

...

The portage package manager is configured with a list of mirrors, to which one of these three directory names gets appended, so the e.g. distfiles/ string will appear in the cacheable url.