Draft: Add zstd support for Packages files
Hi!
What do you think about support for zstd compression of Packages files?
The performance increase for compression with zstd instead of xz is significant. Speed comparison for doing 5 compressions of the trixie Packages file of ~54MB on my laptop:
jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time xz > Packages.xz; done; du -h Packages.xz
54M Packages
13.35user 0.06system 0:06.19elapsed 216%CPU (0avgtext+0avgdata 336528maxresident)k
0inputs+19240outputs (0major+17574minor)pagefaults 0swaps
13.20user 0.05system 0:06.12elapsed 216%CPU (0avgtext+0avgdata 336660maxresident)k
0inputs+19240outputs (0major+17572minor)pagefaults 0swaps
13.22user 0.07system 0:06.21elapsed 213%CPU (0avgtext+0avgdata 335796maxresident)k
0inputs+19240outputs (0major+17379minor)pagefaults 0swaps
13.27user 0.09system 0:06.16elapsed 216%CPU (0avgtext+0avgdata 334456maxresident)k
0inputs+19240outputs (0major+17509minor)pagefaults 0swaps
13.38user 0.06system 0:06.25elapsed 214%CPU (0avgtext+0avgdata 335836maxresident)k
0inputs+19240outputs (0major+17811minor)pagefaults 0swaps
real 0m31,027s
user 1m6,461s
sys 0m0,444s
9,4M Packages.xz
jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time zstd > Packages.zst; done; du -h Packages.zst
54M Packages
0.17user 0.03system 0:00.06elapsed 304%CPU (0avgtext+0avgdata 77344maxresident)k
0inputs+24704outputs (0major+2565minor)pagefaults 0swaps
0.13user 0.02system 0:00.04elapsed 340%CPU (0avgtext+0avgdata 75008maxresident)k
0inputs+24704outputs (0major+1920minor)pagefaults 0swaps
0.17user 0.02system 0:00.05elapsed 332%CPU (0avgtext+0avgdata 76868maxresident)k
0inputs+24704outputs (0major+1919minor)pagefaults 0swaps
0.19user 0.02system 0:00.06elapsed 324%CPU (0avgtext+0avgdata 79672maxresident)k
0inputs+24704outputs (0major+2520minor)pagefaults 0swaps
0.14user 0.02system 0:00.05elapsed 322%CPU (0avgtext+0avgdata 75044maxresident)k
0inputs+24704outputs (0major+1911minor)pagefaults 0swaps
real 0m0,323s
user 0m0,826s
sys 0m0,221s
13M Packages.zst
jas@frallan:~$ That is going from over a minute to below a second!
What matters for users is transfer time and decompression time though.
Transfer time depends on network speed, and sometimes the size difference of 13M vs 9.4M can be significant.
But let's look at decompression speed:
jas@frallan:~$ du -h Packages.zst; time for i in $(seq 1 10); do cat Packages.zst | /usr/bin/time unzstd > Packages; done; du -h Packages
13M Packages.zst
0.04user 0.00system 0:00.03elapsed 148%CPU (0avgtext+0avgdata 6700maxresident)k
0inputs+110384outputs (0major+1235minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6260maxresident)k
0inputs+110384outputs (0major+1204minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6188maxresident)k
0inputs+110384outputs (0major+1203minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6312maxresident)k
0inputs+110384outputs (0major+1173minor)pagefaults 0swaps
0.04user 0.00system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6372maxresident)k
0inputs+110384outputs (0major+1237minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 142%CPU (0avgtext+0avgdata 6352maxresident)k
0inputs+110384outputs (0major+1207minor)pagefaults 0swaps
0.02user 0.02system 0:00.03elapsed 150%CPU (0avgtext+0avgdata 6692maxresident)k
0inputs+110384outputs (0major+759minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6496maxresident)k
0inputs+110384outputs (0major+1207minor)pagefaults 0swaps
0.03user 0.02system 0:00.03elapsed 157%CPU (0avgtext+0avgdata 6664maxresident)k
0inputs+110384outputs (0major+1270minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 154%CPU (0avgtext+0avgdata 6368maxresident)k
0inputs+110384outputs (0major+1206minor)pagefaults 0swaps
real 0m0,484s
user 0m0,341s
sys 0m0,312s
54M Packages
jas@frallan:~$ du -h Packages.xz; time for i in $(seq 1 10); do cat Packages.xz | /usr/bin/time unxz > Packages; done; du -h Packages
9,4M Packages.xz
0.25user 0.03system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 87876maxresident)k
0inputs+110384outputs (0major+3654minor)pagefaults 0swaps
0.25user 0.03system 0:00.12elapsed 226%CPU (0avgtext+0avgdata 87468maxresident)k
0inputs+110384outputs (0major+3653minor)pagefaults 0swaps
0.24user 0.04system 0:00.12elapsed 223%CPU (0avgtext+0avgdata 87396maxresident)k
0inputs+110384outputs (0major+3652minor)pagefaults 0swaps
0.27user 0.01system 0:00.13elapsed 222%CPU (0avgtext+0avgdata 88012maxresident)k
0inputs+110384outputs (0major+4205minor)pagefaults 0swaps
0.28user 0.03system 0:00.14elapsed 217%CPU (0avgtext+0avgdata 88840maxresident)k
0inputs+110384outputs (0major+3655minor)pagefaults 0swaps
0.27user 0.01system 0:00.13elapsed 225%CPU (0avgtext+0avgdata 84768maxresident)k
0inputs+110384outputs (0major+3658minor)pagefaults 0swaps
0.26user 0.03system 0:00.13elapsed 235%CPU (0avgtext+0avgdata 85596maxresident)k
0inputs+110384outputs (0major+4257minor)pagefaults 0swaps
0.26user 0.03system 0:00.12elapsed 230%CPU (0avgtext+0avgdata 88008maxresident)k
0inputs+110384outputs (0major+4257minor)pagefaults 0swaps
0.26user 0.02system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 88828maxresident)k
0inputs+110384outputs (0major+4293minor)pagefaults 0swaps
0.27user 0.03system 0:00.13elapsed 227%CPU (0avgtext+0avgdata 87400maxresident)k
0inputs+110384outputs (0major+4253minor)pagefaults 0swaps
real 0m1,565s
user 0m2,677s
sys 0m0,461s
54M Packages
jas@frallan:~$ That is three times as fast.
Regardless of performance, having support for this in debootstrap allows for further real-world testing.
Cheers, Simon