Skip to content

Use zstd compression by default

Heitor Alves de Siqueira requested to merge halves/kdump-tools:zstd-default into debian/latest

Hi @dannf!

I've been running tests with zstd compression, and the results are quite promising. When using it with makedumpfile, I've been able to get better compression rates than the default zlib, and it's also significantly faster. As a bonus, zstd seems to require less memory overhead, freeing up a valuable resource in the crash context (even if ever so slightly). Below are some excerpts:

Debian Sid VM, with 2Gb RAM

root@sid:~# makedumpfile -d31 -F -c /proc/kcore > dump.c
root@sid:~# makedumpfile -d31 -F -z /proc/kcore > dump.z
root@sid:~# ls -l dump.*
-rw-r--r-- 1 root root 79639290 May  8 17:09 dump.c
-rw-r--r-- 1 root root 71906209 May  8 17:08 dump.z

Using heaptrack, peak memory consumption turned out to be slightly smaller for zstd as well:

  • zlib: peak heap memory consumption: 677.16K
  • zstd: peak heap memory consumption: 482.79K

Finally, using hyperfine to have a performance comparison between both:

root@sid:~# hyperfine --warmup 5 -L compress c,z 'makedumpfile -d31 -F -{compress} /proc/kcore > /dev/null' --export-json sid-vm.json
Benchmark 1: makedumpfile -d31 -F -c /proc/kcore > /dev/null
  Time (mean ± σ):      1.584 s ±  0.014 s    [User: 1.485 s, System: 0.097 s]
  Range (min … max):    1.567 s …  1.611 s    10 runs

Benchmark 2: makedumpfile -d31 -F -z /proc/kcore > /dev/null
  Time (mean ± σ):     534.5 ms ±   6.7 ms    [User: 436.6 ms, System: 96.7 ms]
  Range (min … max):   527.3 ms … 545.9 ms    10 runs

Summary
  makedumpfile -d31 -F -z /proc/kcore > /dev/null ran
    2.96 ± 0.05 times faster than makedumpfile -d31 -F -c /proc/kcore > /dev/null

Ubuntu Noble bare-metal, with 96Gb RAM

root@elite:~# ls -l dump.*
-rw-r--r-- 1 root root 304322897 May  8 21:20 dump.c
-rw-r--r-- 1 root root 202317735 May  8 21:19 dump.z

Heaptrack data:

  • zlib: peak heap memory consumption: 4.89M
  • zstd: peak heap memory consumption: 4.69M

Benchmark with hyperfine:

root@elite:~# hyperfine --warmup 5 -L compress c,z 'makedumpfile -d31 -F -{compress} /proc/kcore > /dev/null' --export-json bare-metal.json
Benchmark 1: makedumpfile -d31 -F -c /proc/kcore > /dev/null
  Time (mean ± σ):     23.616 s ±  0.171 s    [User: 21.447 s, System: 2.167 s]
  Range (min … max):   23.480 s … 23.989 s    10 runs

Benchmark 2: makedumpfile -d31 -F -z /proc/kcore > /dev/null
  Time (mean ± σ):     11.931 s ±  0.073 s    [User: 9.755 s, System: 2.175 s]
  Range (min … max):   11.832 s … 12.051 s    10 runs

Summary
  makedumpfile -d31 -F -z /proc/kcore > /dev/null ran
    1.98 ± 0.02 times faster than makedumpfile -d31 -F -c /proc/kcore > /dev/null

I've uploaded the hyperfine JSON results as well as the heaptrack files to [0], in case you want to take a look.

[0] https://people.canonical.com/~halves/kdump_zstd/

Overall, the benefits seem significant enough to justify changing our defaults. I've also added zstd support to kdump-config (e.g. when not using makedumpfile), for completeness sake. Feel free to cherry-pick either one, or squash them as needed.

Let me know what you think, and if you'd like to see any more tests! Thanks!

Merge request reports