Skip to content

Potential ccache regression with new sbuild builds

Hi,

I noticed the Godot builds at https://salsa.debian.org/games-team/godot/-/pipelines are no longer passing. They are slow and depend on ccache working, so I suspect problem with cache. See example failure https://salsa.debian.org/games-team/godot/-/jobs/8272984 or https://salsa.debian.org/games-team/godot/-/jobs/8276302

When builds start they download the cache:

Restoring cache
Checking cache for build-amd64_-protected...
Downloading cache from https://storage.googleapis.com/debian-salsa-prod-runner-cache/project/29880/build-amd64_-protected  ETag="65ab0aeb6bf58223f0ab3cf7a042d8d3"
Successfully extracted cache

Cache directories are moved around:

$ if [ "${ENABLE_CCACHE}" = 1 ] ;
then
  mkdir -vp ${CCACHE_WORK_DIR}
  mv -v ${CCACHE_WORK_DIR} ${CCACHE_TMP_DIR}
  find ${CCACHE_TMP_DIR} -type d | xargs chmod g+s
fi
renamed '/builds/games-team/godot/.ccache' -> '/builds/games-team/godot/../.godot.ccache'
...
$ mkdir -vp ${WORKING_DIR} ${CCACHE_WORK_DIR}
mkdir: created directory '/builds/games-team/godot/.ccache'

..but stats indicate the cache is empty?

$ test ! "${ENABLE_CCACHE}" != "0" || su salsa-ci -c "ccache -vv -s"
Cache directory:                          /builds/games-team/.godot.ccache
Config file:                              /builds/games-team/.godot.ccache/ccache.conf
System config file:                       /etc/ccache.conf
Stats updated:                            never
Stats zeroed:                             never
Cacheable calls:                            0
  Hits:                                     0
    Direct:                                 0
    Preprocessed:                           0
  Misses:                                   0
...
Local storage:
  Cache size (GB):                        0.0 / 3.0 ( 0.00%)

Number of cache hits seems way lower than expected?

$ test ! "${ENABLE_CCACHE}" != "0" || CCACHE_DIR=${CCACHE_TMP_DIR} ccache -svv 2> /dev/null || true
Cache directory:                          /builds/games-team/.godot.ccache
Config file:                              /builds/games-team/.godot.ccache/ccache.conf
System config file:                       /etc/ccache.conf
Stats updated:                            Sat Sep 13 04:45:27 2025
Stats zeroed:                             never
Cacheable calls:                          153 / 178 (85.96%)
  Hits:                                     2 / 153 ( 1.31%)
    Direct:                                 1 /   2 (50.00%)
    Preprocessed:                           1 /   2 (50.00%)
  Misses:                                 151 / 153 (98.69%)
...
Local storage:
  Cache size (GB):                        0.0 / 3.0 ( 1.49%)
  Files:                                  303

Upload:

$ test ! "${ENABLE_CCACHE}" != "0" || mv -v ${CCACHE_TMP_DIR} ${CCACHE_WORK_DIR}
renamed '/builds/games-team/godot/../.godot.ccache' -> '/builds/games-team/godot/.ccache/.godot.ccache'
$ cd ${WORKING_DIR}
$ du -sh
748K	.
$ if [ "$(du -s | cut -f1)" -gt ${SALSA_CI_MAX_ARTIFACTS_SIZE} ]
Saving cache for failed job
00:04
Creating cache build-amd64_-protected...
.ccache: found 2051 matching artifact files and directories 
Uploading cache.zip to https://storage.googleapis.com/debian-salsa-prod-runner-cache/project/29880/build-amd64_-protected 
Created cache
Edited by Otto Kekäläinen
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information