downloads existing packages and discards them

The System: Debian buster (many packages from buster-backports) with apt-cacher-ng_3.2.1-1_amd64.deb, which has been serving a number of other machines with packages for years. Until yesterday.

The issue: storage error [503 Inconsistent file state], last errno: File exists When trying to upgrade packages on a client, all packages gave an HTML 503 error, even though all those packages must have existed in the cache. A second attempt downloaded all packages from the internet again.

Possible cause(s): No idea. I moved the entire system onto a new disk last week, but... read errors? Transfer issues? File attributes incorrectly transferred (with rsync -HSavz)?

I checked file attributes on the cache dir and its contents.

#: getfacl var/cache/apt-cacher-ng/
# file: var/cache/apt-cacher-ng/
# owner: apt-cacher-ng
# group: apt-cacher-ng
# flags: -s-
user::rwx
group::rwx
other::r-x

I apt-get purged apt-cacher-ng, verified that configs, logfiles and cache were gone, reinstalled it, fed it with a backup of the packages in /var/cache/apt-cacher-ng/_import and imported about 20GB of 37GB of packages. I successfully upgraded a local chroot (even without internet connection). Then, with another client I got 503s again.

Example: Package is available:

#:/var/cache/apt-cacher-ng/debrep/pool/main/t/tcpdump# ls -l *.deb
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 400212 Dez 13 21:53 tcpdump_4.9.3-1~deb10u2_amd64.deb
#:/var/cache/apt-cacher-ng/debrep/pool/main/t/tcpdump# sha256sum *.deb
c8f053e45563ac479c841b15bdef705530351101a1d20cfd8c657ea7b8c609a7  tcpdump_4.9.3-1~deb10u2_amd64.deb

Package is in package list, checksum is OK, so it should be known to acng:

#:/var/cache/apt-cacher-ng/debrep/dists/buster/main/binary-amd64# ls -l *.xz
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 7907472 Dez 13 21:50 Packages.xz
#:/var/cache/apt-cacher-ng/debrep/dists/buster/main/binary-amd64# xzgrep c8f053e45563ac479c841b15bdef705530351101a1d20cfd8c657ea7b8c609a7 Packages.xz
SHA256: c8f053e45563ac479c841b15bdef705530351101a1d20cfd8c657ea7b8c609a7

When I try to aptitude install it on the client, I get an HTML 503 error. I watched the traffic, and acng tries to download the file although it already exists, then finds it exists, interrupts the almost completed transfer of the new file, deletes the local file from the cache and reports error 503, so 1+1=0.

logfiles:

apt-cacher-ng.log:
1607934363|M|Download of debrep/pool/main/t/tcpdump/tcpdump_4.9.3-1~deb10u2_amd64.deb started
1607934378|E|426|192.168.2.10|debrep/pool/main/t/tcpdump/tcpdump_4.9.3-1~deb10u2_amd64.deb [HTTP error, code: 503]

apt-cacher-ng.err:
Mon Dec 14 09:26:03 2020|debrep/pool/main/t/tcpdump/tcpdump_4.9.3-1~deb10u2_amd64.deb storage error [503 Inconsistent file state], last errno: File exists

Now I put the same file into /var/cache/apt-cacher-ng/_import and I can import it. After import, the file exists in /var/cache/apt-cacher-ng/debrep/pool/main/t/tcpdump again, like above. And again the client gets a 503 error. But this time, when I copy the file back into _import, it no longer gets imported. Packages.xz was refreshed, the package is still listed there - but acng ignores it in _import, although it has deleted the file from the cache.

I am lost.