Delete orphaned files from stores more effectively
If delete_expired ever crashed after deleting artifacts but before
deleting orphaned files from stores, it would forget about removing
those files and nothing would ever get round to them again. Instead of
returning a set of deleted files from _sweep and then passing that to
_delete_files_from_stores, just have _delete_files_from_stores ask
the database directly for all the orphaned files.
Part of #891 (closed).
It may be worth taking some defensive measures before deploying this, because this appears to account for 1.8 TB on poseidon, about 62% of the store (!). In theory there should be no problem with just letting it delete the backlog itself (since it ought to all be files not referenced from any artifacts), although the first delete_expired job after deploying this will take quite a while. But we might want to play it safe and see whether anything goes wrong. Perhaps we should take a snapshot of the store backup first, if we have that capability? (Alternatively, we could move all the affected files aside and see whether anything explodes, other than about half a million warnings from vacuum_storage; but since db_file and db_fileinstore rows can easily be reconstructed from local files, being prepared to put things back from a snapshot that we plan to delete after a couple of weeks would be enough.)