left-over from lintian after being OOM-killed

We just had an issues on mentors where there was no space left on the unique / partition.

The culprit are left-over temporary files from lintian after being killed by the OOM-killer:

du -shc lintian-pool-*
3.5G    lintian-pool-3ezF4kbnkG
4.5G    lintian-pool-3oKAGXJumI
4.1G    lintian-pool-6_A75ZXPpn
4.4G    lintian-pool-J0oTGCF4IS
4.2G    lintian-pool-j7LttprDvT
4.2G    lintian-pool-pvkEEPCMOy
4.2G    lintian-pool-qqPLUyBptx
3.8G    lintian-pool-ZFZJyrJ5a6
33G     total

I propose the following changes:

  • Cleanup of /tmp/lintian-pool* after a lintian run. Since the importer cannot run in parallel, it should work properly
  • Having /tmp in a separate partition. Unfortunately, no lvm niether free space left on mentors. Should we ping the hoster to ask for an additional 20GB for /tmp ?
  • Having some kind of monitoring + alerting to avoid being spammed by postfix failing to allocate space in the spool. We could:
    • Have a standalone monitoring system on mentors. The upside is it's accessible by all mentors admins, the downside is alerting might not trigger if we have resources exhaustion on mentors
    • Ask DSA if we can add mentors to icinga. Is it possible to do that for non-official project? Is it possible to notify only mentors' admin?
    • Build or use our own monitoring system. @mattia has a munin and I have an Icinga running. We might be able to use those.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information