left-over from lintian after being OOM-killed
We just had an issues on mentors where there was no space left on the unique / partition.
The culprit are left-over temporary files from lintian after being killed by the OOM-killer:
du -shc lintian-pool-*
3.5G lintian-pool-3ezF4kbnkG
4.5G lintian-pool-3oKAGXJumI
4.1G lintian-pool-6_A75ZXPpn
4.4G lintian-pool-J0oTGCF4IS
4.2G lintian-pool-j7LttprDvT
4.2G lintian-pool-pvkEEPCMOy
4.2G lintian-pool-qqPLUyBptx
3.8G lintian-pool-ZFZJyrJ5a6
33G total
I propose the following changes:
- Cleanup of
/tmp/lintian-pool*after a lintian run. Since the importer cannot run in parallel, it should work properly - Having /tmp in a separate partition. Unfortunately, no lvm niether free space left on mentors. Should we ping the hoster to ask for an additional 20GB for
/tmp? - Having some kind of monitoring + alerting to avoid being spammed by postfix failing to allocate space in the spool. We could:
- Have a standalone monitoring system on mentors. The upside is it's accessible by all mentors admins, the downside is alerting might not trigger if we have resources exhaustion on mentors
- Ask DSA if we can add mentors to icinga. Is it possible to do that for non-official project? Is it possible to notify only mentors' admin?
- Build or use our own monitoring system. @mattia has a munin and I have an Icinga running. We might be able to use those.