Python-level deadlock in debusine-server
The daphne debusine-server is occasionally "stuck" and no longer responds to anything.
According to strace all Python-level threads are calling futex, i.e. waiting for some kind of lock. The postgresql connections look idle.
Running austin reveals three kinds of threads:
- Some are worker threads of a
concurrent.futures.ThreadPoolExecutorwaiting for work items. It is pretty normal for an executor to not have any work, so these probably are harmless. -
asgirefhas aCurrentThreadExecutorwhich also seems like a work pooling mechanism that has run out of work to do. - Another class of threads is trying to join other threads from
asgiref'sThreadSensitiveContext.__aexit__. None of these stands out as an obvious cause.
Normally, the asyncio thread resides in EpollScheduler. When locked up, it belongs to the last class.