Move work request cascades to Celery?
Several changes to work request statuses cascade across work requests within a workflow. `mark_completed` and `mark_aborted` may both unblock reverse dependencies, and may both update the status of parent workflows if their children are no longer in progress. This is deadlock-prone (everything that might update multiple work requests in the same transaction must do so in the same order), and therefore we have `WorkRequestQuerySet.lock_workflows_for_update` to ensure that everything that updates anything in a workflow locks the whole workflow in one go.
However, this has its own problems. We probably can't get away from workflow orchestrators having to lock the entire workflow, but it would be good if things that are mainly just updating a single work request didn't need to take such heavyweight locks.
I think we should consider moving the "update other work requests affected by a status change" operations into a Celery task, similar to `debusine.server.celery.update_workflows`. That way, views wouldn't need the full workflow lock, and it would be easier to implement robust "try again later" logic.
I expect non-trivial test fallout from this, and it's possible it won't work for some reason I can't foresee, but if it works it would be a helpful structural simplification.
issue