Add design for retrying dependency-wait tasks
Part of #462 (closed).
This includes a simple design for periodic workflows that I think satisfies all the needs for them we've had so far and should be quite easy to implement.
I've tried to make sure that the design will allow us to slot in something like a dose-builddebcheck strategy in future. The split between task and workflow is similar to what we've been doing in several other recent design discussions where we instruct the task very specifically on what to do and put most of the smartness into the workflow orchestrator, although since we don't have such a thing as lookups that return work requests I had to just use a list of work request IDs. I believe we could quite easily replace the current update-debian-environments workflow arrangements on debusine.debian.net and debusine.freexian.com using this approach. The APT mirroring arrangements would take a little longer since those currently use a server task directly rather than a workflow, but otherwise it should work for that too.
The question of how to model the dependency-wait state itself is still open here, but I've done my best to lay out how I think the design would work in both cases. I find myself not really a neutral party here: the more I tried to work out how using a "Wait" task type for this would work, the more disadvantages I saw, and I haven't thought of enough advantages to make it seem worth all the effort and complexity. The view I'm coming to is that "Wait" tasks as separate workflow steps make a lot of sense where the workflow can plan in advance that it will have to wait for something, but they seem less suitable where the waiting is conditional and only discovered when a task fails.
By contrast, the approach of using a new unblock strategy has no practical problems I can think of: we'd just need to ensure we have a suitable index so that the retry_dep_wait workflow orchestrator can find such work requests quickly. And it's non-invasive, so if in future we decide that this was the wrong approach then migrating to something else wouldn't be much more difficult than doing it from scratch.