Emit metrics for latency between work request pending and running statuses
Each time we mark a work request running, we should emit a histogram metric with the time between it being marked pending and it being marked running, with some reasonable buckets. This time represents how long the work request had to wait for a suitable worker to be available.
The metric should include labels for the work request's task type, and probably also its task name.
The metric should include a label derived from the work request's priority. We want to be able to distinguish latency for "normal" work requests (priority >= 0) and "background" work requests (priority < 0). Because the Prometheus query language doesn't really support numeric labels (the best you can do is regex-match on their string values), and because we don't want to have metrics with excessive cardinality, this label should be something like priority=normal or priority=background rather than using the actual value. We may also want to define a minimum priority (e.g. -50) below which we don't bother to track latency at all, because the work request creator is assumed to be OK with waiting indefinitely.
We don't currently record when a work request was marked pending; the best we can do is when it was created, which may have been quite a bit earlier if it was blocked on a dependency. We'll need to add a new pending_at field and have WorkRequest.mark_pending set it; it may also be worth displaying that field somewhere in the web UI.