Improve work request scheduling with tag based approach
Right now all the scheduling relies on a `can_work_on()` function that require instantiating each task to find the next work request for a given worker. This scales poorly with a very large number of work requests. We can improve this process significantly by having each work request document a set of tags that the desired worker must have.
The tags exposed by a worker can be created/extracted from the static and dynamic worker_data.
The tags requested by a work request can be a combination of user-submitted tags and tags generated by the task after analysis of the task_data. Sometimes the architecture requirement is quite explicit in the task_data (via "host_architecture" fields for example) but sometimes it's much less explicit, for example if the user provides a custom "environment_id" then that environment is built for a specific architecture and we must run the work request on that architecture, even if the task in itself is fairly architecture agnostic (like a lintian analysis).
Then the scheduling logic can exclude work requests that need a tag that the current worker does not have.
Among the expected tags, we should have:
- the architectures that can run (natively) on the worker
- some small/medium/big classification based on the number of CPU cores/RAM/disk (to be documented/defined)
- maybe the kind of worker ? i.e. internal / external (server-side tasks vs worker-side tasks)
Tags (or worker-tag relationships and work-request-tag relationships) likely have some associated meta-data/permissions too:
- whether a given user can set a given tag (might rely on a workspace membership?)
- whether a worker exclusively runs tasks with a given tag, or if it can run work requests without that tag
- whether the tag can be provided by the user or whether it can only be set indirectly by debusine through analysis of history and/or task parameters
Some initial design notes are available [here](https://freexian-team.pages.debian.net/debusine/reference/devel-blueprints/build-instructions.html#control-workers-assigned-for-specific-packages-tasks) and [here](https://freexian-team.pages.debian.net/debusine/reference/devel-blueprints/build-instructions.html#changes-related-to-the-worker-requirements-feature). Other parts remain to be specified, in particular at the database model level and at the scheduler level.
TODO (not exaustive):smiley:
* [x] Provide/require worker tags but do not use them for scheduling (!2688)
* [x] Provide/require task tags (!2716)
* [x] Use tags for scheduling (once the workers are updated to provide scheduling tags), and remove `can_run_on` (!2727)
* [x] Allow to configure tags using task configuration (!2822)
For this task and its children, please track time under Work Package D for STA
issue