Add way of "cancelling" pending jobs
sometimes we detect that some packages are problematic for the infrastructure, and we add them to the reject list. However, when we still have jobs for those packages in the queue, and we don't have a way of deleting/cancelling them, because rabbitmq does not allow one to manipulate the queue arbitrarily. you can only get a job from the front of queue, and confirm it's completion. Our current solution for this is clearing out the queue (i.e. deleting all pending jobs). This is a manual and error-prone process.
Being able to cancel jobs for specific packages would need to be handled on the workers, i.e. they would need to query the reject list, and then finish the job by not running the tests, mark the test as failed, and produce some logs in the output mentioning why the job was failed.
One problem is that the worker daemon is one of the pieces that are still written in shell. we would need to either rewrite it in Ruby, or add a new command line tool that it can call to query if a given package (maybe for a specific suite/architecture pair) is in the reject list.