Skip to content

Implement a publish/subscribe mechanism for events and changes to collections

We identified the need to build a publish/subscribe mechanism to be notified of various events and in particular changes to collections. We need the ability to filter events that we want to monitor and we want to be able to consume them in a batched way.

Some notes from the Toulouse meeting:

  • Allow some events to come from outside debusine, for webhook-based automation.
  • Would external events need to be authenticated? Could be optional.
  • The events would likely trigger workflows. Those are configured via workflow templates, so would run with the permissions of the user who owns the workflow template (or an explicit delegation).
  • Potentially useful for caching reverse-dependency information.
  • Workflows may need to be able to get diffs between states in order to efficiently process batches of events.

Other things to consider:

  • we might want to be able to create a Wait task that waits for a specific event, we'd like minimal delays here, so maybe have some on-the-fly processing of events
  • we want some way to trigger a workflow with "as soon as you get the event but not more than once every 3h", so we need some external scheduler that takes care of implementing this rate-limit mechanism
  • when a workflow process events in batched way, it should be able to somehow record the events that it has consumed so that it doesn't get triggered needlessly (optimization: at worst it gets executed and ends up empty with nothing to do)
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information