Support pub/sub for task status about funnel HOT 8 CLOSED

So funnel would publish task updates to a messaging system, and clients would subscribe to that (more scalable) system?

from funnel.

Yes, that's a sensible implementation. Maybe there's a way at task submission time to also provide a URL to POST status updates to. Then Funnel doesn't have to know about the pub/sub system, and it can be a separate thing. I don't now if pub/sub systems typically have a REST API though.

from funnel.

What is the delivery guarantee for the publishing? At most once?

from funnel.

Good question. Whatever guarantee the selected pub/sub implementation provides?

from funnel.

At most once delivery from Funnel would (obviously) be the least complex, and for that reason my preference. However, while the pub/sub server maybe be 99.999% available, in that 0.001% of the time when the message is lost, a subscriber would not recover (e.g. it might never see that a task finished).

An alternative could be publishing "current state" messages instead of "state change" messages. I.e. Funnel publishes an event every N seconds that says "task X is running".

from funnel.

Sure, letting Funnel be a pub/sub provider is the easiest first pass.

A pub/sub server would be more reliable. It would be up to callers to check for the rare cases where messages get lost. That seems fine.

I'd recommend against sending heartbeats out by pub/sub. The main motivation for pub/sub rather than polling is to cut down overhead with too-frequent state updates.

from funnel.

We have defined an events package here:
https://github.com/ohsu-comp-bio/funnel/tree/master/events

I have stackdriver logging event writer on a branch here: https://github.com/buchanae/funnel/blob/smcrna/gcp/stackdriver.go
A Google pub/sub writer would be easy to add.

This issue also mentions using pub/sub (or app engine task queues, rabbitmq, celery, etc) as the internal work queue (as opposed to events from a task that is already running). The smcrna branch linked above is using pub/sub.

Do we need an API endpoint which clients can use to subscribe to task events? Would this become part of the TES spec? Or are clients expected to connect directly to the underlying infrastructure (e.g. Google pub/sub)?

from funnel.

Basic support added for our event stream in #465

The event messages Funnel currently emits are defined here:
https://github.com/ohsu-comp-bio/funnel/blob/master/events/events.proto

from funnel.