mxzinke / distribuerad Goto Github PK

To give the users of the program an easy-to-use integration, I would want to offer an SDK for Golang and Node.js.
Therefore I would want to implement an interface implemented by gRPC (for faster interaction).

Node.js is needed more, because it's likely to be used by more people.

Communication description will follow in this thread.

For the first viewer it seams to be just once another simpleMQ, but actually we do some different stuff with the delayed messages and jobs.

To be more flexible and reactive on client side, we need some kind of websocket communication

Adding a TTL to events (and jobs), with an easy definition and a default value of 10min.

A search, when the queue of a channel is requests, should clean up the events. Also a background job should do it on every channel (like every 5min).

To scale a both 2 mio./sec. new events, plus handling the lifecycle of each event is not easy to do.

In a first solution, we could create some read-write replicas of the service to effectively scale the service. These read-write replicas would need to copy over every change transaction to the master, so that in the end the throughput is not that much higher.

The second solution, which comes to my mind, could be a read replica, but it includes the same trouble like the first solution with the read-write replicas. Also every transaction has to be transferred to the master node. And because the service is more write intensive (add new events + acquire events), it has only a small impact.

For now I don't have any greater idea than the things above. Maybe someone could help me out with this.

Make it possible that an event is recurring in a cron-job manner.

It should be possible to define intervals and by normal cron-job definitions.

The event lifecycle does not implement currently the case that the event is eventually already expired and but still some service is working on it (had acquired a lock).

For example:

1. Creating an event with TTL of 10min
2. Immediately acquire a lock on it with a TTL of 1h (long taking process)

I don't know if this case really comes to speech in real world, because you usually make then just a higher TTL on the event, but I just want to notice it here. Maybe it has to be included in the documentation.