P2P messaging about cloudstate HOT 5 OPEN

What approximate value would a "very large number" of topics be (that would impact the scalability of distributed pubsub)? Are there any use cases that may fit this scenario?

If this is an edge case and pubsub is the best overall option, perhaps there's a compromise to consider, such as a hard upper limit on topics that can be subscribed to within a specific boundary (I'm assuming this value is the total number of topics being subscribed to within a single service / bounded context?)

from cloudstate.

I'm not actually sure, I'm just going off what I've been told, that is that distributed pubsub doesn't scale well for the P2P case. I think it's a case of having the right tools for the job, I imagine we'll offer a generalised protocol for messaging, and then the backend can be selected from a configuration option. Perhaps we may even include some options based on third party tech.

from cloudstate.

Hi guys! Probably a gossip protocol would be more scalable eg. https://www.serf.io/docs/internals/gossip.html.

from cloudstate.

Great issue; Use case: How to support 30,000,000 peers in 300,000 topics? This is very common for group chatting / live streamings / virtual meetings.
Sharded mediator is what we are using now

from cloudstate.

@hepin1989 I'd love to hear more about your sharded mediator setup, in particular, what are the biggest problems you've encountered with it? How well does it respond to elastic scaling - eg do you get any thundering herd problems when scaling up? How many nodes/how much memory/how much cpu is required to handle various amounts of load/topics/peers?

from cloudstate.