Establishing direct connections through a central broker about zguide HOT 6 OPEN

I've used the following before (on another protocol) and have implemented and using this on zmq.

Central broker uses XSUB/XPUB and listens for and republishes 'HELLO' messages.
It may also maintain a list of service endpoints and responds to requests
for the service list.

Client connects to central broker subscribing to HELLO messages.
Client sends HELLO message contains list (service name, listen socket) it provides.
Client ignores HELLO messages sent from self.
When client receives HELLO message from other clients, update internal
registry of service names, with socket endpoints.

Client responds to HELLO with a OLLEH (or some other message) with it's
service list, or the broker can respond to the joining client with the current
service list.

When a 'client' wants to send to service name, initiate connection endpoint with
service by service name.
If expecting async reply, create unique 'reply' service name on local 'client's'
listener endpoint. Include REPLY (service name, listen socket) in the protocol
message to service.

On zmq there's the 'slow initial connection problem' so you may have
establish the pub/sub connection between services during service registration
instead of at message initiation.

On the Central broker, when sockets close, remove the service from the list
and broadcast GOODBYE with service name(s) to all clients.

from zguide.

Dafka uses the approach you described. Have a look the central tower (https://github.com/zeromq/dafka/blob/master/src/dafka_tower.c) and the beacons used in the peers of the network (https://github.com/zeromq/dafka/blob/master/src/dafka_beacon.c)

from zguide.

Thanks for these fast and promising answers.
Dafka seems to fit very well.
If i may ask @sappo , as you are a maintainer of Dafka as well:
Has it been tested on 20.04 LTS? I have some dependencies problems with leveldb upon configuring even though i installed it (apt install libleveldb-dev)

from zguide.

@SegfaultCreator I do my development on 20.04, so it should work!

from zguide.

@sappo Great to hear that you have it running under 20.04!
Unfortunately, i still get linking errors with leveldb after trying out clang-11 with lld-11 and gcc with gnu-ld linker. Do you have any special installation of leveldb or did you also use the liblevel-db-dev=1.22-3ubuntu2 provided by ubuntu repository?

from zguide.

No I don't have any special installation. This is what apt says I have installed:

libleveldb-dev/focal,now 1.22-3ubuntu2 amd64 [installed]
libleveldb1d/focal,now 1.22-3ubuntu2 amd64 [installed,automatic]

from zguide.