'fanout' with parallel service management? about pitstop HOT 7 CLOSED

Hey Kevin,

I'm not Edwin, but I like to think he thought me a lot of the things I know :-)

You can definitely run the same service multiple times, using Docker Compose or Kubernetes. They would all publish their messages to the same exchange. There is something about that on the Wiki: https://github.com/edwinvw/pitstop/wiki/Run%20the%20application%20using%20using%20Docker%20Compose#scaling-the-services

All of the services you mentioned do not handle events though. Since the API's you mentioned all store their state in the database, you should be able to scale them as many times as you want.

Where it gets a little more complicated is if you would want to run some of the services that are consuming events in parallel. For example, say your invoicing service is quite slow so you want to have another instance up. Or your notification service is really important so you want several instances up. In such cases, you have to make sure that each event only gets handled once.

RabbitMQ has some support for these scenarios. You could have several instances of a service up and listening to the same queue. RabbitMQ will initially send each message to only one of the listening services. If the service doesn't acknowledge a message in time, it will get requeued and might end up at another service.

Another option is to use either a direct exchange or a topic exchange. These allow you to route messages more intelligently, for instance if you had one Invoice service printing messages and another sending invoices via e-mail.

I myself haven't used this direct form of routing, but the project I'm doing right now is having a good time using the first case - multiple listeners on one queue. We can handle hundreds of thousands of messages with a few listeners. Adding more will make the processing faster.

Finally, if you have a very large throughput of messages, you can use Sharding (see https://github.com/rabbitmq/rabbitmq-sharding) to scale your services a bit more intelligently.

One thing to be aware of: if you use the fanout exchange, and you bind multiple queues to it, each queue will get all of the messages. This is probably not a good idea when scaling a service.

from pitstop.

Hi Olaf,

 Thank you so much as always for your help and guideline!

Best,
Kevin

from pitstop.

Hi @kevinding0218. Olaf is a colleague of mine at Info Support. He also contributes to the project. Did his reply answer your questions? In that case I will close the issue.

from pitstop.

Hi Olaf, Thank you so much for answering my question and showed me more possibilities of implementing the parallel services approach! I really appreciated your knowledge share and case analysis here!
Just some quick question, if we have all different services but each would have only one instance running through RabbitMQ/Event bus, what's the benefit by using RabbitMQ/Event bus with multiply services? Was that used for decoupling the services and work as a media to connect each service?
Also, if I implement below scenario that message will be distributed through multiply instances of services with order from A --> B --> C

• 1 instance of service A
• 2 instances of service B
• 3 instances of service C
  should I consider there are two queues(A-->B and B --> C) or should I consider there are eight queues(2 queues from 1 A --> 2 B and 6 queues from 2 B --> 3 C)?

Best,
Kevin

from pitstop.

Hey Kevin,

You can indeed use a message bus just for decoupling. This will give you the advantage that you can continue providing some of your experience even when some of the services are down. Also, if some operation takes a long time (again, for instance printing an invoice), a message bus will give you the advantage of doing it asynchronously. In that case it's also being used as a queue.

When you run multiple instances of the same service that listens to events, without any specific routing, I would say in most cases you will want them to each have only one queue. If not, your messages will get handled by each of the services which in most cases will result in undesired results.

Consider for example running 3 instances of the Invoice service, with each having their own Queue, bound to the same exchange. Being a fanout exchange, it will send all incoming messages to each of the queues. So now each of the Invoice service instances will receive all messages, and will print or send the invoices to the customers. The customers receive 3 invoices and will be wondering why they're being charged three times.

So, in short: in your example there would be two queues.

from pitstop.

Hello Olafsk, thank you very much for your reply! I understand the drawback of using 'fanout' with multiply service instances running now! However, by using 'direct' or 'topic' exchange, it makes me feel I am still dealing with one instance at a time. For example:
The Invoice service can print invoice using PDF or Email, then by using 'direct' or 'topic' means I will have two queues,

• one is dealing with 'PDF Invoice'(as binding key)
• the other is dealing with 'Email Invoice'(as binding key).
• But if my message bus contains 100 'print invoice', 90 of the 100(90%) messages are requesting 'PDF Invoice' and the rest of 10(10%) are for 'Email Invoice', my 'PDF Invoice' services will become a lot busier and heavier load while my 'Email Invoice' service is hanging there for most of the time, correct?

That might not be what I would like to proceed, I would like to have two instances of invoice service that both can do 'PDF Invoice' or 'Email Invoice', then when the 100 messages come, it will distribute them into those two services continuously, so the direct or topic or header might not work here, and fanout works but produce duplicate result there.

By using EventingBasicConsumer/Task Queues seems a way to doing that but I feel it might not be good for the micro-services architecture . I also look at the rabbitmq-sharding which seems to matches my idea but I am not sure if it's possible to work with C# .Net Core since it's a plugin. Do you have any suggestion for implementing above scenario?

Thank you so much for your knowledge share again! It really helps me in the real-world case scenario a lot!

from pitstop.

Hey Kevin,

Using fanout is not necessarily a drawback. I was merely giving an example of the options you have.

With the specifications you gave, you could still just use one queue with all the messages in it. Have both instances of your service listen to that same queue, and RabbitMQ will take care of distributing the messages. That way neither of them will be idle until all the messages have been processed.

The other routing options really are more advanced, to be used if you have specific requirements (for instance, one of the services has to run on a server with a printer attached, so only that service should receive the 'Print' jobs).

I haven't personally tried the Sharding plugin but I don't think it should be a problem for .NET Core. It's a plugin you install on RabbitMQ, your clients won't need to know that the plugin is installed. That's a lot more advanced though, I wouldn't recommend starting with that unless you know in advance that you'll be handling millions of messages.

As always with software development: start easy. Make your software in such a way that you can easily extend or adapt it later on, if you find out your easy solution isn't enough.

from pitstop.