Comments (8)
It absolutely doesn't have to be 1 channel for 1 consumer using 1 thread. Most clients (Java, .NET, several Ruby ones, Erlang and, I believe, Go) maintain a pool of threads (or similar) which is either fixed or fixed by default.
Dynamically creating a thread per delivery is pretty wasteful unless you only have cases when processing a delivery takes a long time.
from amqp.
Thanks Michael. I'm pretty much a noob with this stuff in general.
Am I right in thinking I would need to create and manage a channel pool manually? Or does this library have some way of helping me with that?
To my original question, would you recommend a particular way of throttling my consumption of messages? Right now I'm thinking about using flow
. That is, every time a SqlError is raised in a callback thread, I'd handle it by calling flow chan False
, then sleep for say 2 seconds before restarting flow.
from amqp.
You can use basic.qos
with manual acks to control how many messages are pushed to your consumer at a time. How to process the deliveries concurrently or in parallel is up to you.
There is no channel pooling in this library (or almost any RabbitMQ client) and because error handling is scoped to channels, it is not a great idea.
You can use just one channel if you only consume. It can take you a long way.
from amqp.
It seems like I'm confused about terminology and the underlying concepts.
When you say "You can use just one channel if you only consume", I think
that's clear enough to me.
I also understand now that there's no "1 thread == 1 channel" parity.
So when you say "Dynamically creating a thread per delivery is pretty
wasteful", I'm not sure what you mean. Isn't thread spawning handled by
Network.AMQP's internals -- it just spawns one thread per message it
consumes?
Thanks for bearing with me.
On Mon, Oct 14, 2013 at 2:46 PM, Michael Klishin
[email protected]:
You can use basic.qos with manual ackshttp://www.rabbitmq.com/tutorials/tutorial-two-python.htmlto control how many messages are pushed to your consumer at a time. How to
process the deliveries concurrently or in parallel is up to you.There is no channel pooling in this library (or almost any RabbitMQ
client) and because error handling is scoped to channels, it is not a great
idea.You can use just one channel if you only consume. It can take you a long
way.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26278801
.
from amqp.
I need to take a look but if it does, it's simply unnecessary.
You can use one channel if your code is very unlikely to produce a channel exception (e.g. reference a queue that does not exist or try to redeclare it with different attributes), so you don't really care about fine grained error handling.
from amqp.
Perhaps this protocol concepts overview would be useful.
from amqp.
We do not spawn a thread for every message. All messages that a channel receives are processed one after another on the same thread.
In your situation I would probably use one channel that just pushes all incoming messages into some kind of a queue data-structure (for example Control.Concurrent.Chan). You can then create several worker threads that read from your queue and process the items.
from amqp.
This has been extremely helpful -- thanks very much, both of you.
I've just confirmed -- had to see it with my own eyes -- can I can use
qos
with prefetchCount
of 1 (plus making sure to require ack, and not
to send it until my job is complete) to throttle my process to 1 job at a
time. I'll play with ways to throttle it up -- I see how I might be able
to use a different prefetchCount
value, and I also see how and why I
might fetch all messages, queue them in my own data structure, and pop them
off that structure whenever there's an available database connection.
On Mon, Oct 14, 2013 at 6:36 PM, Holger Reinhardt
[email protected]:
We do not spawn a thread for every message. All messages that a channel
receives are processed one after another on the same thread.In your situation I would probably use one channel that just pushes all
incoming messages into some kind of a queue data-structure (for example
Control.Concurrent.Chan). You can then create several worker threads that
read from your queue and process the items.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26294763
.
from amqp.
Related Issues (20)
- Delivery timeout does not close channel HOT 2
- Using a green thread per message consume HOT 2
- `closeConnection` can swallow IO exceptions and block / hang indefinitely HOT 5
- Blocked Connection Handler Not Triggered HOT 2
- Example showing exception handling and reconnects HOT 2
- x-stream-offset is 64bit unsigned HOT 2
- (removed)
- Support Consumer Cancellation Notification HOT 10
- Add From/ToFieldValue classes HOT 10
- Why is publishing on the same channel thread while consuming a problem? HOT 4
- openConnection/closeConnection not thread safe HOT 1
- No Channel.Close_ok sent when channel is closed from server HOT 2
- Deadlock on calling cancelConsumer. HOT 11
- "decodeUtf8: Invalid UTF-8 stream" when accessing non-UTF8 header data HOT 5
- consumeMessages hangs when committing transaction HOT 5
- Design: Exception handling (swallowing?) in subscriber callback. HOT 7
- Add withConnection function HOT 4
- Dangerous uses of forkIO HOT 19
- closeChannel results in an unhandled exception message HOT 1
- Does aknowledging the message kills handler thread? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amqp.