Comments (5)
If you're using django-redis for your caching, I just added an option that allows a level of indirection, using your connection definition for Redis in your cache as the connection info for one or more queues. See the docs for this.
But on a wider not, I think you do have a point, and it may be beyond the scope of this package. It seems to me that it's really not ideal to define the connection in the cache settings either. It would make a lot more sense to have a package that defines and handles the connections, and all of the services that use Redis would ride on that--caching, queuing, etc.
I'm thinking https://github.com/niwibe/django-redis is probably the place to put this into action. I think it's the most generalist of the Django Redis packages, and it would be really cool to see everything come together around it. That's just my opinion.
from django-rq.
@meteozond the main purpose of django-rq
is to provide convenience. For me, being able to centrally define the queues and refer to them by name, is a big win because this means you don't need to worry about managing the low level Redis connection in your application code.
I agree with @acjay 's assessment that it would be useful if all Redis backed services could agree on a single syntax, or use a single package whose sole purpose is to manage Redis connections. If that were to happen we can definitely support that:
REDIS_BACKENDS = {
'server1': {
...
},
'server2': {
...
}
}
As to creating queues dynamically, you can do so without using django-rq
at all:
from redis import Redis
from rq import Queue
for name in dynamic_queue_names:
queue = Queue(name, connection=Redis())
As for your second point of extracting queue and worker data directly from Redis in addition of the queues defined in settings
, I'll gladly accept a patch for that.
from django-rq.
@selwin I'm absolutely agree with you that today there is no sigle backend statement syntax. More than that, disputing with my colleague I've found out that key-value storages are not caches and shouldn't be connected as a cache backends. Today it is autonomous technology like data bases, caches, templates, it should not be mixed with others. I think we need some kind of proxy library to provide unified interface for key-value features (including pub and sub and so on) for differed backends.
At my previous job we had some internal solution but it didn't go open. I believe some day django will provide key-value storage backends support out of the box.
@acjay I've got your point, I just wanted to say to things:
- Working with queues should be more flexible.
- Backend settings should be separated from the que-management, because it is some kind of different things.
from django-rq.
I've got some time to put part of my idea into code.
cybergrom@8b706d5
cybergrom@474604b
cybergrom@cb5c207
- Now to create new queues is enough to enqueue new task.
- Changed
RQ_QUEUES
toRQ_CONNECTIONS
(with same syntax). - Removed
get_queue_by_index
requirement. - Enhanced arguments of
rqworker
andrqscheduler
for specifying connection - Added argument
connection_name
for some methods - Old
rqworker
andrqscheduler
arguments could be used as same as earlier
./manage.py rqworker - will start worker for default
queue in the default
connection
./manage.py rqworker low - will start worker for low
queue in the default
connection
./manage.py rqworker redis.low - will start worker for low
queue in the redis
connection
Any suggestions?
Could it be a merge?
from django-rq.
I somehow missed your previoius reply, sorry! Haven't checked out your commit to comment on implementation, but to the extent that my vote counts, I'm all for anything that increases DRYness and flexibility. My change was meant less as the "optimal" setup, and more so a backwardly compatible extension. That's my 2 cents :)
from django-rq.
Related Issues (20)
- Regarding Logging
- django.core.exceptions.ImproperlyConfigured: Requested setting RQ_SHOW_ADMIN_LINK, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. HOT 2
- Worker pool implementation
- job_timeout is not killing subprocess
- Separate RQScheduler into its own fake model
- RQ Job Terminated Unexpectedly HOT 2
- Allow Specifying Default Serializer for Django-RQ Queue HOT 2
- How to run django-rq worker via Webhook/API call HOT 1
- Django dumpdata will fail because of unmanaged model HOT 1
- KeyError accessing stats page HOT 2
- Error in job_detail.html at line 226 HOT 2
- Sentinel support broken since 2.9.0 HOT 5
- Add support for floating point intervals in rescheduler command HOT 1
- ValueError: Invalid attribute name/AttributeError: module has no attribute
- keys of command in MULTI calls must be in same slot HOT 1
- TypeError in job_detail.html with Python 12 HOT 3
- Database Errors When Running rqworker-pool HOT 14
- RQ WorkerPool is not loading models at all
- `get_scheduler` should support a custom connection
- Tag 2.10.2 on Git HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-rq.