Comments (3)
@mafrosis celery_once
is designed to be a extension on top of celery that prevent's multiple tasks executing at the same time.
If you have requirements that mean you shouldn't be running or queuing repeated tasks, that where celery_once becomes useful.
In your example above, having a concurrency of 1 doesn't prevent you from queuing multiple tasks with the same args.
For example, if you ran the following task...
@task()
def slow_convert():
sleep(10)
like so...
for _ in range(100):
slow_convert.delay()
It would be queued up 100 times, and preformed 100 times, 1 by 1.
If you declared it like so...
@celery.task(base=QueueOnce)
def slow_convert():
sleep(10)
An AlreadyQueued
exception will be raised after the first loop.
>>> for _ in range(100):
>>> slow_convert.delay()
Traceback (most recent call last):
..
AlreadyQueued()
It would of been queued once, and preformed queued. Until it is run, no other tasks like that can be queued.
Depending on the application, that can be very useful! :)
Hope that answers the question!
from celery-once.
Thanks! I can see that AlreadyQueued
could be useful in some applications.
from celery-once.
Thinking about it a bit - it should be possible to use your AMQP broker only to provide the locking mechanism, removing the need for Redis. Querying the broker but not acknowledge the message will leave it available for other consumers (although I've never done that with celery!)
from celery-once.
Related Issues (20)
- Controlling lock behaviour on task failure HOT 6
- File Backend: lock files are not cleaned automatically HOT 3
- Only delay and apply_async check against the lock. HOT 1
- 1 Running, 1 Queued (per set of keys)
- No such file or directory: '/tmp/celery_once/ using file based backend HOT 7
- Recheck that dup task is actually running
- Ability to chain tasks HOT 1
- get_key function return different key for same function.
- Deprecation warning due to invalid escape sequences
- Missing apply_async ConnectionError handling HOT 1
- File backend hashlib.md5 not supported in FIPS mode
- self.retry doesn't retry the task on unittests
- Set the redis lock expire time according to the ETA countdown if "default_timeout" is not given
- Feature request: Queue last
- Once didn't unlock task in redis if settled time_limit in celery HOT 2
- Adding minimum interval param HOT 1
- Still maintained? HOT 2
- Passwords have special symbols
- Not working for scheduled tasks HOT 3
- Unsupported protocol sentinel
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from celery-once.