rq / django-rq Goto Github PK

The current enqueue helper doesn't support the 'async=False' option as described in the RQ docs under the Bypassing Workers section. Is there any intention to support this in the short term - happy to do this and submit pull request if not?

Seems a bit pointless to have tests and travis if you just ignore it.

When i visited django-rq, exception occured, and the configuration of url do not take any extra params for rq, why exception occured?

Did i have any problem in some places?

I'm using the "high", "medium", "low" queue pattern and my instrumentation is indicating that my request loops are spending a lot of time just connecting to Redis. Since I'm using Redis for caching as well via django-redis, it seems like I should be able to use cache.client to retrieve the cache connection that get created in the middleware. In RQ, it looks like I could just use the Connection context manager. It would be nice to be able to do this within the standard django-rq paradigm, but I think this might require some modification to do cleanly.

But I thought I'd propose the idea of making an alternative configuration that cleanly ties into django-redis system, for people who want to do things that way. In my mind, you would simply provide the key for the cache entry to use and the names of the queues to create, and the system would take it from there. If I have the time, I might fork, but I thought I'd ask your thoughts first.

RQ_QUEUES = {
    'default': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0,
    },
    'high': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0,
    },
    'low': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0,
    }
}

Forgot coma before 'low'

And not rqworkers
python manage.py rqworker high default low

Although Travis shows that builds are passing, tests were never run https://travis-ci.org/ui/django-rq/jobs/6159928

In a fresh pip install of the latest version:

$ pip freeze | egrep '(redis)|(rq)'
django-redis==3.4.0
django-rq==0.6.0
redis==2.8.0
rq==0.3.12

The following error is inevitable:

File ".../site-packages/django_rq/queues.py", line 63, in get_redis_connection
    return cache.client.client
AttributeError: 'DefaultClient' object has no attribute 'client'

The error is caused by https://github.com/ui/django-rq/blob/58b309458a434f075546e6b05e3911ee5178e1c2/django_rq/queues.py#L63

It is a similar error but more important than #36 because in this case it is not a third party monkey patch breaking things, but simply that the private field of django-redis that django-rq relies upon is not where it is expected.

Hi,

I get this error:

ValueError: Queues must have the same redis connection."default" and "high" have different connections

when DEFAULT_TIMEOUT is different for the two queues I use on the same worker.
I did not think DEFAULT_TIMEOUT was part of the "connection" since it is the job timeout and is not related to the connection, correct?

By just changing the DEFAULT_TIME of the high queue to "120" instead of "300", I got the error. If I set it back to "300", it works fine.

CACHES = {
    'queues': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': os.environ['DJANGO_CACHE_QUEUES_LOCATION'],
        'TIMEOUT': None, # Never expires by default
        'OPTIONS': {
            'CLIENT_CLASS': 'redis_cache.client.DefaultClient',
            'PARSER_CLASS': 'redis.connection.HiredisParser',
            'CONNECTION_POOL_KWARGS': {'max_connections': 100},
            'IGNORE_EXCEPTIONS': not DEBUG, # Get the errors only in DEBUG
            'PICKLE_VERSION': -1  # Use the latest protocol version
        }
    }
}

RQ_QUEUES = {
    'high': {
        'USE_REDIS_CACHE': 'queues',
        'DEFAULT_TIMEOUT': 120, # 2 mins  # =>  MUST BE 300 to avoid the error
    },
    'default': {
        'USE_REDIS_CACHE': 'queues',
        'DEFAULT_TIMEOUT': 300, # 5 mins
    },
}

Thanks

I'm getting an error running django-rq on Heroku:

redis.exceptions.ResponseError: max number of clients reached

The error is perfectly reasonably, I'm using the free redis cloud service which only allows 10 connections and running a script to generate content which calls django-rq lots of times and thereby exceeds the connection limit. But what's best fix?

I could probably force the queue to be synchronous for this task. But that only fixes this script, what happens in proper deployment if the system exceeds the max number of clients limit?

Basically is there an elegant way of using rq's connection pool in django-rq? Would it be easiest to use django-redis then specify max_connections?

Hi,

I noticed that the Empty Queue button on the admin interface wasn't working and tried to figure out what happened. The problem is that EVAL/EVALSHA commands were implemented in Redis 2.6.0, and I'm using the versions 2.4 in development (which comes with Debian Wheezy) and 2.2 in production (which comes with Ubuntu Precise). Running the django_rq tests on my machine, four of them (test_commit, test_error, test_success and test_clear_queue) fails with the same message: ResponseError: unknown command 'EVALSHA'.

Digging a little more, I've found out that the problem is actually not caused by django-rq, but by rq itself, which uses a Lua script to clear the queue in its empty() method. Then, when redis-py tries to execute it, you can guess what happens. Also, it's worth mentioning that this does not affect the bulk delete action that was added on 3dd89ad/0.7.0.

So, how can we fix this? I guess that there are other people using wheezy/precise on its systems as well. Some options are:

• Detect which version of Redis is being used by the queue, then hide/show the button if it's < 2.6.0 or >= 2.6.0.
• Change the default action of the Empty Queue button, so it would bulk-delete the entire queue instead - something which would be like selecting all jobs then use the new bulk delete action mentioned, although this is probably less efficient.
• Implement a mix of the previous two options, letting the clear_queue view decide on which one is best, based on the detected Redis version.

Regards,
Tiago.

With rq, I am able to pass in connection to create a queue. I am using Heroku and the way I create a queue is by using redis.from_url. The way I use this without django-rq is from the [Heroku tutorial on rq]:(https://devcenter.heroku.com/articles/python-rq)

import os

from rq import Queue
import redis

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
redis_connection = redis.from_url(redis_url)
q = Queue('email', connection=redis_connection)

Currently I am planning on parsing the url like this:

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
url = urlparse.urlparse(redis_url)
RQ_QUEUES = {
    'default': {
        'HOST': url.hostname,
        'PORT': url.port,
        'DB': 0,
        'PASSWORD': url.password,
    },
}

Source: redis/redis-py#246

Hello!

It's possible to check that the worker is running?

I'm try to add job in the queue when worker is not running, and I'm get no errors or messages. I understand that the jobs are added to the redis, and when a worker will run, these jobs will be executed. But that happens if a worker has crash or it has not started?

I'm looking for something better (more highlevel and more python) than:

ps ax | grep rqworker

Is it possible to pass a timeout value to an enqueued job? I believe the default is 180 seconds -- which is short for some long-running jobs.

Thanks for the great tool!

In rq a 'queue' is an arbitrary named pipe and doesn't imply anything about the underlying connection. In django-rq it implies a connection. So the best practice of 3 queues suggested by rq ('low', 'medium', 'high') doesn't really work with django-rq unless you define 3 connections.

I would propose that you either change the syntax of RQ_QUEUES to include some form of glob matching, or change it to be RQ_CONNECTIONS, which is what it really contains, and separate out the connection from the queue name completely.

Option 1: Automatic queue->connection routing by glob.

RQ_QUEUES = {
    '*': {
        'HOST': REDIS_HOST,
        'PORT': REDIS_PORT,
        'DB': REDIS_DB['jobs'],
    },
    'test-*': {
        ...
    }
}
...
queue = django_rq.get_queue('test-low')
queue.enqueue(func, foo, bar=baz)

Option 2: Differentiation completely.

import django_rq
queue = django_rq.get_queue('low', connection='test')
queue.enqueue(func, foo, bar=baz)

Issue: Why is redis filling up if the results of jobs are discarded immediately?

I'm using django-rq to queue up jobs to create PDFs asynchronously and then save the result to my database. Since its saved, I don't need to access the object a later date and so I don't need to keep store the result in redis after its been processed.

To keep the result from staying in redis I've set the result_ttl to 0:

parameter_dict = {          
    "order": serializer.object,
    "photo": base64_image,
    "result_ttl": 0
}
django_rq.enqueue(procces_template, **parameter_dict)

The problem is although the rq worker says the job expires immediately:

15:33:35 Job OK, result = John Doe's nail order to 568 Broadway
15:33:35 Result discarded immediately.
15:33:35
15:33:35 *** Listening on high, default, low...

Redis still fills up and throws:

ResponseError: command not allowed when used memory > 'maxmemory'

Is there another parameter that I need to set in django-rq to keep redis from filling up if the job result is already not stored?

The version dependency on rq in setup.py is >=0.3.0, however the latest version of django-rq uses the rq.utils.ColorizingStreamHandler in its logging configuration, and this was introduced in rq 0.3.2, and so those running 0.3.0, 0.3.1 will fail:

ValueError: Unable to configure handler 'rq_console': Cannot resolve 'rq.utils.ColorizingStreamHandler': No module named ColorizingStreamHandler

I propose upping the dependency to 0.3.2.

Obviously, you forgot to push changes to github, because right now 0.7.0 is on the pypi, but I can't see these changes on the GitHub.

Django 1.5.5. produces this error with django-rq

AttributeError: 'CacheStatTracker' object has no attribute '_client'

When I downgrade to 1.5.4 , all is well. Been having problems with 1.5.5 and many other packages. I think some 1.6 code snuck into the 1.5.5 security update.

Hi there - is there any chance of pushing a new update to PyPI?

Using job decorator, django-rq tries to connect to redis even though ASYNC is set to False in RQ_QUEUES. When redis is available it executes synchronously but after connecting to redis first.

Hello

In documentation I found a mistake, You require use:
worker = django_rq.get_worker() # Returns a worker for "default" queue
worker.run()

But object worker doesn't has a method run()

How I can start new worker in background from django code?

Thank You

The django-rq's admin interface shows timestamps in UTC, instead of the local timezone. This is inconsistent with how the django admin interface works.

Hi, I am interested to see what it looks like, but don't have enough time right now to install all the stack ATM. Is there any example, demo or screenshot that I've missed?

Hello,

I would like to use the job decorator 'around' a post_save (or any other) signal handler, have you got an example of this working.

Here is some code I "imagine", but don't know if I can achieve it with django-rq

from django.db.models.signals import post_save
from django_rq import job

@job
def after_save_instance_handler(sender, **kwargs):
    print sender
after_save_instance_handler.delay(*?!*)

post_save.connect(after_save_instance_handler, sender=MyModel)

There is an interesting article here, trying the same thing with celery http://dougalmatthews.com/2011/10/10/making-django%27s-signals-asynchronous-with-celery/

Regards,

I'm using django-rq on Heroku with the Heroku Postgres DB setup. When my RQ worker jobs finish, I always see this in the logs:

LOG:  could not receive data from client: Connection reset by peer
LOG:  unexpected EOF on client connection

Any idea is there's a way to close the database connection when a job ends?
For example, I probably need to put this somewhere:

from django.db import connection
connection.close()

Perhaps it should just be in the worker function itself... but that seems like a hack... thoughts?

Should return 404 page instead

RQ supports creating custom Worker classes (here, bottom). These are launched like this:

$ rqworker -w 'path.to.GeventWorker'

Any chance we could add this flag to django-rq?

Hello,

there is an issue with python 2.6: the dictConfig used in management/commands/rqworker.py is only available on python2.7.

There was a similar issue with rq: see rq/rq#172

So using the django version

from django.utils.log import dictConfig

should fix it.

Is it a normal behavior to have several failed queues displayed in the admin?

failed  0   
failed  0   
failed  0

Screenshot: http://s28.postimg.org/lpq26ifzh/2014_10_22_103715.jpg

Thanks

Hello!
I'm has some mamagement command in my django project, and I need to add them to queue. Thats work fine:

django_rq.enqueue(call_command, 'load_new_price', file, queue=True)

But I also need to pass django request object to this function, like this:

django_rq.enqueue(call_command, 'load_new_price', file, queue=True,  request=requst)

And I has error: "expected string or Unicode object, NoneType found".
In file: rq/job.py in save:
line: 303 obj['data'] = dumps(self.job_tuple)

Hello,

I try to test an app which has some asynchronous tasks using django-rq, and it's not always as easy as it could/should be.

For the moment I have got two use cases - Mail tests and Signal tests, that work with synchronous job, but doesn't work with asynchronous tasks.

Mail test:

# in jobs.py for example

@job
def send_async_mail():
    print get_connection()  # return an instance of locmem.EmailBackend
    send_mail('foo', 'bar', '[email protected]', ['[email protected]'], fail_silently=False)


def send_custom_mail():
     send_async_mail.delay()

# in tests.py

from django_rq import get_worker
from jobs import send_custom_mail

class AsyncSendMailTestCase(unittest.TestCase):
    def test_async_mail(self):
        send_custom_mail()
        get_worker().work(burst=True)
        print get_connection() # return an OTHER instance of locmem.EmailBackend
        print len(mail.outbox)  # return 0, should return 1

As you see above, the "problem" is that the instance of get_connection() (https://docs.djangoproject.com/en/dev/topics/email/?from=olddocs#django.core.mail.get_connection) is not the same.

Also, I'm facing (I think) the same problem using mock_signal_receiver https://github.com/dcramer/mock-django/blob/master/mock_django/signals.py in asynchronous jobs. A kind of a reference to the receiver is losted with asynchronous jobs.
I'm quite sure of it because, when I remove the asynchronous part of my code, it works.

Any ideas on how to deal with that kind of problem ?

Regards,

There is management command to run rqworker with Django environment set (python manage.py rqworker ...) but no analogue for running rq-scheduler.
It would be great to have one instead of running DJANGO_SETTINGS_MODULE=... bin/rqscheduler.

I have a very simple task:

@job
def log_info():
    logger.info('Job running')

Queuing this task 500 times nearly takes 2 minutes to process with a single worker.

for i in range(500): tasks.log_info.delay()

I guess its because of the issue described in the Performance notes here.
Is django-rq is setting up the django environment for every single job?

If I start a periodic job with a repetition count of 5, and from within each invocation of that job I submit a single non-periodic job to another worker, the parent job's repetition count never makes it all the way to zero - the repetition count is effectively ignored, making the job last forever. Printing the repeat count within enqueue_job, I can see that it decrements a few times, and then stops going down before reaching zero.

However, if I start a periodic job with a repetition count of 5, and I don't submit another job from within it, the repetition count of the periodic job decrements as one would expect, and the job's periodic evaluation is eventually finished.

I have a small test program that shows the problem, but I'll need to get permission to share it - so let me know if you think it might be helpful to see.

Hello, I think that mixing connections data and queue names in configuration is not a good idea. At least it won't give you to create queues dynamically (CELERY_CREATE_MISSING_QUEUES analogue).

More than that there is some kind of duplication, because queues list is stored in the hash of each worker.

As I can see this was done only because of only one reason - to show queues list in the interface.

It would be more consistent to pass the combination of connection and queue names as the arguments of worker manage command and extract data about workers and queues directly from the redis.

What do you think about such pattern?

It would be great to be able to config the «async» mode from the settings so I could write:

if DEBUG:
    RQ_QUEUES['default']['async'] = True

This doesn't seems to be supported?
The workers are instantiated upon command without any possibility to push_exc_handler(my_handler)

ValueError: Queues must have the same redis connection."default" and "high" have different connections.

{'DEFAULT_TIMEOUT': 300, 'HOST': 'localhost', 'DB': 0, 'PORT': 6379}
!=
{'DEFAULT_TIMEOUT': 60, 'HOST': 'localhost', 'DB': 0, 'PORT': 6379}

It's just the timeout which is different. The error message should say: "A worker cannot handle queues with different timeouts" (if thats true).

Hi,

I'm trying to use get_current_job() function from rq , but I can't figure how to do this.

import rq
rq.get_current_job()  # return None

I'm using django_rq, and think it's a problem of connection ?

When looking a code in rq/jobs.py, we see it uses LocalStack, and it seems that LocalStack is not 'aware' of our redis connection, as mentioned in django_rq rqworker command.

Cheers,

Sorry if I missed something, but this difference in behavior is not explained in the documentation as far as I can see.

In this example, test is run one time, which is expected.

def test():
    print 'asdf'
    time.sleep(2)

django_rq.enqueue(test)

But in this example, test is being run over and over (seemingly endlessly), each time getting a new job id.

@job
def test():
    print 'asdf'
    time.sleep(2)
test.delay()

test()

Is there an explanation for this? Am I using it wrong? Or is this a real problem?

Using rq-scheduler I can define a periodic task but I am not sure where I should put this code. I have used celery before and it puts such tasks in the settings module, but this is not possible with django-rq because of how get_scheduler is defined. Is there a recommended location for infinite periodic tasks that need to be launched with the application?

Hi,

I'm running some classic tests with manage.py test myapp. RQ is running with python manage.py rqworker default.

I have some jobs that write some stuff to DB. While my tests write in the test database Django creates, the jobs are working fine but are writing to the "normal" DB, no the test one... How can I make RQ use the test settings while in test mode?

Thanks in advance, great product BTW.

Hi,

Very nice app. Good job. Thanks

I have a question regarding sentry.
Is it mandatory to use the following logging configuration to use Sentry?
https://github.com/ui/django-rq#configuring-logging

Especially about defining a sentry handler?
I never had to define any sentry handlers for any django apps and I received the errors via sentry.
What is different with django-rq?

Thanks

Support django's database transaction, similar to https://github.com/chrisdoble/django-celery-transactions

Sorry for asking question in an issue tracker. But I am wondering how to gracefully reload django rqworker without losing any running jobs.

Thanks in advance!

When I run the following code:

    now = datetime.datetime.now()
    future_3_min_delta = datetime.timedelta(minutes=3)
    print('now is {}'.format(str(now)))
    future_3_min_absolute = now + future_3_min_delta
    print('future_3_min_absolute is {}'.format(str(future_3_min_absolute)))
    result3 = scheduler.enqueue_at(future_3_min_absolute, test_mod.func, 3)

The job is not delayed for 3 minutes - instead, it runs right away. In fact, even if I change it to 1 hour in the future, it still runs right away.

It is terrible override default admin template.

A least make it optional depending on setting variable and mention it in readme.