Giter Club home page Giter Club logo

Comments (12)

selwin avatar selwin commented on May 28, 2024

@ouhouhsami you can pass an connection instance to send_async_mail to ensure that it uses the same connection. Something like this should work without requiring you to change your production code.

@job
def send_async_mail(connection=None):
    send_mail('foo', 'bar', '[email protected]', ['[email protected]'], fail_silently=False, connection=connection)


def send_custom_mail():
     send_async_mail.delay(connection=get_connection())

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

Well, thanks for your answer to my first point for asynchrone mail testing.

For the signals test, I'm a bit confused. Here is the python example, using the mock_signal_receiver (https://github.com/dcramer/mock-django/blob/master/mock_django/signals.py) :

# in jobs.py for example

@job
def send_async_signal():
    print my_custom_signal.receivers  # I got the one created below by mock_signal_receiver
    print my_custom_signal.receivers[0][1]().call_count  # I have 0, which is right
    my_custom_signal.send(sender=Foo)
    print my_custom_signal.receivers[0][1]().call_count  # I have 1, which is right too, and 'should' be the case at the end of my testcase (see below). 

def send_custom_signal():
     send_async_signal.delay()

# in tests.py

from django_rq import get_worker
from mock_django import mock_signal_receiver
from jobs import send_custom_signal

class AsyncSendSignalTestCase(unittest.TestCase):
    def test_async_signal(self):
        with mock_signal_receiver(my_custom_signal) as receiver:
            print my_custom_signal.receivers  # I have exactly the same receiver as the first print in send_sync_signal()
            send_custom_signal()
            print receiver.call_count  # I get 0, and it's right, because the signal is sent in an asynchronous job that hasn't been yet treated.
            get_worker().work(burst=True)
            print receiver.call_count  # I get 0, and not 1 as I would have expected.
            print my_custom_signal.receivers[0][1]().call_count  # I also get 0 here (in send_async_signal, the value was 1)

The thing that is done by mock_signal_receiver is just to connect the signal to a receiver.
The problem is that the receiver isn't reached when doing asynchronous jobs, even if, as I wrote in the comments in my example above, my_custom_signal has the "right" reference to the mock_signal_receiver.

Regards,

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

I tried to get rid of mock library, but don't really succeed. Here is the code for the test:

# in jobs.py for example

@job
def send_async_signal():
    my_custom_signal.send(sender=Foo)

def send_custom_signal():
     send_async_signal.delay()

# in tests.py

from django_rq import get_worker
from django.dispatch import receiver
from jobs import send_custom_signal


def count_calls(fn):
    def _counting(*args, **kwargs):
        _counting.calls += 1
        return fn(*args, **kwargs)
    _counting.calls = 0
    return _counting

@receiver(my_custom_signal)
@count_calls
def my_custom_signal_callback(sender, **kwargs):
    print 'well, callback reached'

class AsyncSendSignalTestCase(unittest.TestCase):
    def test_async_signal(self):
            send_custom_signal()
            print my_custom_signal_callback.calls  # 0, which is right
            get_worker().work(burst=True)
            print my_custom_signal_callback.calls  # 0, which is wrong, it should return 1 

from django-rq.

selwin avatar selwin commented on May 28, 2024

@ouhouhsami I think it's better to queue the long running function instead of sending an asynchronous signal because it adds another layer of indirection with no obvious benefit (in my opinion).

So instead of doing:

def long_running_function(*args, **kwargs):
    # do stuff

my_custom_signal.connect(long_running_function)

@job
def send_async_signal():
    my_custom_signal.send(sender=Foo)

def send_custom_signal():
     send_async_signal.delay()

class AsyncSendSignalTestCase(unittest.TestCase):
    def test_async_signal(self):
            send_custom_signal()
            print my_custom_signal_callback.calls  # 0, which is right
            get_worker().work(burst=True)
            print my_custom_signal_callback.calls  # 0, which is wrong, it should return 1 

I would just do:

@job
def long_running_function(*args, **kwargs):
    # do stuff


class MyTestCase(unittest.TestCase):
    def test_async_job(self):
            long_running_function.delay()
            get_worker().work(burst=True)
            # Check that work gets done

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

Thanks for your answer @selwin . The fact is that the things aren't so simple in my real use case.

You are totally right, refactoring my example, but in my case I need this design: the possibility to send signal in an asynchronous function.

I investigated a lot and didn't find any solution to face that problem. I inspected the function ids, and they are the same, I also used weak=False for the signal, it doesn't changed anything. What appears it that the property 'calls' (using my decorator) or 'call_count' (using mock library) are updated 'inside the job function', but when I try to get this attribute inside my test after a get_worker().work(burst=True) , it is reset, even if the functions are the same 'instance' in memory.

Regards,

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

Hello,

After all, I reopen this issue, with a test code sample that should achieve my goal, but doesn't pass for thz test_async_signal function (it uses https://github.com/dcramer/mock-django to mock signal receive)

from django_rq import job
from django.dispatch import Signal

from mock_django import mock_signal_receiver

# set up a Signal
bar = Signal()


# the async function
@job
def async_foo():
    bar.send(sender='baz')


def foo():
    async_foo.delay()


# the sync function for demonstration purpose
def sync_foo():
    bar.send(sender='baz')


class DjangoRQSignalTestCase(unittest.TestCase):

    def test_async_signal(self):
        with mock_signal_receiver(bar) as receiver:
            foo()
            get_worker().work(burst=True)
            self.assertEquals(receiver.call_count, 1)

    def test_sync_signal(self):
        with mock_signal_receiver(bar) as receiver:
            sync_foo()
            self.assertEquals(receiver.call_count, 1)

I'd like your help on this subject.

Regards,

from django-rq.

selwin avatar selwin commented on May 28, 2024

Hi there,

The code above works, but just not during testing, right? It seems like the mocked received didn't receive the signal, did you try opening an issue at mock-django's repository?

I've never used mock although I've heard good things about it, I haven't delved that deep into signals either so I'm not sure I can be of much help. I'll try to have a look at it this weekend.

In the mean time, I'll reopen this issue to give this a better visibility.

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

Thanks for reopening this issue. As you say: the code works, but not for during testing.
I wrote the test_sync_signal function to show that it works without the async process.
So, here is a version without mock-django, but using evil global :

from django.utils import unittest
from django_rq import job 
from django.dispatch import Signal

# set up a Signal
bar = Signal()

globvar = 0

# the async function
@job
def async_foo():
    global globvar
    globvar = 1

def foo():
    async_foo.delay()

# the sync function for demonstration purpose
def sync_foo():
    global globvar
    globvar = 1

class DjangoRQSignalTestCase(unittest.TestCase):

    def test_async_signal(self):
        foo()
        get_worker().work(burst=True)
        self.assertEquals(globvar, 1)  # this should pass, but globvar is 0

    def test_sync_signal(self):
        sync_foo()
        self.assertEquals(globvar, 1)  # this pass

I think this is more or less the same problem as above.

from django-rq.

selwin avatar selwin commented on May 28, 2024

I played around with it a little bit, no luck. However, when I fix #15 , you'll be able to do this, which works:

global globvar
globvar = 0

def async_foo(async=True):
    q = get_queue(async=async)
    q.enqueue(sync_foo)

# the sync function for demonstration purpose
def sync_foo():
    global globvar
    globvar = 1

class DjangoRQSignalTestCase(TestCase):

    def test_async_signal(self):
        async_foo(async=False)
        self.assertEquals(globvar, 1) 

    def test_sync_signal(self):
        sync_foo()
        self.assertEquals(globvar, 1)  # this pass

from django-rq.

ouhouhsami avatar ouhouhsami commented on May 28, 2024

Hi there,

So, I see you have changed the code base adding async param now: a57ee60

The thing I finally have to 'hack' is to allow job to be run synchronously.

My decorators.py is now like this, setting async param to False in a job decorator, run the job synchronously:

from rq.decorators import job as _rq_job

from .queues import get_queue


def job(func_or_queue, connection=None, *args, **kwargs):
    """
    The same as RQ's job decorator, but it works automatically works out
    the ``connection`` argument from RQ_QUEUES.

    And also, it allows simplified ``@job`` syntax to put job into
    default queue.
    """
    async = kwargs.pop('async', False)

    if callable(func_or_queue):
        func = func_or_queue
        queue = 'default'
    else:
        func = None
        queue = func_or_queue

    if isinstance(queue, basestring):
        try:
            queue = get_queue(queue, async=async)
            if connection is None:
                connection = queue.connection
        except KeyError:
            pass

    decorator = _rq_job(queue, connection=connection, *args, **kwargs)
    if func:
        return decorator(func)
    return decorator

BTW, I no longer need get_worker().work(burst=True) in my tests, I have just added a SYNC value (True/False) in my django settings, and set the async param to settings.SYNC. The tests pass.
In your point of view, do you see any bad method to do it like this ? More precisely, is async=True in get_queue more or less the same as burst=True mode ?

Regards,

from django-rq.

selwin avatar selwin commented on May 28, 2024

I originally added the async argument to Queue for debugging purposes here rq/rq#112

Testing with async turned on will still properly execute and test all logic in your code.

It differs slightly in that it completely bypasses the Worker class - which means that you may not find bugs in RQ's Worker and django-rq's get_worker during testing, if there's any.

from django-rq.

selwin avatar selwin commented on May 28, 2024

I'm closing this issue. Use Queue(async=False) to run tests.

from django-rq.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.