Without creating an actual pool

i have set TIMEOUT=3600 in mrq-config.py,but it is also says the timeout is 300,i don'e know how to set the timeout correctly,any help would be appreacicated!

Vulnerable to XSS right now

Test disconnecting at other stages:

• long-running queries in tasks & worker
• specific spots in the worker code: how to make reproducible?

from mrq.context import progress

... in a task ....
progress(0.42)

• Show in dashboard
• Where to store? Redis? Worker logs?
• What happens with subtasks?

for now we have to do

$ python
from version import version
version.VERSION

which is not very convenient

I have some documentation I'd like to add, is there any plans to set-up any documentation other than README.md?

I can add to the readme for now, just thought it would be worth starting a discussion :)

The code style is not very pep8-like
we can use autopep8 to convert code style according to pep8

as the mrq use supervisor to manage the process ,i want to use mongodb to write/read the data in job,is the mongodb support the multiprocess?
use mongodb read/write in environment(--gevent 3 --processes 40) it emeges errors:
CursorNotFound: cursor id '36660188981453623' not valid at server
how to resolve?
any help will be appreaciated?

As Eddie said, the dark one is heavy on the day-to-day :)

Sparklines make this one a bit complicated... the actual value must be the log one, and then we'd need to exponentiate it again in the tooltip. Set the log factor in the raw_queues config?

Have a task called with the results of a group of task when they finish (success/failed?)

Investigate refresh_timed_set from Pricing Assistant

When the queue view show different count for redis and mongo, you can requeue those forgotten jobs by using the queue level "requeue" button. However, in this particular case the task level requeue button does not seem to have any effect.

zsets could be unqueued from the right too.

Must merge the coverage files of all the worker processes that ran.

We have some int(time.time()) in the code, but redis actually accepts float arguments.

Iif you do mrq-run mrq-test.Fetch '{"url":"xx"}' it will be run in the same python process, but if you do mrq-run --async mrq-test.Fetch it will be queued and return the job ID.

From the code the default is different, send_task() is async by default but can be used with send_task(sync=True).

We need to rename a couple things there to be consistent :)

Also, from mrq.queue import send_task is not the best we can do.

base64 or binary?

Upon upgrading to 0.0.46 from 0.0.36 I realized we need redis 2.6.0.
Not sure how you want to specify dependencies in the README, but thought it should be noted :)

Would be incredibly useful to have a list of current IO in the Dashboard:

• MongoDB/Redis queries
• HTTP
• ?

We'd collect them by monkey patching in a few places (like we already do for Mongo) and store them in worker logs.

Not an explicit goal ATM.

Will wait for full gevent support first: gevent/gevent#38

They are already saved in Mongo by the monitoring.

We want some jobs to be only queued once.

Store a hash of the task+params ?

Implement a ratelimit() primitive backed by Redis on top of which we could add more useful features

Even if running a single mongo+redis instance backend, could be useful to have the IP addresses of them on the dashboard somewhere.

• Dashboard button
• JobAction task

Just sometimes?

Useful for linking MongoDB queries to running workers.

Currently context.py uses db=0 on the Redis connection.

https://github.com/pricingassistant/mrq/blob/master/mrq/context.py#L92

Perhaps should be able to support URL's in the form:

redis://127.0.0.1:6379/db

May submit a pull-request for this over coming weekend

Currently they are only supported for mrq-worker

@sylvinus
if use mongodb in the jobs,it will be more slow due to the mongodb used to update job's status in the mrq,and the gevent context switch from mongodb will be expensive! It will be a disaster when many mongodb connections from job's function.
any suggestion available?

The current "ascending IDs" algorithm doesn't quite work. List queues and requeue tasks when redis=0 and mongo>0 ?

Like in a thread?

Explicitly tell people to use some gevent.sleep(0) ?

JS is currently in requirejs debug mode

It might be more user-friendly than needing both _jobs and _logs

max_retry ?

Where should the tasks go after that? In the failed queue? Somewhere else?

Gevent's support for PyPy doesn't seem great, check with the PyPy team.

For maximum throughput with expendable jobs, we should have "Bulk" queues that don't store anything in Mongo and store just the data they need in a Redis list/set/zset

Currently there is no way to handle a job timeout from within the task it's self, for jobs where timeouts are acceptable but need to cleanup, @sylvinus suggested we add event handlers for the on_retry, on_exception and on_timeout exceptions.

Just starting a discussion on the design/implementation of the event handlers :)

on_retry(self, exception)
on_exception(self, exception)
on_timeout(self)

and perhaps

on_success(self)?

this will allow scheduling tasks at a specific time of the day.

it's already in there but does not work and the test is commented out.