Comments (8)
Did anyone find out if Pyres is compatible?
from pyres.
@louisameline and @bluegod i guess the answer depends on the definition of compatible. Can you enqueue jobs with another implementation and pop them off with pyres? If the json that is push is the same, then yes. For instance, a pyres job looks for a json payload that has the following keys:
class, args, enqueue_timestamp. From the tests you can see what a sample payload looks like
{'class':'tests.Basic','args':['test1'],'enqueue_timestamp':job.enqueue_timestamp}
from pyres.
Thanks for your answer.
Ok, I see that class
and args
seem to be the only properties needed to enqueue a job (you add an extra enqueue_timestamp
and php-resque adds an extra id
but I guess this does not matter).
It gets more complex when you deal with job failures though. Coffee-resque has this payload similar to Resque's one (https://github.com/resque/resque/blob/master/lib/resque/failure/redis.rb) that makes its errors seamlessly monitorable in Resque-web too :
{
"worker": "node:7236:node_contact_import",
"queue":"node_contact_import",
"payload": {
"class":"import",
"args":[....],
"id":"716e8ac85c64e10286df04881c28f70d"
},
"exception":"error",
"error":"my error name",
"backtrace": [
" at Connection.parseE",
" at Connection.parseMessage,
" at TCP.onread (net.js:526:21)"
],
"failed_at":"Wed Aug 13 2014 11:04:12 GMT+0200 (Paris, Madrid (heure d�été))"
}
Have you done the same for Pyres ?
I could check the source myself but I thought I'd ask you at the same time to include this information in the documentation if you're willing to.
Thank you.
from pyres.
I agree @louisameline that we should unambiguously document the level of compatibility between pyres and resque. I am about to wrap some python code in a worker job of some kind, and would prefer to use pyres, but want to know "Well is it a clone of resque, or not a clone of resque?"
from pyres.
@louisameline sorry about the long delay, i've not been seeing github notifications in my gmail. Pyres follows that same failure json. There may be some slight differences because it's not been a priority to track resque in the last few years.
from pyres.
Hello again !
Thank you, I am eventually implementing Pyres. Unfortunately with difficulty because the documentation falls short of thorough and explained examples for python total newbies like me.
Anyway, I am trying to send a job from Python for Node to pick it up. I have created a parse
class that has a queue = 'node'
attribute and I call resque.enqueue(parse, arg)
. The problem is that the class
property in the JSON payload is __main__.parse
while Node only expects parse
. What can I do ?
Thank you.
from pyres.
Looking at the source, I found a resq.enqueue_from_string(className, queue, *args, **kwargs)
method that does just what I need. I think it should be documented.
Next issue : the other way around, when enqueing from Node to Python. From the payload, Pyres seems to expect a class property in the form packageName.className
. I guess I could include the package name when I enqueue in my Node app, but that does not really sound right. My Node app should probably not have to care for what package will be used by Python, that's outside of its scope.
If pyres_worker and pyres_manager need the location/package of the class, could we pass it as an IMPORT_PATH
option in the command line instead ? I don't know if it would be a very python-like way of doings things but it would :
- solve the issue of the package name being present in the
class
property - save me the trouble of having to learn how
import
works and how I'm supposed to declare my app as a package. I know that sounds lazy :p ^^
Thanks
from pyres.
I confirm that everything works well when I package my python worker class under the name "workers" and give a "workers.parse" class when I enqueue in Coffee-Resque (Node). However I haven't tested to run ResqueWeb yet.
For newbies like me : to package my parse
class (included in a file called "workers.py") so that Pyres will find it, I followed the simple instructions given in this helpful discussion. Basically, you just have to execute this file :
#!/usr/bin/python
from distutils.core import setup
setup(name='workers',
version='1.0.0',
py_modules=['workers'],
)
from pyres.
Related Issues (20)
- cannot create ResQ instance if Redis requires a password HOT 5
- Releasing a new version HOT 7
- worker pool
- document perform hooks
- Speed & More Pythonic HOT 1
- "No module named xxxx" raised when enqueue a local class HOT 3
- merge work in from pinterest branch HOT 4
- pyres_worker script can't connect to password-protected redis server
- It seems that pyres(1.4.1) installed by pip is not the same as the 1.4.1 version of git HOT 1
- Python 3 Support HOT 2
- Remove a Delayed Job
- is there a before_first_fork or similar hook? HOT 2
- release new version HOT 3
- Why doesn't after_perform run when an exception is raised?
- retry wont work if retry_timeout not set, is this intentional?
- Module Import Error HOT 1
- Docs link broken HOT 3
- Is this project still active? HOT 4
- AttributeError: 'int' object has no attribute 'iteritems'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyres.