named jobs about beanstalkd HOT 43 CLOSED

This is probably the easiest way to create a synchronous job trough beanstalkd. One additional bonus would be something in style with TTL on these jobs, so if the other side of the synchronous side hangs up (not too uncommon), these would be deleted after TTL secs.

from beanstalkd.

Yes please! ;-)

from beanstalkd.

+1 I saw that in the roadmap, and know exactly what I'd use it for.

from beanstalkd.

What would be the difference between this and the existing job ids? If we're suggesting features I'd love deleting or re-prioritising jobs by id - related?

from beanstalkd.

You have no control of existing job ids. The idea is to be able to assign job ids on the client. This way, multiple (distributed) clients could interact (reschedule, cancel, pop, etc) the same job.

from beanstalkd.

You can already delete any job by id. The only exception: you cannot delete a job some other client currently has reserved.

Re-prioritising jobs is a nice idea. Feel free to open a ticket for it.

from beanstalkd.

kr, you can delete by ID, but it assumes I know the id. I want the following scenario: client A queues new job (name: job1), client B, which knows nothing about client A, reaches in and modifies the payload / time to run / etc for job1. To accomplish this, the name has to depend on some external knowledge (not beanstalkd assigning job ids). For example.. let the job id be an email of the user, or a qualified url, etc.

from beanstalkd.

Got it. That's exactly what this ticket is for.

I should try to be more clear about who I'm replying to. Sorry about that.

from beanstalkd.

Ok now I understand the motivation for this feature, and it's another +1 from me. Definitely opens up useful possibilities. I'll open another ticket for reprioritizing jobs.

from beanstalkd.

+1

from beanstalkd.

For synchronousness to work, there would have to be some sort of wait-job-done command or so, as well. Actually, synchronousness could then be achieved without the use of named jobs at all.

from beanstalkd.

See #3 for a great way to notify yourself when a job completes.

from beanstalkd.

this would be extremely useful

from beanstalkd.

+1
very useful feature.

from beanstalkd.

+1

from beanstalkd.

+1 ... please.. is this ever going to happen? .. it would be nice to be able to query the tube on a key to get the particular job, very useful

from beanstalkd.

All the pluses in the world probably won't change the fact that Keith works with beanstalk when he wants and has time to spare. The best way to get this done within the timeframe you find necessary is to attach a patch or possibly pay someone else do do it for you.

from beanstalkd.

+1

from beanstalkd.

Will this feature allow us to coalesce jobs based on a unique identifier? That would be awesome.
+1

from beanstalkd.

+1 as well. Could really use this feature.

from beanstalkd.

@kr so i wanted to try working on this feature since we do need it. Did you have a spec somewhere of how you wanted to implement it ?

I was thinking of reusing job.id for this. So when enqueueing a job, if the id parameter is passed in, then do a job_find(id). If the job exists, drop the JobRec struct and create a new one with the job information passed in. Thoughts ?

from beanstalkd.

Clients might assume the job id is always an integer, and
I think the job name should be a string. This means it needs
to be another field. This also implies a new put command,
probably put-named, with a signature like this:

put-named <name> <pri> <delay> <ttr> <bytes>

There will have to be corresponding changes elsewhere,
such as the output of stats-job.

One point of debate: what to do when submitting a new
job with the same name as an existing job. Either the job
remains unchanged, or it's replaced by the contents of
the new job. Until now, beanstalkd never modifies the
contents of an existing job, and, in the absence of a good
reason to change, we should probably stick to that rule.

from beanstalkd.

I agree on the new put command and its signature, my only question would be how do you recommend we update job_find to find the job. Currently, it looks like, all jobs are maintained in a hash indexed by the job.id, do you recommend a second hash ? or ..

One point of debate: what to do when submitting a new
job with the same name as an existing job. Either the job
remains unchanged, or it's replaced by the contents of
the new job.

The use case i need the named jobs functionality for is as follows. In the current setup, there are multiple points of entry from where jobs can be enqueued. Work here is a primary key identifier and a method call. So what i expect put-named to do is to re-enqueue (delete the original job, add new job with same ident) the job with the new data. This way we aren't changing the existing job, we are expiring it for the new job enqueued with the same put-name. The onus is now on the user to either use priority to change when the job is processed or be content with the new job that has been enqueued.

Thoughts ?

from beanstalkd.

The primary use case for this is "de-duplication" of work, where the same
job can potentially be submitted multiple times, but only needs to be
processed once. In that case, it seems better to let the original job remain
in its position in the queue.

If a user wants to delete the existing job, that should still be possible, but
made explicit: look up job, delete old job, submit new job.

This highlights a requirement for a way to look up a job by name, perhaps
stats-job-name, but that seems like a good idea anyway.

from beanstalkd.

This would be a great feature indeed and we'd need it for a beanstalk implementation of a generic job queue that needs to support unique & recurring jobs (for TYPO3 Flow).
RE "what to do when submitting a new job with the same name as an existing job": I'd say beanstalkd should fail if you try to reuse an identifier w/o removing the respective job first.

from beanstalkd.

+1 We need this feature, too! Looking forward to it! Everything else works like a charm, but the missing possibilities for implementing some kind of deduplication really hurts.

from beanstalkd.

+1 we also need this feature. any plan in the future?
thx

from beanstalkd.

+1 we like the name features

from beanstalkd.

+1 here :)

from beanstalkd.

Any word on this? Been 4 years.

from beanstalkd.

+1 I definitely need this feature.

what is the status ?

from beanstalkd.

Status is I've moved onto Kafka :)

On Thu, Aug 29, 2013 at 9:25 AM, chandeeland [email protected]:

+1 I definitely need this feature.

what is the status ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-23448974
.

from beanstalkd.

+1 to this as well.

I'd also accept just being able to specify an integer for the job ID in the put command, rather than just taking whatever's assigned. Really, any way to do this at all would be a huge improvement and could eliminate a few horrible hacks from my usage.

from beanstalkd.

+1 from me too for the named jobs. I like the concept and got here when I was thinking of controlled environment in beanstalkd server. So I was thinking of a job to live in more than one tube with the same job-id so that its easier to maintain a job and its lifecycle in beanstalk. So @kr when will it be available for us though ? Just asked out of curiosity, will it be happening or it will be just a concept.

from beanstalkd.

Hi,
I use beanstalkd 1.8
I think such improvements would only complicate the beanstalkd.
There many problems with named jobs - duplicates, more memory using and some potential bugs in future.
May be to use beanstalkd & redis? Who want they can use maping: a name of job in redis -> beanstalkd ID from redis? Please don't do from the beanstalkd a the new redis for example.

from beanstalkd.

+1 I would really appreciate this feature.

Or, in addition, It would be nice to flag an entire queue as a "Set", so that two jobs with the same data cannot co-exist in the same queue.

from beanstalkd.

+1

from beanstalkd.

+3.14

from beanstalkd.

Any news on this feature?
+1

from beanstalkd.

+1 I need this feature

from beanstalkd.

+1

from beanstalkd.

This would be super useful as a way to reduce redundant work by queue consumers. I'd love to see it make it into a future release!

from beanstalkd.

Closing in favor of #379

from beanstalkd.