Worker disconnects 30 sec after PRE_SLEEP about java-gearman-service HOT 10 OPEN

If it's happening consistently on some machines and never on others, it sounds 
like an environmental issue. It may not be an issue with the application. It 
may have something to do with the windows keepalive settings. It's possible 
that it's 30 seconds. The workers should ping the server frequently. I think 
it's every 59 seconds.

Can you attach some source that produces this behavior?

When I have a free second, I'll try to reproduce this with different 
environmental settings. If you figure out a solution, can you please keep me 
updated?

Thanks,
Isaiah

Hi Isaiah,
I'm sorry not replying earlier but I'm on holidays now. This week and next week.

After that I'll give you something from the application, but it's really like 
having workers registered to a remote server with 4-5 different function names 
but on 1 thread, and having clients on the same machine as the server 
submitting foreground jobs with async callbacks.
As my point with using Gearman was to keep this part of my application as 
simple as possible, there's really no extra tricks I used here.

Thanks,
Balazs

Hi Isaiah,

I'm back and still struggling with this problem. I've already checked the 
Windows' keepalive settings, but nothing specific is set on the machines - so 
it has to be the default 2 hours. Any other idea to pinpoint the origin of the 
problem?

Thanks,
Balazs

Maybe it's a different problem but after a couple of hours (>8) of running the 
server all of the clients lose their connections and they never come back. They 
periodically try to reconnect - there's no problem with that - but they don't 
see the server.
My GearmanLostConnectionPolicy logs 
GearmanLostConnectionGrounds.FAILED_CONNECTION on the workers.
There's nothing of it in the server's log.

I'm worried because when I started to use your framework it all seemed to be 
OK, just when I finally want to test it in real-life environment it all seems 
to be very unstable.

Of course I'm aware that lots of other people are using it without problems so 
it might be my environment not the framework, I'm just asking for help what to 
check to discover what's wrong.

Thanks,
B

"They periodically try to reconnect - there's no problem with that - but they 
don't see the server. My GearmanLostConnectionPolicy logs 
GearmanLostConnectionGrounds.FAILED_CONNECTION on the workers. There's nothing 
of it in the server's log."

This sounds like something totally different. In fact, it sounds a lot like 
Issue 41. I've been having a hard time tracking this down. If you can attach 
some source that reproduces this behavior somewhat regularly, I'd really 
appreciate it.

I've checked Issue 41, but it's not like that. Restarting the worker does not 
help, it's the server. Has a possible relation with Issue 13, see my comments 
there.

Here are my sources. It's rather complicated but not the Gearman parts.
It creates the GearmanServer in "ContextManager" and passes it to the 
constructor of "Queue". That also stores it because for every "BatchImpl" (that 
is a batch of jobs) it creates a GearmanClient. Otherwise Gearman-related parts 
of the client and the worker is based on your "EchoClientAsynchronous" and 
"EchoWorker" example classes.

And I've created a thread dump from the "hanging" server. The "gearman-xxx" 
thread looks exactly the same as in a normally functioning server.

Balazs,

Just to clarify,  is the only issue you are seeing is the worker disconnecting 
after 30 seconds?  I'm fairly certain that this is an envoirnmental issue. I'm 
think that I'll address this by allowing the user to set the ping interval.  
It'll be a few days before I have the time, but I'm hoping to get your issues 
addressed asap.

No, unfortunately.
There is this one with the 30 seconds and there is Issue 13, what might be a 
64bit JRE issue. I can live with that other one as long as I don't try to run 
the server on 64bit, but I can't tell what environment setting causes this one, 
so I can't stop this happening.
The good part: it's only happening when there are no jobs and the workers are 
reconnecting after a minute. It's just annoying and giving the feeling to my 
bosses that the system is not stable enough.