A SignalException occurred... SIGTERM about redis-rb HOT 7 CLOSED

SIGTERM means that another process on the system sent a SIGTERM to your process that forced your process to crash with a stack trace of whatever it happened to be doing when it received the SIGTERM. So this is not a code problem with redis-rb, you need to look elsewhere on your system to see who is sending you the SIGTERM's and make it stop.

from redis-rb.

Wow you are super responsive! Thanks for the quick reply.

I understand what you're saying -- but my question is: which process actually received the SIGTERM? The rails server? Or the Redis server? Or something else? And do you have an idea how we could go about determining who is sending the SIGTERM? We were unable to reproduce this problem in our staging environment... this happened when we pushed to production. We were forced to roll back within a few hours.

Unfortunately I was out while this effort was going on, but I think they had setup 3 Redis servers behind a load balancer. I personally think that this added complexity was the cause of the problems... I'm not sure if the load balancer was configured for round-robin or failover, but at any rate, I'm guessing that that was the issue.

from redis-rb.

Yeah redis doesn't really work behind a load balancer out of the box, it kind of relies on being its own atomic server, you can do replication but it requires you to write to master and read from slave, no simple round robin.

It looks to me like your ruby process or rails process(whichever process produced that log message) got a SIGTERM sent to it right in the middle of reading a reply from redis. Do you have any watchdog daemons on your system that will try to kill things that use too much cpu or memory? Any monitoring tools that would cause this? Nothign in redis or redis-rb ever sends a SIGTERM so it has to be something else in the operating system doing it. Are you trying to run the rails processes under monit perhaps?

from redis-rb.

It's a rails process, running under Passenger. I have a question out to ops about whether or not we have a watchdog daemon or monitoring tool. But also... if the Redis servers weren't setup properly for replication, maybe the request to Redis timed out, and the SIGTERM was from Passenger itself, after reaching some timeout on the HTTP request (?).

Anyway thanks for the responses. I'll let you know if we make any progress with this.

from redis-rb.

OK so the SIGTERM exceptions were a bit of a red herring I think. They only occurred during deployments. But we did notice a slowdown after deploying our app that had vanity implemented; our Nagios monitoring started sending alerts that it was unable to connect to our servers. Unfortunately, we don't really have much more information than that... so we're going to deploy with vanity again, in hopes that we can gather more information about where the bottleneck is.

Thanks again for your help.

-Steve

from redis-rb.

Well if anyone else stumbles upon this thread, we found the problem: Vanity initiates the connection to Redis during app startup, which causes application instances (under Passenger) to share the same connection. This is bad and causes application instances to hang and eventually freeze up Apache itself.

See this thread for a fix: http://groups.google.com/group/vanity-talk/browse_thread/thread/532d0f4e707cf814?pli=1

from redis-rb.

YEah using redis-rb with passenger you need to make sure you redis objects dont get created until after the passenger process is forked, otherwise all yoru passengers will share the same redis socket and badness unnsues.

from redis-rb.