Jobs that take longer than 60s fail. about active-elastic-job HOT 16 CLOSED

According to the Nginx docs, the max timeout is 75s http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout

from active-elastic-job.

+1 on this. Any way to get around this?

from active-elastic-job.

@Tawas @jcoyne do you know of a way to get around this? I'm testing the gem in AWS in a staging environment and we have some jobs that take over 60s, they are all failing. I have been looking at other options/gems but none of them seem to take into consideration the SQS and Worker environments working together.

from active-elastic-job.

Hi @javs-perez @jcoyne

Thanks for your submission. I was doing a little research as well and unfortunately I think there is no elegant solution. You will have to increase nginx' timeout setting be configuring a container command. Depending on whether you have a Docker setup in place or not you can try these approaches:

• Without docker
• With docker

In my understanding, AWS Elastic Beanstalk fails to infer the nginx timeout setting automatically from the SQS visibility setting. That's something that AWS could / should do better.

Cheers,

Tawan

from active-elastic-job.

Hi @tawan,

Thanks for taking the time to look at this. The issue has been resolved. I did not realize our app was running rack-timeout with a 60 second default, so I changed that value together with the nginx' configuration and it resolved the issue.

Either way, I wish the SQS daemon inside the worker would handle the connection to the app differently, an HTTP request could cause various issues, specially for jobs that take a long time.

Thanks again.

from active-elastic-job.

If you add a message on the README about the default config of NGinx causing this problem, I think that would resolve this ticket.

from active-elastic-job.

@jcoyne:

According to the Nginx docs, the max timeout is 75s

That's the timeout to establish a connection with the backend server, not the timeout to wait for a response. I'm still trying to figure out if such a parameter exists for the upstream/proxy modules. So far all I've found is a fastcgi_timeout (that will, in fact, take an arbitrarily high value).

from active-elastic-job.

Oh, there it is:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

from active-elastic-job.

@jcoyne @tawan In case you're still looking for a fix: https://github.com/nulib/avalon/blob/deploy-aws-staging/.ebextensions/05_nginx_worker_timeout.config

Seems like documenting that (or something similar) in the README would close this issue.

from active-elastic-job.

Instead of increasing the nginx timeout to be longer that what jobs are expected to last (which could be hours), can ActiveJob respond with a 200 OK if the job started processing and then re-queue it if it fails?

Setting nginx's timeout to minutes or hours seems inappropriate and could have problematic side effects (e.g. when the worker is actually unresponsive).

from active-elastic-job.

@waissbluth it was my understanding that active-elastic-job is are actually executing the worker in a HTTP request context. So, once the connection dies, the work stops.

from active-elastic-job.

yeah that's a good point; how about streaming a response back in order to keep the connection alive?

The worker environment actually defines:
Connection timeout Number of seconds to wait for a response from the application when establishing a new connection. (Max 60)
Inactivity timeout Number of seconds to wait for a response from the application on an existing connection. (Max 36000)

This would probably require streaming an initial chunk with an HTTP 200, then starting the job and hopefully finishing within the "Inactivity Timeout"

from active-elastic-job.

@mbklein https://github.com/nulib/avalon/blob/deploy-aws-staging/.ebextensions/05_nginx_worker_timeout.config

This link is dead, and I can't find that file in the repo. I tried using ebextensions to raise the timeout but it's not working.

from active-elastic-job.

I was able to get this working by adding a file with this to the nginx/conf.d directory:

fastcgi_read_timeout 1800;
proxy_read_timeout 1800;

Of course, the 1800 seconds could be changed to whatever fits your needs.

--

Although I understand the concerns about just increasing the nginx read timeout, I feel this is sufficient for the time being. I'm attempting to clean up this git and get the gem more up to date so I am going to go ahead and close this issue. If there is a stronger desire to add the 200 OK response, then re-queue on fail that can be made as a new issue.

Side note: that may be able to be "cheated" by running the job in a thread, but I have not tested if that works or not.

from active-elastic-job.

@joeyparis should we document this nginx hack?

from active-elastic-job.

@januszm that's a good idea. Especially considering the nature of jobs and that taking longer than 60 seconds probably isn't that uncommon. That being said, my implementation of this in production has a few extra steps that way the nginx timeout is only extended on the worker server and not the main one. I'm on Amazon Linux 2 so I'm using the platform hooks, but I'm sure this would work in the older environment as well just in a different location.

.platform/hooks/predeploy/01_nginx_read_timeout.sh

#!/usr/bin/env bash
set -xe

if [ $PROCESS_ACTIVE_ELASTIC_JOBS ]
then
	cat >/var/proxy/staging/nginx/conf.d/read_timeout.conf <<EOL
fastcgi_read_timeout 1800;
proxy_read_timeout 1800;
EOL
fi

from active-elastic-job.