Tasks interrupted due to nginx timeout (504)

[Mdelaf]

Hello @DataGreed, thank you very much for this library, I have been using it for a while and I love its simplicity and how well it integrates with AWS SQS.

Environment settings:

• Platform: Python 3.7 running on 64bit Amazon Linux 2/3.1.4
• Web server: nginx (default in AL2)
• SQS InactivityTimeout: 1650
• SQS VisibilityTimeout: 1680

Yesterday I realized that some of my tasks were failing in production. After checking the worker environment logs I noticed there were some requests failing with status code 504. This status code is returned by nginx when the request timeout. The default timeout is one minute, so every task with a duration of one minute or longer is forcibly interrupted.

As shown above, my InactivityTimeout and VisibilityTimeout are way more than that (around 28 minutes), but it seems that the timeout must be configured independently.

My current solution was to increase the nginx timeout. I added the file .platform/nginx/conf.d/timeout.conf:

keepalive_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
fastcgi_send_timeout 600;
fastcgi_read_timeout 600;

And it worked!

However, I am not sure if this is the correct approach. First, because I thought that worker environment had an automatic increased timeout in ElasticBeanstalk, otherwise everyone else using them would have this problem, right? Second, because the timeout I set, also affects the web environment, and I don't like that either.

Is there anyone else having this same problem? How did you go about it?

[DataGreed]

@Mdelaf that's an interesting situation.

I didn't encounter this issue before and I have some heavy jobs in one of my projects that are taking several minutes, although I am using apache with default timeout config on this one, and I have a load-balancer set up, so the timeout on the load-balancer is different from the timeouts on the actual instances. From my experience with worker environments on EB so far, InactivityTimeout and VisibilityTimeout seem to be respected when set up with a load balancer.

I totally agree that increasing timeouts on both environments does not sound like an elegant solution to the problem.

I can suggest the following solutions:

1. Migrating to load-balanced environment, so timeouts to external clients will be separate from internal ones.
2. Using different timeout.conf files for worker and web environments. You can do this by creating your own deploy script that will receive environment type as an argument (worker or web) and will copy the corresponding timeout.conf to .platform/nginx/conf.d/ before running the actual eb deploy command.

Not sure if there is another solution to this.