Downloads stop after 1GB depending of network

[nudgegoonies@…]

Hi,
we tried nginx version 1.6.2 till version 1.12.2 and have a problem when used as proxy before artifactory. Downloads get interrupted at 1GB.

This behavior depends on the internal VLAN. On one VLAN this always happens. On an other VLAN it never happens. This is size limited, not time limited. From some network it stops after 30 seconds and from one other slow network it stops after 13 minutes.

We made a minimal proxy setup with apache and this works with all VLANs. This is why we expect it has something to do with nginx or the combination of nginx and TCP/IP stack of linux.

In wireshark we see "TCP Dup ACK" on the client side sent to the nginx server.

Wget fails with connection closed at byte 1083793011 but continues download with partial content. docker can't handle this and our customers can't download docker images with layers greater 1 GB.

The following text shows two anonymized minimal configs. The nginx config that is problematic and the apache config that works:

NGINX config:
server {
    listen *:80;
    server_name NAME;
    client_max_body_size 3G;
    access_log /var/log/nginx/NAME.access.log;
    error_log /var/log/nginx/NAME.error.log;

    if ($request_uri !~ /artifactory/) {
        rewrite ^ $scheme://NAME/artifactory/ permanent;
    }

    location /artifactory {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://ARTIFACTORY:PORT;
        proxy_pass_header Server;
        proxy_read_timeout 90s;
    }
}

APACHE config:
<VirtualHost *:80>
    ServerName NAME
    ServerAdmin NAME

    ErrorLog ${APACHE_LOG_DIR}/error.log

    LogLevel warn

    ProxyRequests Off
    <Proxy *>
      Order allow,deny
      Allow from All
    </Proxy>

    ProxyPass / http://ARTIFACTORY:PORT/
    ProxyPassReverse / http://ARTIFACTORY:PORT/
</VirtualHost>

[Maxim Dounin]

The 1GB limit suggests that the problem is due to ​proxy_max_temp_file_size. It is one gigabyte by default, and if the limit is reached, nginx will stop reading from the backend till all disk-buffered data are sent to the client. This in turn can result in a send timeout on the backend side.

Please check nginx and your backend logs to see what happens here. Likely there are something like "upstream prematurely closed connection" in nginx error log, and send timeouts in your backend logs. Alternatively, just check if proxy_max_temp_file_size 0; helps (this will disable disk buffering completely).

If the above suggestion is true, the two possible solutions are:

• Tune proxy_max_temp_file_size. Consider either configuring the limit above the size of all expected responses, or small enough for your backend to don't time out. In particular, proxy_max_temp_file_size 0; might be a good choice when proxying large files.
• Tune your backend timeouts appropriately.

[nudgegoonies@…]

This was it! Thank you very much for your answer!

We used the first solution and raised the limit.

We were so confused because the circumstances with the different networks. We already tried throttleing and stuff to reproduce it.

[Maxim Dounin]

Ok, thanks for the confirmation.

For the record, this seems to be more or less typical problem, reported from time to time (see ​here for an example). While this behaviour is not a bug, we might consider doing some improvements to make such timeouts less likely to happen. In particular, reviving and improving the ​p->cyclic_temp_file feature (currently disabled in all modules) might be the way to go.