Large file download is not completed with sendfile option.

[jinyongchoi@…]

Hi there.

I found some issue in downloading test.
the problem is only use 'sendfile on'.
If 'sendfile_max_chunk' option or sendfile off is set in conf, download is ok.

And i seem the problem is epoll(EPOLLET) and sendfile is not working well.
If i remove EPOLLET option when event is NGX_WRITE_EVENT at ngx_epoll_add_event function in ngx_epoll_module.c, it working well.
or in testing, if it cached the content file in kernel, it working well.
(no used this command. 'echo 3 > /proc/sys/vm/drop_caches')

i found similar issue.#1174.

So, how i resolve this issue?

Testing Host : Ubuntu 19.04

Steps to Reproduce:

> mkdir -p /data0/nginx-data/
> truncate -s 10G /data0/nginx-data/10G.dat
> docker run --name nginx-10G-test -v /data0/nginx-data/:/usr/share/nginx/html:ro -d nginx:1.17.4
> sudo su
> export docker_ip=`docker inspect -f "{{ .NetworkSettings.IPAddress }}" nginx-10G-test`
> echo 3 > /proc/sys/vm/drop_caches; curl -v -o /dev/null http://${docker_ip}/10G.dat
* Expire in 0 ms for 6 (transfer 0x557ba829a5c0)
*   Trying 172.17.0.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x557ba829a5c0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 172.17.0.2 (172.17.0.2) port 80 (#0)
> GET /10G.dat HTTP/1.1
> Host: 172.17.0.2
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.17.4
< Date: Fri, 11 Oct 2019 07:29:02 GMT
< Content-Type: application/octet-stream
< Content-Length: 10737418240
< Last-Modified: Fri, 11 Oct 2019 02:23:06 GMT
< Connection: keep-alive
< ETag: "5d9fe78a-280000000"
< Accept-Ranges: bytes
< 
{ [65536 bytes data]
 19 10.0G   19 2047M    0     0  33.7M      0  0:05:03  0:01:00  0:04:03     0* transfer closed with 8589938688 bytes remaining to read
 19 10.0G   19 2047M    0     0  33.7M      0  0:05:03  0:01:00  0:04:03     0
* Closing connection 0
curl: (18) transfer closed with 8589938688 bytes remaining to read

[Maxim Dounin]

From the numbers — connection closed at 2G point — it looks like the problem is that sendfile() doesn't block on sending at all, due to fast network connection and slow enough disk. As such, it sends up to the maximum number of bytes it can send on Linux, 2G. Currently this limitation, if ever reached, is not accounted by the top-level code, and sending is not automatically retried. An obvious workaround is to use sendfile_max_chunk, which is specially designed to avoid cases when nginx spins in sendfile() for a long time when network connection is faster than disk subsystem.

Further, it looks like using sendfile_max_chunk is anyway required here, as "Time Spent" is 1 minute. If the above analysis is correct, this means that all other connection in the worker process in question were blocked for 1 minute, which is certainly not an acceptable behaviour.

Not sure if we need/want to do anything with this. The sendfile_max_chunk directive already exists, resolves this, and anyway have to be used in such configurations. It might be a good idea to introduce some reasonable default for sendfile_max_chunk though.

[jinyongchoi@…]

Thanks for the explanation.
I'm glad to know about a reason and best way.

[Maxim Dounin]

See also #1924.

[maxim]

Ticket retargeted after milestone closed

[Maxim Dounin]

See also #2211.

[Maxim Dounin <mdounin@…>]

In 7947:51a260276425/nginx:

Simplified sendfile_max_chunk handling.

Previously, it was checked that sendfile_max_chunk was enabled and
almost whole sendfile_max_chunk was sent (see e67ef50c3176), to avoid
delaying connections where sendfile_max_chunk wasn't reached (for example,
when sending responses smaller than sendfile_max_chunk). Now we instead
check if there are unsent data, and the connection is still ready for writing.
Additionally we also check c->write->delayed to ignore connections already
delayed by limit_rate.

This approach is believed to be more robust, and correctly handles
not only sendfile_max_chunk, but also internal limits of c->send_chain(),
such as sendfile() maximum supported length (ticket #1870).

[Maxim Dounin]

Fix committed, sendfile() internal limits should be correctly handled now. Further, sendfile_max_chunk now defaults to 2m (e3dbd9449b14), so worker monopolization shouldn't be a problem either.