http/2 downloads broken during reload

[jdewald@…]

In the situation where an HTTP/2 download spans nginx reloads, the client will receive a failure even though NGINX drains the stream it is working on prior to closing the worker.

As far as I can tell, the issue is that because HTTP/2 is actually two-way communication, when the client goes to send WINDOW_UPDATE, PING or even request more data it receives back an RST from the kernel because the worker has now gone away. So the stream is complete but the client is unaware that the connection is actually closed.

This has been reproduced 1.10.3, 1.11.2 and 1.13.12

To be clear, this is unrelated to the send_timeout. From NGINX's point of view the download completes successfully.

I understand this would be a tricky problem as I do not know that the combination of TCP and HTTP/2 provide a mechanism to tell the client"the current stream will be completed but that no other application-level messages should be sent on this connection".

Step to Reproduce
1 - Create a server block with http2 enabled (for comparison, I also have a non-http2 listen port)
2 - Initiate a download that takes, say, 1+ minutes to download (for local testing, I used rate_limit) over the http2 port.
3 - While that is running, issue an nginx reload
4 - Watch access logs and when you see it complete in nginx (with no errors) the download will fail, often at 99%. In case of curl it reports "unexpected EOF". Chrome's http2 inspector will show a read error

For my purposes I ran the curls in a loop and issued reloads after I saw "Done with 3" and "Done with 8"

for i in `seq 1 10`; do curl -s -o /dev/null  -k "https://127.0.0.1:2443/7mb.dat?foo=$i"; if [ $? -eq 0 ]; then echo "Done with $i"; else echo "Failed $i"; fi ; done
Done with 1
Done with 2
Done with 3
Failed 4
Done with 5
Done with 6
Done with 7
Done with 8
Failed 9
Done with 10

Note: This works best if it takes 1 or 2 minutes as it's important to be able to have data in the tcp buffer I believe (otherwise the closing of the connection will immediately correspond with all data delivered).

Perform same steps using HTTP/1 and of course it will complete successfully

Configuration I use

server {
    listen 127.0.0.1:443 ssl;
    listen 127.0.0.1:2443 ssl http2;
    limit_rate 512k;
    access_log /tmp/http2test.access.log;
    error_log /tmp/http2test.error.log;

    ssl_certificate unittest.crt;
    ssl_certificate_key unittest.key;

    root html;
}

[Maxim Dounin]

See also #1250, which is related. Both problems should be solved once lingering close will be introduced in HTTP/2.

[Ruslan Ermilov <ru@…>]

In 7673:c5840ca2063d/nginx:

HTTP/2: lingering close after GOAWAY.

After sending the GOAWAY frame, a connection is now closed using
the lingering close mechanism.

This allows for the reliable delivery of the GOAWAY frames, while
also fixing connection resets observed when http2_max_requests is
reached (ticket #1250), or with graceful shutdown (ticket #1544),
when some additional data from the client is received on a fully
closed connection.

For HTTP/2, the settings lingering_close, lingering_timeout, and
lingering_time are taken from the "server" level.

[Ruslan Ermilov]

HTTP/2 connections now support lingering close mechanism.