TCP RST with SSL and HTTP Connection:close header

[elukey@…]

Hi everybody,

we use Nginx to terminate TLS in front of a wide range of systems, but the biggest use cases are Varnish and Apache httpd.

While running some tcpdumps on one application server running Nginx in front of Apache and HHVM I noticed regular RST packets returned by Nginx to the client when the request contained "Connection: close" and https was used.

I investigated the issue in ​https://phabricator.wikimedia.org/T163674 and came up with the suspicion that Nginx is eager to close the TCP connection after the HTTP request is completed, not leaving enough time for SSL to shutdown cleanly (​https://phabricator.wikimedia.org/T163674#3257675 contains debug error logs to compare).

While reading ​https://wiki.openssl.org/index.php/Manual:SSL_shutdown(3)#NOTES it seems that this behavior is allowed but I am wondering if it is intended for Nginx to return a RST or not. In the former case it would be really nice to have a note in the docs advertising this "feature" so people will know it.

Last but not the least, this issue is not causing any errors on the client side, the RST seems to be handled fine. The only side effect is polluting TCP RST metrics and confuse a bit admins while debugging :)

Thanks in advance!

Luca

[Maxim Dounin]

Quoting the code:

        /*
         * The majority of browsers do not send the "close notify" alert.
         * Among them are MSIE, old Mozilla, Netscape 4, Konqueror,
         * and Links.  And what is more, MSIE ignores the server's alert.
         *
         * Opera and recent Mozilla send the alert.
         */

        c->ssl->no_wait_shutdown = 1;

If there are reasons to assume this comment is no longer true, the code may be reconsidered. It seems to be still true though.

[elukey@…]

Thanks a lot for the answer, I didn't find the comment in the code before opening the task. I checked also Apache httpd and it seems doing the same thing, and the specs are definitely respected. It would be great to support the use case in which both parties send the close notify, but probably it is not that easy to do.

Would it be possible to advertise this behavior in your documentation with a little text box? It seems something worth to share.

Thanks!

[Maxim Dounin]

Replying to elukey@…:

Thanks a lot for the answer, I didn't find the comment in the code before opening the task. I checked also Apache httpd and it seems doing the same thing, and the specs are definitely respected. It would be great to support the use case in which both parties send the close notify, but probably it is not that easy to do.

It is not possible to find out if close notify is expected or not, so the only working approach is to follow majority of clients.

Would it be possible to advertise this behavior in your documentation with a little text box? It seems something worth to share.

I don't really see how it can help users, as it is an implementation detail. The comment in question should be enough, IMHO.

[elukey@…]

I am a user and I had to spend a lot of time to figure out where that RST was coming from, I don't really agree that it is an implementation detail. Anyhow, thanks a lot for the explanation!