mail proxy connect issue with Outlook

[9yw@…]

When I use Outlook2016 (with wrong password) to connect a "mail proxy" type SSL port(465/993/995), always get 0x800CCC0F error.But it's ok when using "stream proxy".
After some research,I got a "I don't know how but it works" solution:

Edit mail/ngx_mail_handler.c:
1.copy void ngx_mail_close_connection to "ngx_mail_close_connection_keep"

2.remove "ngx_close_connection(c);" from void ngx_close_connection(c);
3.edit void ngx_mail_send change line 674 to ngx_mail_close_connection_keep(c);

        if (s->quit) {
            ngx_mail_close_connection_keep(c);
            return;
        }

Add "void ngx_mail_close_connection_keep(ngx_connection_t *c);" to mail/ngx_mail.h

=============
I don't know how it works, also don't know dos it has side effects,so I hope the connect issue can be fixed official.

[Maxim Dounin]

In theory, command pipelining during authentication might result in connection being closed with RST, and the response to authentication removed from the client's socket buffer due to RST. In HTTP, ​lingering_close is used to close connections without triggering RST when additional data can be sent by the client: this ensures that the client will be able to see the response. This is not what expected to happen with mail protocols though, as command pipelining is not expected to be used at all unless explicitly allowed by the server, and not expected to be used during authentication.

I did some testing with Microsoft's email clients I was able to obtain, notably:

• Windows 10 Mail, Version 16005.14326.20544.0
• Outlook for Mac, Version 16.56 (21121100)
• Outlook for Windows, Version 2112 Build 16.0.14729.20224

When tested with nginx 1.21.5, all tested clients seem to work fine, including with various artificial delays introduced. Authentication errors are properly recognized as such, and not as 0x800CCC0F error, and suggest to re-enter credentials. As expected, no tested clients are using command pipelining during authentication, and lingering close in mail shouldn't be needed (or at least unrelated to the issue observed).

The next best guess is that the particular issue reported in the ticket is a bug/misbehaviour of Outlook which drops the response to authentication when SSL connection is closed. This can be affected by SSL shutdown fixes is 09fb2135a589 (nginx 1.19.2; with the fixes, mail proxy now uses bidirectional SSL shutdown instead of only sending close notify alert without waiting for client's one).

Further testing with mail proxy switched back to unidirectional SSL shutdown suggests this is indeed the case: while Windows 10 Mail and Outlook for Mac work correctly regardless of the SSL shutdown behaviour, Outlook for Windows behaves differently. With bidirectional shutdown it asks for credentials again and again, but with unidirectional shutdown it only asks for credentials occasionally, and in most cases just fails with the error 0x800CCC0E. This seems to be the issue identical to the one with Outlook 2016.

Overall, this seems to be an issue in Outlook, which does not tolerate SSL connections being closed by the server before Outlook handles the response to the authentication command. It is worked around in nginx 1.19.2, as bidirectional SSL shutdown ensures that the connection is only closed after the response is received.

Closing this as fixed. Thanks for bringing attention to this Outlook behaviour.

[Maxim Dounin]

Also, just a side note. In most cases it is a good idea to allow users to re-enter login and password on the same connection at least a couple of times instead of closing the connection right after the first error. This can be done by returning an auth response with the Auth-Wait header (usually till Auth-Login-Attempt reaches some limit), see ​authentication protocol description. With this approach handling of connection closing in different email clients becomes mostly irrelevant.