proxy_protocol broken with SSL

[Brice Figureau]

Using official nginx docker container (version 1.13.0).

with the following configuration:

...
server {
  listen *:443 ssl proxy_protocol default_server deferred backlog=1024;

  ssl                  on;

  ssl_certificate      /etc/nginx/ssl/certificate.pem;
  ssl_certificate_key  /etc/nginx/ssl/certificate.key;
  ssl_dhparam          /etc/nginx/ssl/dhparam.pem;
  ssl_session_timeout 1d;

  set_real_ip_from 172.0.0.0/8;
  real_ip_header proxy_protocol;

  access_log  /var/log/nginx/access.log;

  include "/etc/nginx/ssl-intermediate.conf";

  root /var/www/html/api/current/public;

  location / {
    try_files $uri $uri/ /index.php$is_args$args;

    include "/etc/nginx/fastcgi_params";
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/html/api/current/public/index.php;
    fastcgi_pass app_upstream;
  }

  error_page 404 /404.html;

  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }

  location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass app_upstream;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/html/api/current/public/index.php;
    fastcgi_param HTTP_X_FORWARDED_PROTO $scheme;
    include "/etc/nginx/fastcgi_params";
  }
}

server {
  listen  *:80 proxy_protocol default deferred backlog=1024;
  set_real_ip_from 172.0.0.0/8;
  real_ip_header proxy_protocol;
  return 301 https://${host}/;
}

When accessing the ssl vhost with curl through a proxy_protocol enabled proxy (currently the docker image from: github.com/convox/proxy), nginx closes the connection after reading the proxy_protocol data but before actually doing the SSL handshake, as seen in the debug log:

nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: epoll: fd:6 ev:0001 d:00007FA2B8A08010
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: accept on 0.0.0.0:443, ready: 0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: posix_memalign: 00007FA2B8F7DA50:512 @16
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 accept: 172.19.0.1:48315 fd:11
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 http check ssl handshake
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 http recv(): 42
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 PROXY protocol address: 127.0.0.1 52367
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 recv: eof:0, avail:0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 close http connection: 11
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 reusable connection: 0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 free: 00007FA2B8F7DA50, unused: 223
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: timer delta: 224143
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: worker cycle
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: epoll timer: -1

And curl complains:

curl: (35) Unknown SSL protocol error in connection to 127.0.0.1:443 

I get the exact same issue with the same nginx container running on AWS EC2 behind an ELB setup with proxy_protocol enabled.

I tracked this down to this change:
​https://github.com/nginx/nginx/commit/12f436718963f8343e38ad6d0e8f7251c95984cd

Especially in my context (I don't know if it's an issue of running in a container), but the socket has no data available, and thus the c->recv() call in ngx_http_request.c line 692 returns NGX_AGAIN which obviously is different than the size value, and the connection is closed.

Here's a patch for a workaround that apparently fixes the issue:

diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
index 476f039..de1a679 100644
--- a/src/http/ngx_http_request.c
+++ b/src/http/ngx_http_request.c
@@ -688,6 +688,7 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
 
         size = p - buf;
 
+        c->read->available = 1;
         if (c->recv(c, buf, size) != (ssize_t) size) {
             ngx_http_close_connection(c);
             return;

[Roman Arutyunyan]

Thanks for reporting this.
Please try the attached patch.

[Brice Figureau]

Replying to arut:

Thanks for reporting this.
Please try the attached patch.

Thanks!

Your patch is indeed fixing the issue (and look more correct than my fix :)

I haven't done any extensive testing, though, so I can't guarantee it is not introducing any regressions.

[Roman Arutyunyan <arut@…>]

In 7005:3e2d90073adf/nginx:

Fixed deferred accept with EPOLLRDHUP enabled (ticket #1278).

Previously, the read event of the accepted connection was marked ready, but not
available. This made EPOLLRDHUP-related code (for example, in ngx_unix_recv())
expect more data from the socket, leading to unexpected behavior.

For example, if SSL, PROXY protocol and deferred accept were enabled on a listen
socket, the client connection was aborted due to unexpected return value of
c->recv().

[Roman Arutyunyan <arut@…>]

In 7137:0e05b35beebf/nginx:

Fixed deferred accept with EPOLLRDHUP enabled (ticket #1278).

Previously, the read event of the accepted connection was marked ready, but not
available. This made EPOLLRDHUP-related code (for example, in ngx_unix_recv())
expect more data from the socket, leading to unexpected behavior.

For example, if SSL, PROXY protocol and deferred accept were enabled on a listen
socket, the client connection was aborted due to unexpected return value of
c->recv().