Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#1278 closed defect (fixed)

proxy_protocol broken with SSL

Reported by: Brice Figureau Owned by:
Priority: major Milestone:
Component: nginx-core Version:
Keywords: proxy_protocol ssl bug Cc:
uname -a: Linux 24be690fac3d 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.13.0
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.1t 3 May 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' --with-debug

Description

Using official nginx docker container (version 1.13.0).

with the following configuration:

...
server {
  listen *:443 ssl proxy_protocol default_server deferred backlog=1024;

  ssl                  on;

  ssl_certificate      /etc/nginx/ssl/certificate.pem;
  ssl_certificate_key  /etc/nginx/ssl/certificate.key;
  ssl_dhparam          /etc/nginx/ssl/dhparam.pem;
  ssl_session_timeout 1d;

  set_real_ip_from 172.0.0.0/8;
  real_ip_header proxy_protocol;

  access_log  /var/log/nginx/access.log;

  include "/etc/nginx/ssl-intermediate.conf";

  root /var/www/html/api/current/public;

  location / {
    try_files $uri $uri/ /index.php$is_args$args;

    include "/etc/nginx/fastcgi_params";
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/html/api/current/public/index.php;
    fastcgi_pass app_upstream;
  }

  error_page 404 /404.html;

  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }

  location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass app_upstream;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/html/api/current/public/index.php;
    fastcgi_param HTTP_X_FORWARDED_PROTO $scheme;
    include "/etc/nginx/fastcgi_params";
  }
}

server {
  listen  *:80 proxy_protocol default deferred backlog=1024;
  set_real_ip_from 172.0.0.0/8;
  real_ip_header proxy_protocol;
  return 301 https://${host}/;
}

When accessing the ssl vhost with curl through a proxy_protocol enabled proxy (currently the docker image from: github.com/convox/proxy), nginx closes the connection after reading the proxy_protocol data but before actually doing the SSL handshake, as seen in the debug log:

nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: epoll: fd:6 ev:0001 d:00007FA2B8A08010
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: accept on 0.0.0.0:443, ready: 0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: posix_memalign: 00007FA2B8F7DA50:512 @16
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 accept: 172.19.0.1:48315 fd:11
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 http check ssl handshake
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 http recv(): 42
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 PROXY protocol address: 127.0.0.1 52367
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 recv: eof:0, avail:0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 close http connection: 11
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 reusable connection: 0
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: *1 free: 00007FA2B8F7DA50, unused: 223
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: timer delta: 224143
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: worker cycle
nginx_1    | 2017/05/23 09:04:53 [debug] 1#1: epoll timer: -1

And curl complains:

curl: (35) Unknown SSL protocol error in connection to 127.0.0.1:443 

I get the exact same issue with the same nginx container running on AWS EC2 behind an ELB setup with proxy_protocol enabled.

I tracked this down to this change:
https://github.com/nginx/nginx/commit/12f436718963f8343e38ad6d0e8f7251c95984cd

Especially in my context (I don't know if it's an issue of running in a container), but the socket has no data available, and thus the c->recv() call in ngx_http_request.c line 692 returns NGX_AGAIN which obviously is different than the size value, and the connection is closed.

Here's a patch for a workaround that apparently fixes the issue:

diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c
index 476f039..de1a679 100644
--- a/src/http/ngx_http_request.c
+++ b/src/http/ngx_http_request.c
@@ -688,6 +688,7 @@ ngx_http_ssl_handshake(ngx_event_t *rev)
 
         size = p - buf;
 
+        c->read->available = 1;
         if (c->recv(c, buf, size) != (ssize_t) size) {
             ngx_http_close_connection(c);
             return;

Attachments (1)

ssl-deferred-proxy.patch (993 bytes ) - added by Roman Arutyunyan 3 years ago.

Download all attachments as: .zip

Change History (6)

comment:1 by Roman Arutyunyan, 3 years ago

Thanks for reporting this.
Please try the attached patch.

by Roman Arutyunyan, 3 years ago

Attachment: ssl-deferred-proxy.patch added

in reply to:  1 comment:2 by Brice Figureau, 3 years ago

Replying to arut:

Thanks for reporting this.
Please try the attached patch.

Thanks!

Your patch is indeed fixing the issue (and look more correct than my fix :)

I haven't done any extensive testing, though, so I can't guarantee it is not introducing any regressions.

comment:3 by Roman Arutyunyan <arut@…>, 3 years ago

In 7005:3e2d90073adf/nginx:

Fixed deferred accept with EPOLLRDHUP enabled (ticket #1278).

Previously, the read event of the accepted connection was marked ready, but not
available. This made EPOLLRDHUP-related code (for example, in ngx_unix_recv())
expect more data from the socket, leading to unexpected behavior.

For example, if SSL, PROXY protocol and deferred accept were enabled on a listen
socket, the client connection was aborted due to unexpected return value of
c->recv().

comment:4 by Roman Arutyunyan, 3 years ago

Resolution: fixed
Status: newclosed

comment:5 by Roman Arutyunyan <arut@…>, 2 years ago

In 7137:0e05b35beebf/nginx:

Fixed deferred accept with EPOLLRDHUP enabled (ticket #1278).

Previously, the read event of the accepted connection was marked ready, but not
available. This made EPOLLRDHUP-related code (for example, in ngx_unix_recv())
expect more data from the socket, leading to unexpected behavior.

For example, if SSL, PROXY protocol and deferred accept were enabled on a listen
socket, the client connection was aborted due to unexpected return value of
c->recv().

Note: See TracTickets for help on using tickets.