Nginx segmentation fault on uploading file via DAV after Expect: 100-continue

[Nelson Benítez León]

Hi, I can reliably reproduce a crash in nginx when uploading a file via default DAV module, the client program which triggers this behaviour is Cyberduck ( ​http://cyberduck.ch/ ) which is an open source ftp/dav client available for windows and mac os x.

I've investigated the issue by installing a custom build of nginx on fedora and printf debugging, this is what I've gather:

• Cyberduck issues a PUT command with Expect: 100-continue header. See complete request below:

PUT /davsite/directorio/index.html HTTP/1.1
Expect: 100-continue
Content-Length: 6359
Content-Type: text/html
Host: 10.47.128.91
Connection: Keep-Alive
User-Agent: Cyberduck/4.2.1 (9350) (Windows XP/5.1) (x86)

• Above request reaches ngx_http_test_expect() function in ​http://tinyurl.com/dy6ton7 which sends to the client a "100 Continue" status.
• Then cyberduck after receiving the "100 continue" from nginx, it sends the file, nginx crashes after receiving the file, following this trace:

post_handler(r); is executed on ​http://tinyurl.com/ck5tdox because r->request_body is TRUE, this is in ngx_http_read_client_request_body() of ngx_request_body.c

Above post_handler(r) is in turn a call to ngx_http_dav_put_handler() in ngx_http_dav_module.c , link ​http://tinyurl.com/cuygo7d , at the start of that function you have:

ngx_str_t                *temp;
...
temp = &r->request_body->temp_file->file.name;

and above line is the problem, because a little down on that function temp is passed to ngx_ext_rename_file() here:

if (ngx_ext_rename_file(temp, &path, &ext) != NGX_OK) {
        ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
}

and then nginx crashes when trying to access temp->data inside ngx_ext_rename_file() , which the first of that cases is

if (ngx_change_file_access(src->data, ext->access) == NGX_FILE_ERROR) {
            ngx_log_error(NGX_LOG_CRIT, ext->log, ngx_errno,
                          ngx_change_file_access_n " \"%s\" failed", src->data);
            err = 0;
            goto failed;
        }

If you need some more debugging I can try patches in my nginx build, but the bug is easily reproducible by anyone downloading and installing cyberduck.

My dav site nginx.conf below:

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    client_max_body_size 50m;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        listen       10.47.128.91:80;
        server_name  localhost;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        location ^~ /davsite/ {

  root /usr/share/nginx;
  error_log /usr/share/nginx/davsite/error_log warn;

  dav_methods PUT DELETE MKCOL;
  #dav_access group:rw all:r;
  create_full_put_path on;

  autoindex on;
  autoindex_exact_size off;

  # Variables necessary for proper execution of the PHP script used below.
  include /etc/nginx/fastcgi_params;
  fastcgi_param SCRIPT_FILENAME /etc/nginx/webdav-extensions.php;
  fastcgi_param DEPTH $http_depth;
  fastcgi_param HOST $host;
  fastcgi_param DESTINATION $http_destination;
  fastcgi_param OVERWRITE $http_overwrite;
  fastcgi_param REQUEST_METHOD $request_method;

    # NGINX WebDAV support is incomplete and somewhat too strict. We handle
    # a few WebDAV methods manually in a PHP script to fill out the cracks.
    if ($request_method ~ ^(PROPFIND|OPTIONS|COPY|MOVE)$) {
      fastcgi_pass 127.0.0.1:9000;
      break;
    }

    # NGINX WebDAV requires trailing slashes on directories, yet certain
    # common WebDAV clients don't support them. Do rewrites to fix it,
    if (-d $request_filename) { rewrite ^(.*[^/])$ $1/ break; }
    if ($request_method = MKCOL) { rewrite ^(.*[^/])$ $1/ break; }

  }
}
}

A not very useful GDB backtrace, could try to provide one with more symbols but it will show my above analysis anyway:

Reading symbols from /usr/sbin/nginx...(no debugging symbols found)...done.
[New LWP 9141]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `nginx: worker pr'.
Program terminated with signal 11, Segmentation fault.
#0  0x0806043b in ngx_ext_rename_file ()
Missing separate debuginfos, use: debuginfo-install nginx-1.2.5-1.fc18.i386
(gdb) bt
#0  0x0806043b in ngx_ext_rename_file ()
#1  0x080c382d in ngx_http_dav_put_handler ()
#2  0x0809a23b in ngx_http_read_client_request_body ()
#3  0x080c33e8 in ngx_http_dav_handler ()
#4  0x08086a9b in ngx_http_core_content_phase ()
#5  0x08085c31 in ngx_http_core_run_phases ()
#6  0x08091b37 in ngx_http_request_handler ()
#7  0x0807d08f in ngx_epoll_process_events ()
#8  0x08071d09 in ngx_process_events_and_timers ()
#9  0x0807b34d in ngx_worker_process_cycle ()
#10 0x08078a76 in ngx_spawn_process ()
#11 0x0807a88e in ngx_start_worker_processes ()
#12 0x0807a133 in ngx_master_process_cycle ()
#13 0x08054e93 in main ()

[Maxim Dounin]

Looks like another variant to trigger segfault reported in ticket #238 (already fixed). See also ticket #284, which has some additional info (in short: problem is completely fixed in 1.3.9+, and there is no segfault in 1.2.6+).

[Nelson Benítez León]

Thanks for the quick response! :-).

Can I ask if the real fix on #284 can be backported to the stable 1.2.x branch? because otherwise a reliable dav server cannot be run on a stable nginx, and there is no server distros(rhel,debian 6,etc) which are packaging 1.3.x versions..

[Maxim Dounin]

Real fix isn't backported due to concerns that it might break some 3rd party modules.