id summary reporter owner description type status priority milestone component version resolution keywords cc uname nginx_version 1750 Nginx + sendfile serves wrong content from NFS share, may leak confidential information to unauthorized party. imjosi@… "Nginx + sendfile serves wrong content from NFS share, may leak confidential information to unauthorized party. We have a NFS share attached to Nginx server, and recently noticed that under heavy NFS server load Nginx logs an I/O error and then serves wrong content in next request. Actual files on NFS share: {{{ # ls -li /mnt/net/ktn/TML/data/{2,5}0041{7,8}.pdf 218685 -rw-r--r-- 1 root root 200417 Mar 21 13:05 /mnt/net/ktn/TML/data/200417.pdf 218687 -rw-r--r-- 1 root root 200418 Mar 21 13:05 /mnt/net/ktn/TML/data/200418.pdf 218686 -rw-r--r-- 1 root root 500417 Mar 21 13:05 /mnt/net/ktn/TML/data/500417.pdf }}} You can see that Nginx aborted the transfer for file 500417.pdf and then returned 500417 bytes in next request for file 200418.pdf when actual size must be 200418 bytes. access.log {{{ 10.10.248.21 - - [22/Mar/2019:10:32:05 +0200] ""GET /TML/data/200417.pdf HTTP/1.1"" 200 200417 ""-"" ""curl/7.52.1"" ""-"" 0.000 10.10.248.21 - - [22/Mar/2019:10:32:09 +0200] ""GET /TML/data/500417.pdf HTTP/1.1"" 500 537 ""-"" ""curl/7.52.1"" ""-"" 0.000 10.10.248.21 - - [22/Mar/2019:10:32:10 +0200] ""GET /TML/data/200418.pdf HTTP/1.1"" 200 500417 ""-"" ""curl/7.52.1"" ""-"" 0.000 }}} error.log {{{ 2019/03/22 10:32:09 [crit] 639#639: *223 open() ""/mnt/net/ktn/TML/data/500417.pdf"" failed (5: Input/output error), client: 10.10.248.21, server: _, request: ""GET /TML/data/500417.pdf HTTP/1.1"", host: ""10.10.248.21:8080"" }}} After comparing md5 cheksums we confirmed that in request for file 200418.pdf Nginx returned the contents of file 500417.pdf We were able to reproduce this error numerous times using default config for 1.14.2-1~stretch package from nginx.org on both Debian 8 x32 and Debian 9 x64, using Debian 9 and Solaris 11 as a NFS servers. Next we found that putting ""sendfile off"" for this location changes this behaviour and Nginx starts to return incomplete files instead of wrong ones. It is not clear if this problem is caused by a Nginx or underlaying NFS client, but at least it must be clearly stated in the Nginx documentation that using ""sendfile"" for NFS mounts is dangerous and may leak confidential information to unauthorized party. NFS share mount parameters {{{ xet6:/ktn/TML on /mnt/net/ktn/TML type nfs4 (ro,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,soft,noresvport,proto=tcp,port=0,timeo=10,retrans=2,sec=sys,clientaddr=10.10.248.21,local_lock=none,addr=10.10.248.18) }}} Site config in Nginx: {{{ # cat /etc/nginx/conf.d/56-TML.conf server { listen 8080 default_server; server_name _; proxy_no_cache now; proxy_cache_bypass now; location / { root /mnt/net/ktn/ ; } # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } } }}} " defect closed critical nginx-core 1.14.x invalid Linux xet7 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux "nginx version: nginx/1.14.2 built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) built with OpenSSL 1.1.0f 25 May 2017 (running with OpenSSL 1.1.0j 20 Nov 2018) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.14.2/debian/debuild-base/nginx-1.14.2=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'"