Opened 8 years ago

Closed 6 years ago

Last modified 5 years ago

#251 closed enhancement (fixed)

Option to disable upload buffers

Reported by: Timothy White Owned by: somebody
Priority: minor Milestone:
Component: nginx-core Version: 1.2.x
Keywords: buffer uploaded Cc:
uname -a: Linux vmsrv-lino 3.4.2-x86_64-linode25 #2 SMP Tue Jun 26 16:10:49 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.2.4
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6

Description

Partially related to #159, the ability to disable upload buffers both for FastCGI and HTTP proxying would be very beneficial. Using the Upload module isn't always pratical, requiring a rewrite of some application code. And as a static file accelerator in front of another web server, being able to upload large files and not have it buffer would be very useful. In particular, lots of file uploaders break as the make a second request to get progress, only to find out that the file hasn't even "started uploading" as it's still buffering to Nginx.

At the end of the day, if you have to rewrite application code on the web server behind Nginx, then Nginx isn't as useful as a static file server.

Change History (13)

comment:1 by Timmy Schweer, 7 years ago

We're using nginx to provide an typo3 page and a lot of javascript / PHP stuff. We started a new service using java servlets contained in tomcat / jetty. The nginx upload buffer messes up our upload servlet on large files so please provide the functionality described in this ticket.

comment:2 by Roy Reznik, 7 years ago

Guys, this is a huge issue for us. Every uploaded file needs to be fully buffered to memory or disk prior to sending it to the backend server.
This is both a load & latency issue.
This makes using nginx as a reverse proxy for multiple users simply impractical.
I think this issue's priority is far higher than minor.

Do you have any idea on when this issue is solved?
Thanks.

comment:3 by Luke Plaster, 7 years ago

Am I right to believe the upload buffering was put in place to prevent the blocking of workers in traditional backends? Newer nonblocking backends (like, say, node.js) do not require these measures as the requests are always streamed in. It would be great to have the option to disable this feature when using a newer backend that does not require nginx to babysit its ability to stream uploads.

comment:4 by Brian Rak, 6 years ago

We've run into similar issues with this when using nginx as a server for Ceph. The machine running nginx does not necessarily have sufficient disk space to buffer the entire upload, nor is that something we want it to do.

I'm hoping now that we have control of this for downloads (via fastcgi_buffering), that this is something that will be implemented for uploads as well.

I did find a patch that does this on the mailing list, so it seems feasible: http://mailman.nginx.org/pipermail/nginx/2013-January/037092.html . Also, tengine has support for this (via fastcgi_request_buffering)

comment:5 by Heinrich Hartmann, 6 years ago

As explained in the mailing list post, the functionality has been added to
the tengine-fork of nginx.

http://tengine.taobao.org/
https://github.com/alibaba/tengine

Is there any reason not to integrate this feature back into nginx, or
did just nobody put in a pull request, yet?

comment:6 by maxim, 6 years ago

Status: newaccepted

We do have this feature in our roadmap for 1.7.

comment:7 by Mathieu CARBONNEAUX, 6 years ago

in what 1.7.x while be released ?

comment:8 by mikekennewick@…, 6 years ago

Last edited 6 years ago by mikekennewick@… (previous) (diff)

comment:9 by SkarothNET ., 6 years ago

Any news on this enhancement?
Can someone of the NGINX team say when this enhancement will be deployed?

comment:10 by maxim, 6 years ago

ETA is this quarter.

comment:11 by len bas, 6 years ago

was this removed from the 1.7 roadmap? is it still planned to be released in the next several weeks?

comment:12 by maxim, 6 years ago

It wasn't:

http://trac.nginx.org/nginx/milestone/1.7

The feature was implemented in 1.7.11 with some fixes in 1.7.12.

comment:13 by Maxim Dounin, 6 years ago

Resolution: fixed
Status: acceptedclosed
Note: See TracTickets for help on using tickets.