Opened 7 years ago

Closed 4 years ago

Last modified 4 years ago

#251 closed enhancement (fixed)

Option to disable upload buffers

Reported by: timwhite88@… Owned by: somebody
Priority: minor Milestone:
Component: nginx-core Version: 1.2.x
Keywords: buffer uploaded Cc:
uname -a: Linux vmsrv-lino 3.4.2-x86_64-linode25 #2 SMP Tue Jun 26 16:10:49 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.2.4 TLS SNI support enabled configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6

Description

Partially related to #159, the ability to disable upload buffers both for FastCGI and HTTP proxying would be very beneficial. Using the Upload module isn't always pratical, requiring a rewrite of some application code. And as a static file accelerator in front of another web server, being able to upload large files and not have it buffer would be very useful. In particular, lots of file uploaders break as the make a second request to get progress, only to find out that the file hasn't even "started uploading" as it's still buffering to Nginx.

At the end of the day, if you have to rewrite application code on the web server behind Nginx, then Nginx isn't as useful as a static file server.

Change History (13)

comment:1 Changed 6 years ago by timmyschweer@…

We're using nginx to provide an typo3 page and a lot of javascript / PHP stuff. We started a new service using java servlets contained in tomcat / jetty. The nginx upload buffer messes up our upload servlet on large files so please provide the functionality described in this ticket.

comment:2 Changed 6 years ago by www.google.com/accounts/o8/id?id=AItOawm3VVep_8FRDwgjvCYmY9SQ2qlzsA9-x4Y

Guys, this is a huge issue for us. Every uploaded file needs to be fully buffered to memory or disk prior to sending it to the backend server.
This is both a load & latency issue.
This makes using nginx as a reverse proxy for multiple users simply impractical.
I think this issue's priority is far higher than minor.

Do you have any idea on when this issue is solved?
Thanks.

comment:3 Changed 5 years ago by www.google.com/accounts/o8/id?id=AItOawmfL7vlPPAcCIXg9CJj6Y8hNuxNvPcvsOM

Am I right to believe the upload buffering was put in place to prevent the blocking of workers in traditional backends? Newer nonblocking backends (like, say, node.js) do not require these measures as the requests are always streamed in. It would be great to have the option to disable this feature when using a newer backend that does not require nginx to babysit its ability to stream uploads.

comment:4 Changed 5 years ago by www.google.com/accounts/o8/id?id=AItOawl-mS8UjqHw3Y1-zmwpCktnrvN76kchYM4

We've run into similar issues with this when using nginx as a server for Ceph. The machine running nginx does not necessarily have sufficient disk space to buffer the entire upload, nor is that something we want it to do.

I'm hoping now that we have control of this for downloads (via fastcgi_buffering), that this is something that will be implemented for uploads as well.

I did find a patch that does this on the mailing list, so it seems feasible: http://mailman.nginx.org/pipermail/nginx/2013-January/037092.html . Also, tengine has support for this (via fastcgi_request_buffering)

comment:5 Changed 5 years ago by derhein@…

As explained in the mailing list post, the functionality has been added to
the tengine-fork of nginx.

http://tengine.taobao.org/
https://github.com/alibaba/tengine

Is there any reason not to integrate this feature back into nginx, or
did just nobody put in a pull request, yet?

comment:6 Changed 5 years ago by maxim

  • Status changed from new to accepted

We do have this feature in our roadmap for 1.7.

comment:7 Changed 5 years ago by mcarbonneaux@…

in what 1.7.x while be released ?

comment:8 Changed 5 years ago by mikekennewick@…

Last edited 5 years ago by mikekennewick@… (previous) (diff)

comment:9 Changed 4 years ago by www.google.com/accounts/o8/id?id=AItOawnqMJEdLU46Ojoc98BRoAArf39C8JWMuqw

Any news on this enhancement?
Can someone of the NGINX team say when this enhancement will be deployed?

comment:10 Changed 4 years ago by maxim

ETA is this quarter.

comment:11 Changed 4 years ago by lennartbastian@…

was this removed from the 1.7 roadmap? is it still planned to be released in the next several weeks?

comment:12 Changed 4 years ago by maxim

It wasn't:

http://trac.nginx.org/nginx/milestone/1.7

The feature was implemented in 1.7.11 with some fixes in 1.7.12.

comment:13 Changed 4 years ago by mdounin

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.