Opened 6 years ago
Closed 6 years ago
#1309 closed defect (invalid)
Chunked transfer encoding with Live video stream does not clean tmp files
|Reported by:||Owned by:|
|uname -a:||Linux qa.unified-streaming.com 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux|
nginx version: nginx/1.12.0
configure arguments: --sbin-path=/usr/local/sbin --add-dynamic-module=/var/www/test/nginx_mod_http_catch_body_filter
When using 'chunked transfer encoding' Nginx writes the POSTed content to a tmp file (or files).
This will fill up the disk with a large file, but also when a 24/7 Live video stream is sent using chunked transfer encoding. It basically is a a long standing POST where chunks of data are sent within that POST, so the connection is not closed until the stream stops and as it is a 24/7 live linear stream this equals to 'never'.
Nginx therefore cannot be used with a Live when the ingest is handled with a 'request body filter'.
I attach the reference module that shows this behaviour as well as a curl line to do the post.
2017/07/05 12:57:33 [warn] 25407#0: *4 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000002, client: 127.0.0.1, server: localhost, request: "POST /large-file-pretending-to-be-video.ismv HTTP/1.1", host: "127.0.0.1:8080"
2017/07/05 12:57:33 [crit] 25407#0: *4 pwrite() "/usr/local/nginx/client_body_temp/0000000002" failed (28: No space left on device), client: 127.0.0.1, server: localhost, request: "POST /large-file-pretending-to-be-video.ismv HTTP/1.1", host: "127.0.0.1:8080"
Change History (3)
by , 6 years ago
comment:1 by , 6 years ago
Please note that the following line:
make sure this flag is up, otherwise we may be missing the last part of the body
r->request_body_in_file_only = 1;
does not matter.
When left out of ngx_http_catch_body_filter_module.c the behaviour is still the same as outlined above.
Is there any combination of settings from ngx_http_request.h as in:
or from the 'client_body_...' configuration as in:
that will have effect on the tmp file writing so that
a) the tmp file does not grow infinitely appending posted data
b) remove created tmp files while the POST is running
As a question, why is a tmp file use any way?
The request body filter should handle incoming data and it should decide whether or not to use a tmp file
comment:2 by , 6 years ago
|Status:||new → closed|
By default, the request body is fully read by nginx if it is needed for further processing. As long as a request body fits into
client_body_buffer_size, it is stored in memory. If the buffer size is not enough (or writing to a file is explicitly requested for some reason, as in your example), the body is written to a temporary file under the
client_body_temp_path directory. The maximum size of the body, and accordingly the maximum size of the temporary file, can be controlled using the
The approach of reading the whole body before doing anything allows nginx to keep working with requests simple regardless of whether they have bodies or not, including ability to freely retry requests to different upstream servers and so on. It also makes sure that backend processes are not occupied when the body is not yet arrived from the client, and we are just waiting for the body - much like when nginx buffers a response from a backend server to free the process as fast as possible.
If you don't want the body to be fully read before processing, it is possible to do so when supported by appropriate modules and underlying protocols. Notably, the proxy module has the proxy_request_buffering directive to pass the request body to an upstream once it arrives, without trying to fully read it. Note though that this is a special mode and it implies various limitations, including:
- Needs either body size known in advance, or a protocol which supports unspecified body size. Notably, this means that chunked requests will only work with
- Various limitations on error handling and redirections: the request cannot be passed to a different server.
If you have further question, please don't hesitate to use existing support options.
A zip file containing module (src, makefile and config) as well as nginx.conf and curl script