#533 closed defect (invalid)
Progressive download (http_mp4): delays before responding with actual data streams
| Reported by: | Maxim Novikov | Owned by: | |
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | nginx-module | Version: | 1.0.x |
| Keywords: | http mp4 pseudo streaming | Cc: | csbubbles@… |
| uname -a: | Linux 2.6.32-358.23.2.el6.x86_64 | ||
| nginx -V: |
nginx version: nginx/1.0.15
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E |
||
Description
I am trying to utilize nginx's hhtp_mp4 module for progressive download. In general it works fine, but there is an issue that looks like a deal breaker to me.
There is a delay after sending the request and before receiving the actual byte stream, and it apparently depends on the size of a file. For example, for a 3mb file this pause is about 0.5 second, whereas for a 75mb file it is 5 seconds. My assumption is that the http_mp4 module analyzes the content of the file before starting streaming it (maybe calculating its length or playing time, etc.), and for big files it may take a while.
All my files are AAC-encoded audio (.mp4 files). What I wanted to actually achieve using this module is a media storage for audio players on different platforms. But you can understand that the behavior described above would be inappropriate for such purposes, as it brings huge delays sometimes when the player is trying to change an audio track.
Is it the expected behavior? Can it be solved enabling some sort of "caching" that would allow the http_mp4 module to pre-scan all the information required in advance and to not spend time on this when processing HTTP requests? Is there some configuration I have missed?
I would appreciate any suggestions how to solve this issue, but my expectations would be to start receiving the audio payload immediately.
PS I am attaching screenshots where you can see the pauses for files of various sizes (a.png - handling a 75mb file, b.png - 3.5mb file).
Attachments (5)
Change History (19)
by , 11 years ago
comment:1 by , 11 years ago
Media files should be prepared for streaming to avoid extra processing overhead.
You probably have missed this part of the documentation:
If the metadata are located at the end of the file, nginx must read the entire file and prepare a new stream so that the metadata come before the media data. This involves some CPU, memory, and disk I/O overhead, so it is a good idea to prepare an original file for pseudo-streaming in advance, rather than having nginx do this on every such request.
comment:2 by , 11 years ago
The thing is that some of the files have been encoded with ffmpeg moving the metadata to the beginning of the file ("-movflags faststart" option), and those work fine. However, not all files can be handled this way in general. My point was that in such cases (basically, in all cases) it would be pretty convenient on the part of http_mp4 module to cache the meta info after having read it from the file the first time, and then reuse it if the file content hasn't changed since that time. This would be the expected behavior from the module, not to rescan all the files for every request (regardless where the metadata is located in the file).
comment:3 by , 11 years ago
There's no need in explicit caching of the metadata since it's cached by the OS page cache. Parsing costs almost nothing. According to your screenshots the second request is always fast, so caching is fine.
follow-up: 6 comment:4 by , 11 years ago
The second screenshot is for a 3.5mb file. The first screenshot is for a file of 75mb size. They are screenshots for different files. And it doesn't matter whether OS caches anything or not, as the problem exists for requests from multiple clients (different users, IPs, machines, OS's, whatever). Especially it matters if these are requests from mobile devices, not from desktop browsers as you probably thought. What I can see is that if I request 75mb file from iOS or Android, even from the same device, this pause of 5-10 seconds for 50-100mb files always exists for the same file. Nothing is cached in such cases. And you really cannot count on such type of caching actually.
comment:5 by , 11 years ago
PS By the way, if the file is encoded the "right" way having its media metadata at the beginning, most existing clients handle the content the right way even without this nginx module. They recognize the meta info and start playing media without delays (and allow you to seek through the file as well) while the web server supports partial content retrieval via "Range" HTTP header and returns the correct content type. And I believe most contemporary web servers do. So, the issue I have raised is most likely the only use of the http_mp4 module at all these days.
comment:6 by , 11 years ago
The second screenshot is for a 3.5mb file. The first screenshot is for a file of 75mb size. They are screenshots for different files. And it doesn't matter whether OS caches anything or not, as the problem exists for requests from multiple clients (different users, IPs, machines, OS's, whatever).
What I mean is server-side caching. Once file is read is stays in server page cache for long time.
The second (last) request on each picture is fine. So only the first request to a file is slow. Is that right?
comment:7 by , 11 years ago
This is just 1 request from the end-user perspective. Chrome just splits it to 2 requests. I assume that Chrome first makes a call to receive the metainfo about the file (and that requests took those 5 seconds), and then starts requesting the actual data making another request.
If you want to reproduce, try to follow these steps:
- Open Chrome
- Open its development console (Network tab)
- Open this URL in Chrome: [removed; if needed, please contact me over email]
You will see how it works. Then clear the browser cache and do it again (that would be the case of non-caching clients).
PS Please note, that the file will be unavailable in a couple of hours, but you can easily reproduce it using your own infrastructure, you don't need my help for this I believe - simply run nginx with the http_mp4 module in its configuration, find/generate a media file >= 50mb, request it via HTTP remotely from another machine.
comment:8 by , 11 years ago
Chrome makes several HTTP byterange requests per file. It looks like the first one can be slow and all following requests are fast. I tested your url and all requests are fast probably because the file is already cached at your server.
comment:9 by , 11 years ago
Again, clear the browser cache (ctrl/cmd-shift-del) and open the URL again. You will see the same picture, there will be the initial delay 5 seconds again. Most likely because the http_mp4 module will read the file again trying to read its meta info from its end. It is reproducible very easily.
by , 11 years ago
comment:10 by , 11 years ago
I have attached more screenshots for better understanding how to reproduce and what happens.
01.png - opening the URL in Chrome first time. Chrome makes 2 requests, 1st in red that brings delay 4.18 seconds, 2nd in green - when the actual audio starts playing.
02.png - cleaning up Chrome's cache (otherwise it will read the file from there next time).
03.png - opening the URL in Chrome second time. The behavior is the same, 1st request takes 4 seconds (red, Chrome doesn't play anything), 2nd request (green) - audio starts playing.
You can easily test it from mobile devices (e.g. using MediaPlayer API on Android), you will get the same, but in that case you will not even need to clear any caches for the following requests for the same file. It always will get you that 4-5 seconds delay.
comment:11 by , 11 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
The fact that response code is 206 indicates that the problem is not mp4 module, as mp4 module in nginx 1.0.15 doesn't allow range requests at all. Rather, it's something with your infrastructure (overloaded disk subsystem? slow network?). I'm closing this ticket as it's clearly invalid.
comment:12 by , 11 years ago
This is totally wrong. You are just closing the ticket without any investigation and completely ignoring the absolutely clear steps I have provided to reproduce the issue.
comment:13 by , 11 years ago
This is not a support desk, this is a bug tracker. It is meant to track bugs in nginx, not to investigate what's wrong with your particular setup.
In this particular case I don't think delays you see are at all related to nginx. Rather, it looks like your system constrains (most likely, network speed) and the way how Chrome works. But if you think there is something to do with nginx, you may ask for help in a place specially designed for this - in the mailing list.
comment:14 by , 11 years ago
If start/end arguments are not provided then you don't actually use mp4 module, but simply download static file (you can disable it and try again). It's obvious that you don't provide those since you have HTTP 206 response which is not supported by mp4 module in your version of nginx.
So the problem is with downloading static file from nginx or with your network infrastructure.
Requesting and retrieving audio stream (75mb file)