Avoid writing duplicate temporary cache files

[Wai Keen Woon]

When a file isn't cached and two concurrent downloads are made for it, we get this behavior:

1) Nginx writes twice, to two temporary files. This duplicates disk writes for the same data.
2) Ignore_client_abort is force-enabled when caching, so nginx must download both files to completion even if the clients abort. This can waste bandwidth, diskspace, and disk writes.

This patch makes nginx write to a temporary file only for the first request; subsequent concurrent requests are proxied without writing.

diff --git a/src/http/ngx_http_file_cache.c b/src/http/ngx_http_file_cache.c
index c92de34..be61aa5 100644
--- a/src/http/ngx_http_file_cache.c
+++ b/src/http/ngx_http_file_cache.c
@@ -289,6 +289,10 @@ ngx_http_file_cache_open(ngx_http_request_t *r)
             return c->error;
         }
 
+        if (!c->exists && c->node->updating) {
+            return NGX_HTTP_CACHE_SCARCE;
+        }
+
         c->temp_file = 1;
         test = c->exists ? 1 : 0;
         rv = NGX_DECLINED;
@@ -381,10 +385,6 @@ ngx_http_file_cache_lock(ngx_http_request_t *r, ngx_http_cache_t *c)
     ngx_msec_t                 now, timer;
     ngx_http_file_cache_t     *cache;
 
-    if (!c->lock) {
-        return NGX_DECLINED;
-    }
-
     cache = c->file_cache;
 
     ngx_shmtx_lock(&cache->shpool->mutex);

[Alexis]

Is there a way to not proxy? Instead the subsequent concurrent requests would serve the same tmp file created by the first request.

This avoids overloading the original server.

[Maxim Dounin]

There is ​proxy_cache_lock to avoid requests to an upstream, as well as ​proxy_cache_use_stale updating.

And, while here. I don't strongly object functionality requested in this enhancement, but the patch suggested looks like a dirty hack and will break other cases. I'll close this particular ticket as "wontfix" as we certainly aren't going to integrate the patch.

Please also see ​Contributing Changes for a proper way to submit patches.