Get only 200 responses when "if_modified_since" is set to "off"

[rjorel@…]

Hello,

These last days, I tried to understand how web servers like Nginx deal with cache control of resources like JS, CSS or font files. So I learned some things about If-Modified-Since and If-None-Match headers that allows to determine if a resource must be re-sent or not.

I thought that reading source code should be the best solution to understand how etags are generated and how Nginx uses them to generate 304 responses. But I was surprised to read that in src/http/modules/ngx_http_not_modified_filter_module.c:

        if (r->headers_in.if_modified_since
            && ngx_http_test_if_modified(r))
        {
            return ngx_http_next_header_filter(r);
        }

        if (r->headers_in.if_none_match
            && !ngx_http_test_if_match(r, r->headers_in.if_none_match, 1))
        {
            return ngx_http_next_header_filter(r);
        }

whereas I see on MDN web docs and RFC 7232 that If-None-Match takes precedence over If-Modified-Since if web server supports it.

I supposed it's an optimization because etags seems to be generated from last modified times (if I well understood ngx_http_set_etag function in src/http/ngx_http_core_module.c), so checking last modified times before etags is almost the same thing.

So, to be sure that resource validation via etags was done after last modified times, I deactivate If-Modified-Since check with directive if_modified_since off;. After that, I no longer got any 304 responses but only 200 ones. I thought that etag check didn't work, but reading ngx_http_test_if_modified function gave me a clue:

static ngx_uint_t
ngx_http_test_if_modified(ngx_http_request_t *r)
{
    time_t                     ims;
    ngx_http_core_loc_conf_t  *clcf;

    if (r->headers_out.last_modified_time == (time_t) -1) {
        return 1;
    }

    clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);

    if (clcf->if_modified_since == NGX_HTTP_IMS_OFF) {
        return 1;
    }

    // ...

By default, Nginx sends Last-Modified header, so browsers send back If-Modified-Since one. But as I set to off the check, I think ngx_http_test_if_modified stops at the 2e if.

To be sure, I removed If-Modified-Since header using cURL tool, and I also found how to remove Last-Modified header from response. I correctly got 304 responses using these 2 methods separately.

I suppose you're aware of that, but as I cannot be sure I think it's a good idea to report the "issue" to you.

I just have two questions to be sure I well understood:

1. Am I correct in thinking that the check order is an optimization because etags are generated from timestamps (I suppose checking timestamps is faster than checking etags) ?

2. Is this behavior when deactivating If-Modified-Since check normal? Because thinking to remove Last-Modified header from response is not obvious to guess here, if we want to keep etag check only.

Regards.
Raphael Jorel

[Maxim Dounin]

1. Am I correct in thinking that the check order is an optimization because etags are generated from timestamps (I suppose checking timestamps is faster than checking etags) ?

No. As per HTTP 2616, both validators have to be checked, and this is what nginx does. See previous discussions ​here and ​here.

2. Is this behavior when deactivating If-Modified-Since check normal? Because thinking to remove Last-Modified header from response is not obvious to guess here, if we want to keep etag check only.

Current behaviour seems to be incorrect, thanks for reporting this. The if_modified_since off; setting should either not affect If-None-Match processing (assuming "off" means "the “If-Modified-Since” request header field is ignored", as ​documentation says), or should disable If-None-Match regardless of the If-Modified-Since header presence (assuming "off" instead means "never return 304").

[rjorel@…]

Thank you for bringing me some clarifications :-)

[Maxim Dounin]

Looking into this once again, I tend to think that the current behaviour is actually meaningful: as long as the If-Modified-Since header is present, comparison always fails, and the response is considered modified. This is in line with other if_modified_since possible options, which is documented to specify "how to compare modification time".

It might make sense to better document the current behaviour. Existing documentation was correct when it was introduced, yet no longer correct with introduction of entity tags and If-None-Match handling.

Patch is here:

​https://mailman.nginx.org/pipermail/nginx-devel/2023-January/2S6RLEC5MZNDC5CWLEGQGR224LMCHZ5W.html

Review appreciated.

[Maxim Dounin]

Fixed in [52ea1f45b324/nginx_org], thanks to all involved.