Opened 3 years ago

Closed 3 years ago

Last modified 10 months ago

#2339 closed defect (invalid)

nginx fails to process "allow" statements on subsequent keepalive requests

Reported by: ctheune@… Owned by:
Priority: major Milestone:
Component: documentation Version:
Keywords: Cc:
uname -a: Linux myhostname 5.10.88 #1-NixOS SMP Wed Dec 22 08:31:00 UTC 2021 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.20.2
built by gcc 10.3.0 (GCC)
built with OpenSSL 1.1.1l 24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/nix/store/vx5vpdayrx5h2v22yp81nq1svhcn060j-nginx-1.20.2 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-threads --with-pcre-jit --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --pid-path=/var/log/nginx/nginx.pid --http-client-body-temp-path=/var/cache/nginx/client_body --http-proxy-temp-path=/var/cache/nginx/proxy --http-fastcgi-temp-path=/var/cache/nginx/fastcgi --http-uwsgi-temp-path=/var/cache/nginx/uwsgi --http-scgi-temp-path=/var/cache/nginx/scgi --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_image_filter_module --with-http_geoip_module --with-stream_geoip_module --with-file-aio --add-module=/nix/store/6pb7j6kymf3y4xs5blp3g8mwin2j22kk-dav --add-module=/nix/store/pqbx61nmspn0n1smy3g32m56wrq2g9bw-modsecurity-nginx --add-module=/nix/store/y39g23fn8ikzcd1iy3b1bclqwjk2qmxd-moreheaders --add-module=/nix/store/2ysp5ichpccf4lv1wp2qcwz0bmm840f1-rtmp

Description

This is a config that used to work on nginx 1.8.1 but when ported to 1.20.2 we see that every first request works but subsequent requests (in keepalive) fail with 403.

Config that is broken:

server {
    listen [v6address]:port ssl http2 ;
    server_name myhostname ;
    location /.well-known/acme-challenge {
            root /var/lib/acme/acme-challenge;
            auth_basic off;
    }
    ssl_certificate /var/lib/acme/myhostname/fullchain.pem;
    ssl_certificate_key /var/lib/acme/myhostname/key.pem;
    ssl_trusted_certificate /var/lib/acme/myhostname/chain.pem;
    location / {
            proxy_pass http://localhost:8500;
            allow anipv6network/64;
            deny all;
    }
}

Reproduction with curl:

curl -i "https://url" "https://url"

Results in:

HTTP/2 200
server: nginx
date: Mon, 28 Mar 2022 07:57:51 GMT
content-type: application/json
content-length: 30
vary: Accept-Encoding
vary: Accept-Encoding
x-consul-default-acl-policy: deny

"[somejsonresponse]"
HTTP/2 403
server: nginx
date: Mon, 28 Mar 2022 07:57:51 GMT
content-type: text/html; charset=UTF-8
content-length: 146
vary: Accept-Encoding

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

This also happens with Python requests which also does keepalive. A workaround is to set "keepalive_requests = 0" on the affected location. Running multiple individual requests without keepalive does not trigger this problem.

Change History (5)

comment:1 by Maxim Dounin, 3 years ago

Works fine here. Tested with the following config:

http {
    server {
        listen [::]:8443 ssl http2;
        server_name myhostname;

        ssl_certificate test.crt;
        ssl_certificate_key test.key;

        location /.well-known/acme-challenge {
            root /var/lib/acme/acme-challenge;
            auth_basic off;
        }
        location / {
                proxy_pass http://localhost:8500;
                allow ::0/64;
                deny all;
        }
    }

    server {
        listen 8500;
        listen [::]:8500;
        return 200 ok\n;
    }
}

Testing with curl:

$ curl -ki https://[::1]:8443/foo https://[::1]:8443/foo
HTTP/2 200 
server: nginx/1.21.7
date: Mon, 28 Mar 2022 11:27:27 GMT
content-type: text/plain
content-length: 3

ok
HTTP/2 200 
server: nginx/1.21.7
date: Mon, 28 Mar 2022 11:27:27 GMT
content-type: text/plain
content-length: 3

ok

Could you please provide full self-contained configuration which demonstrates the problem? Alternatively, please provide a ​debug log from the requests.

My best guess is that localhost in your case resolves to both ::1 and 127.0.0.1, but only on one of these addresses have the correct backend service listening on port 8500, and something different on the other address, so every second request is rejected. This, however, does not explain why you only observe errors with keepalive connections. Changing proxy_pass http://localhost:8500; to proxy_pass http://127.0.0.1:8500; might be a good test though.

comment:2 by Maxim Dounin, 3 years ago

Resolution: → invalid
Status: new → closed

Feedback timeout. Looks like a configuration and/or backend issue.

comment:3 by ctheune@…, 2 years ago

Sorry for not replying back then.

We got back to this issue on our internal side and we figured that a patch that we apply on top of nginx (​https://github.com/flyingcircusio/nginx/commit/2ad7b63de0391df4c49c887f2929a72658bce329) seems to have changed with respect to its compatibility. We're fixing the patch now. If I remember correctly my colleague noticed that we need to make a copy of an address object at some point where we currently aren't doing that.

I'll post an update with our fixed patch for posterity later.

comment:4 by Maxim Dounin, 2 years ago

Thanks for the details. The patch indeed explains the observed behaviour, as it changes the client address in the connection structure when the variable is used, so the following requests on the same connection will be handled with the corrupted client address.

comment:5 by dpausp@…, 10 months ago

If someone is still interested, here's the current version of our IP anonymization patch which fixes the problem and works with 1.25.3: ​https://github.com/flyingcircusio/nginx/commit/545461198aa972c5681f4329ba3d8d1e5551a144

Note: See TracTickets for help on using tickets.