Nginx not respecting the TTL for the resolver at the Location block with 418 http response

[kianyang@…]

Hi team,
We recently notice an issue with our Nginx configuration where we attempt to perform redirections using the error_page directive to redirect some requests to our legacy application. Following the guide in ​https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ we have implemented the below rules.

user www-data;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

http {

  access_log  /var/log/nginx/access.log  keyvalue;

  server {
    listen       80;
    server_name  localhost;
    # rewrite ^(.*[^/])$ $1/ permanent;
    proxy_buffering off;
    proxy_set_header HOST $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # RULE for /shared-end-point/ that's shared with legacy and new site
    # NOTE: This rule is known to cause issues on HP returning 503 status when ELB IP addresses are updated. 
    location = /shared-end-point/ {
      error_page 418 = @legacy_site;
      recursive_error_pages on;

      if ($request_uri ~ ((legacy_querystring1|legacy_querystring2)=)) {
        return 418;
      }

      proxy_pass_request_headers on;
      resolver 10.90.32.2  valid=10s;
      set $upstream_endpoint aws-ec2-new-site.com;
      proxy_pass http://$upstream_endpoint$request_uri;

      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ######## Homepage
    location = / {
      proxy_pass_request_headers on;
      resolver 10.90.32.2  valid=10s;
      set $upstream_endpoint  aws-ec2-new-site.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ######## Legacy site
    location @legacy_site {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;

      set $upstream_endpoint legacy-site.com;
      proxy_pass https://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }
  }
}

We realized that when our ELB refreshes the IP addresses, the aws-ec2-new-site.com is still resolving to the old IP and does not respect the TTL until we reloaded the configuration. Once we remove the 418 redirection rules, it behaves as expected and resolves the hosts based on the TTL. This is observed via a tcp dump where no request to the amazon resolver. This does not appear to be a consistent behaviour or is this a bug to be addressed in future versions.

[Maxim Dounin]

Most likely, aws-ec2-new-site.com is written explicitly in a proxy_pass somewhere in your configuration, and hence it is resolved while reading a configuration as normal upstream names. If you think this is not the case, please provide unmodified "nginx -T" output with a full configuration you are able to reproduce the problem with.

[kianyang@…]

Hi,
This is our current Nginx confiuration after removing the following redirection logic

      error_page 418 = @legacy_site;
      recursive_error_pages on;

      if ($request_uri ~ ((legacy_querystring1|legacy_querystring2)=)) {
        return 418;
      }

It is still displaying a cached behaviour of the IP addresses. Kindly assist us on this matter as it is causing a lot of issues on our production servers. Thank you very much.

user www-data;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
  worker_connections 1024;
  }

http {

  log_format keyvalue 'ip=$remote_addr user=$remote_user '
  'req="$request" status=$status bytes_sent=$body_bytes_sent '
  'req_time=$request_time ref="$http_referer" '
  'ua="$http_user_agent" forwarded="$http_x_forwarded_for" upstream_addr=$upstream_addr ';

  access_log  /var/log/nginx/access.log  keyvalue;

  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;
  server_tokens off;

  include             /etc/nginx/mime.types;
  default_type        application/octet-stream;
  include              /etc/nginx/conf.d/*.conf;
  index                index.html index.htm;

  gzip on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

  more_set_headers 'Server: Front-end-User-200';
  more_clear_headers X-Powered-By;

  server {
    listen       80;
    server_name  localhost;
    # rewrite ^(.*[^/])$ $1/ permanent;
    proxy_buffering off;
    proxy_set_header HOST $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    #auth_basic "Restricted";
    #auth_basic_user_file /etc/nginx/.htpasswd;

    ############    RULE for any aspx/ashx file extensions
    location ~ ^/.*\.(aspx|ashx) {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint legacy.iproperty.com.my;
      proxy_pass https://$upstream_endpoint;

      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for SRP PDP Specific Proxy Pass
    location ~ /srp-pdp {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    #RULE for only /rent/ # NOTE: This rule is known to cause issues on HP returning 503 status when ELB IP addresses are updated. 
    location = /rent/ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint$request_uri;

      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    #RULE for only /rent/ 301 /rent/
    location = /rent {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint  internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    #RULE for duplicated cities with state exception list
    location ~ /rent/(selangor|kuala-lumpur|johor|penang|perak|negeri-sembilan|pahang|sabah|sarawak|kedah|putrajaya|kelantan|terengganu|perlis|labuan)/(ampang|ayer-hitam|bandar-sri-damansara|cheras|kepala-batas|nilai|puchong|rantau-panjang|rompin|serdang|simpang-ampat|taman-melawati)/(all-residential|apartment-flat|apartment|apartment-duplex|condo-serviced-residence|condominium|duplex|triplex|condo-serviced-residence-soho|service-apartment|serviced-residence|townhouse-condo|penthouse|terrace-link-townhouse|1-sty-terrace-link-house|2-sty-terrace-link-house|3-sty-terrace-link-house|1-5-sty-terrace-link-house|2-5-sty-terrace-link-house|3-5-sty-terrace-link-house|4-sty-terrace-link-house|4-5-sty-terrace-link-house|Townhouse|semi-d-bungalow|cluster-homes|bungalow-house|bungalow-land|link-bungalow|semi-detached-house|twin-villas|twin-courtyard-villa|zero-lot-bungalow|residential-land|all-commercial|shop-office-retail-space|retail-office|retail-space|shop|shop-office|office|sofo|shop-office-retail-space-soho|sovo|commercial-bunglow|commercial-semi-d|designer-suites|business-centre|hotel-resort|commercial-land|all-industrial|factory-warehouse|factory|semi-d-factory|warehouse|detached-factory|light-industrial|link-factory|industrial-land|all-agricultural|agricultural-land)/ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for rent with level 1 location with no property type ##
    location ~ ^.*/(rent|srpstatic)/(selangor|kuala-lumpur|johor|penang|perak|negeri-sembilan|pahang|sabah|sarawak|kedah|putrajaya|kelantan|terengganu|perlis|labuan)/(?!all-residential|apartment-flat|apartment|apartment-duplex|condo-serviced-residence|condominium|duplex|triplex|condo-serviced-residence-soho|service-apartment|serviced-residence|townhouse-condo|penthouse|terrace-link-townhouse|1-sty-terrace-link-house|2-sty-terrace-link-house|3-sty-terrace-link-house|1-5-sty-terrace-link-house|2-5-sty-terrace-link-house|3-5-sty-terrace-link-house|4-sty-terrace-link-house|4-5-sty-terrace-link-house|Townhouse|semi-d-bungalow|cluster-homes|bungalow-house|bungalow-land|link-bungalow|semi-detached-house|twin-villas|twin-courtyard-villa|zero-lot-bungalow|residential-land|all-commercial|shop-office-retail-space|retail-office|retail-space|shop|shop-office|office|sofo|shop-office-retail-space-soho|sovo|commercial-bunglow|commercial-semi-d|designer-suites|business-centre|hotel-resort|commercial-land|all-industrial|factory-warehouse|factory|semi-d-factory|warehouse|detached-factory|light-industrial|link-factory|industrial-land|all-agricultural|agricultural-land) {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint legacy.iproperty.com.my;
      proxy_pass https://$upstream_endpoint;

      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for rent with level 1 property type ##
    location ~ ^.*/(rent|srpstatic)/(residential|commercial|industrial|agricultural)/ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint legacy.iproperty.com.my;
      proxy_pass https://$upstream_endpoint;

      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for rent/list to return all residential listing ##
    location ~ /rent/list.*$ {
      return 301 https://$host/rent/all-residential/$is_args$args;
    }

    ############    RULE for rent with location level 1 with no property type ##
    location ~ /(rent|sale)\/((?!(all\-residential|apartment\-flat|apartment|apartment\-duplex|condo\-serviced\-residence|condominium|duplex|triplex|condo\-serviced\-residence\-soho|service\-apartment|serviced\-residence|townhouse\-condo|penthouse|terrace\-link\-townhouse|1\-sty\-terrace\-link\-house|2\-sty\-terrace\-link\-house|3\-sty\-terrace\-link\-house|1\-5\-sty\-terrace\-link\-house|2\-5\-sty\-terrace\-link\-house|3\-5\-sty\-terrace\-link\-house|4\-sty\-terrace\-link\-house|4\-5\-sty\-terrace\-link\-house|Townhouse|semi\-d\-bungalow|cluster\-homes|bungalow\-house|bungalow\-land|link\-bungalow|semi\-detached\-house|twin\-villas|twin\-courtyard\-villa|zero\-lot\-bungalow|residential\-land|all\-commercial|shop\-office\-retail\-space|retail\-office|retail\-space|shop|shop\-office|office|sofo|shop\-office\-retail\-space\-soho|sovo|commercial\-bunglow|commercial\-semi\-d|designer\-suites|business\-centre|hotel\-resort|commercial\-land|all\-industrial|factory\-warehouse|factory|semi\-d\-factory|warehouse|detached\-factory|light\-industrial|link\-factory|industrial\-land|all\-agricultural|agricultural\-land)).)*$ {
      return 301 https://$host$uri/all-residential/$is_args$args;
    }

    ############    RULE for rent##
    location ~ ^.*/(rent|srpstatic)/.+$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for /sale/ with no property type ##
    location = /sale/ {
      return 301 https://$host/sale/all-residential/$is_args$args;
    }

    ############    RULE for /sale ##
    location = /sale {
      return 301 http://$host$uri/$is_args$args ;
    }

    ############    RULE for sale##
    location ~ ^.*/sale.*$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############  Legacy property Specific rule ##
    location ~ ^.*/property/([A-Za-z0-9-+()_%'@])+$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass https://legacy.iproperty.com.my;
        if ($http_x_forwarded_proto != 'https') {
          return 301 https://$host$request_uri;
        }
    }

    ############  Legacy used property rule ##
    location ~ /property/usedproperty/.*$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass https://legacy.iproperty.com.my;
        if ($http_x_forwarded_proto != 'https') {
          return 301 https://$host$request_uri;
        }
    }

    ############  Legacy propertylisting rule ##
    location ~ ^/(propertylisting|propertynearby)/.+$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass https://legacy.iproperty.com.my;
        if ($http_x_forwarded_proto != 'https') {
          return 301 https://$host$request_uri;
        }
    }

    ############    RULE for property  Regionalweb  #######
    location ~ ^/(property|review).*$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ############    RULE for HP
    location ~ ^.*/hp.*$ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint  internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    #RULE for only /terms-of-use/
    location = /terms-of-use/ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint  internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ########### MY legacy URL redirect
    location  ~*  /.* {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass https://legacy.iproperty.com.my;
        if ($http_x_forwarded_proto != 'https') {
          return 301 https://$host$request_uri;
        }
    }

    ######## Main / pointing
    location = / {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint  internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }

    ######   For newrelic-only
    location = /nginx_stub_status {
      stub_status on;
      access_log off;
      allow 127.0.0.1;
      deny all;
    }

    ######   For AWS Monitoring avoid log and file
    location = /test.html {
      access_log off;
      return 200;
    }

    ######   Diagnostic Endpoint
    location = /diagnostic/version {
      access_log off;
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      set $upstream_endpoint  internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      proxy_pass http://$upstream_endpoint;
        if ($http_x_forwarded_proto != 'https') {
          return 301 https://$host$request_uri;
        }
    }

    ########### .well-known folder and robots.txt
    location ^~ /.well-known/ {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass http://internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }
    location = /robots.txt {
      proxy_pass_request_headers on;
      resolver 10.90.32.2 valid=10s;
      proxy_pass http://internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
      if ($http_x_forwarded_proto != 'https') {
        return 301 https://$host$request_uri;
      }
    }
  }
}

[Maxim Dounin]

With the provided configuration, the following address is only used in proxy_pass via variables, and hence will be resolved at runtime:

• internal-Load-test-ECSSrpPd-YBD9A6DBL9Q0-27275377.ap-southeast-1.elb.amazonaws.com.

The following address are used explicitly in proxy_pass, and hence they will be resolved during configuration parsing:

• internal-Load-test-EcsHpELB-1UU1MZIHWQJ7P-1365473474.ap-southeast-1.elb.amazonaws.com;
• legacy.iproperty.com.my.

Given that you also try to use these addresses via variables, you are probably complaining about one of these addresses. And, as suggested in comment:1, the problem is that you use these addresses explicitly in proxy_pass, and hence they are resolved during configuration parsing.

If you need further help, please use ​exiting support options.

[kianyang@…]

Thank you for your prompt response!