Opened 3 years ago
Closed 3 years ago
#2326 closed defect (invalid)
One server directive, and logging stops to function after EKS upgrade in AWS
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | http/3 | Version: | |
| Keywords: | Cc: | ||
| uname -a: | Linux ip-10-96-147-39.eu-central-1.compute.internal 4.14.231-173.361.amzn2.x86_64 #1 SMP Mon Apr 26 20:57:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.20.0
built by gcc 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC) built with OpenSSL 1.1.1g FIPS 21 Apr 2020 TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' |
||
Description
Hi!
We have a specific use case. We use nginx as reverse proxy between corporate network and AWS Private cloud. There is confiuration for some domains pointing to AWS Load balancer. And one configuration pointing to AWS EKS API endpoint. We upgraded EKS Control Plane from 1.19 to 1.20 and then we lost server that is listening on one domain, proxying to end point. End point URL was the same. We tried from EC2 instance to curl the endpoint and it works. Curling url where nginx is listening to produced timeout. After restart nginx worked again. We have 2 EC2 instances, and 2 times nginx installed and both acted the same way. And also when server did not respond there was no logging for any of about 10 domains we use nginx as proxy. But other domains served what they should, but no access or error log was generated. After just "service nginx restart" everything went fine.
Then we though this was on one cluster, updated second with the same setup, and the same, 2 rev proxies stopped function for API URL, and no logging. Other domains function okay. Same situation. Restart helped.
This is server that did not work
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name SOMEURL;
include error.conf;
include fallback-ssl.conf;
include proxy.conf;
proxy_no_cache $cookie_nocache $arg_nocache;
proxy_no_cache $http_pragma $http_authorization;
proxy_cache_bypass $cookie_nocache $arg_nocache ;
proxy_cache_bypass $http_pragma $http_authorization;
underscores_in_headers on;
location / {
proxy_pass SOMEOTHERURL
proxy_ssl_session_reuse on;
}
}
And one that worked:
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name APPURL;
ssl_certificate /etc/nginx/ssl.d/ssc20-poc-alb.crt;
ssl_certificate_key /etc/nginx/ssl.d/ssc20-poc-alb.key;
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
ssl_session_tickets off;
include proxy.conf;
include error.conf;
location / {
proxy_pass APPOTHERURL;
proxy_ssl_session_reuse on;
proxy_intercept_errors on;
}
}
Can You please chek up.
Greets
Domagoj
It looks like the IP addresses of the backend service you are using were changed, hence things stopped working.
Note that nginx by default only resolves hostnames during configuration parsing, and if you change IP addresses of backend services you have to reload nginx. To avoid service interruption, usually this is something to be done when a backend service is already responding on the new IP addresses yet still available on the old IP addresses.
Alternatively, you can configure nginx to use resolver for run-time name resolution, by using proxy_pass with variables. This, however, implies some performance costs due to DNS lookups, and also makes nginx dependant on correctly operating DNS server, so usually something to avoid on loaded systems.
Hope this helps. If you have further questions on how to configure nginx, please use support options available.