proxy_next_upstream broken since 1.9.13 including stabe 1.10.1

[areznikov-kabam@…]

Hello.
It looks like I found some issue with proxy_next_upstream.
I am running the following configuration:

log_format timed_combined '$remote_addr - $remote_user [$time_local] '

'"$request" $upstream_addr http_code:$status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $pipe';

gzip on;

gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

tcp_nodelay on;
sendfile on;
tcp_nopush on;

server {

listen 8080;

location / {

proxy_buffering off;
proxy_pass ​http://api_upstream;
proxy_set_header Host $host;
proxy_next_upstream http_503 http_502 timeout error invalid_header;
proxy_next_upstream_tries 3;

}

location /nginx_status {

stub_status on;
access_log off;
allow 127.0.0.1;
deny all;

}

which works fine for nginx versions 1.9.12 and below.

Upstream has about 40 nodejs workers and nginx just proxies requests to it and does gzip compression.

I tried to update to 1.10.1 and discovered issue with proxy_next_upstream when it was sending 502 errors to the ELB without trying to use other upstream workers.

172.31.33.24 - - [03/Jun/2016:21:13:05 +0000] "POST /*dsdsdsd*" HTTP/1.1" 127.0.0.1:8091 http_code:502 173 "-" "DDDDDDDD/1.0" 0.001 0.001 .
172.31.16.204 - - [03/Jun/2016:21:13:06 +0000] "POST*sdsdsdsdsdsd" HTTP/1.1" 127.0.0.1:8091 http_code:502 173 "-" "DDDDDDDDDD/1.0" 0.001 0.001 .

In versions below 1.9.13 it works correctly
172.31.25.39 - - [03/Jun/2016:23:24:53 +0000] "GET /rrrrsdsdsdrr HTTP/1.1" 127.0.0.1:8085, 127.0.0.1:8083 http_code:200 19944 "-" "DDDD/1.0" 1.211 1.196, 0.015 .
172.31.33.24 - - [03/Jun/2016:23:24:53 +0000] "POST /sdsdsds HTTP/1.1" 127.0.0.1:8085, 127.0.0.1:8102 http_code:200 166 "-" "SDDDDDD" 0.882 0.864, 0.018 .
172.31.25.39 - - [03/Jun/2016:23:24:53 +0000] "POST /dddddr HTTP/1.1" 127.0.0.1:8085, 127.0.0.1:8091 http_code:200 1563 "-" "DDDD/1.0" 0.150 0.001, 0.149 .

[Ruslan Ermilov]

When updating, it's a good idea to read the changes.

Changes with nginx 1.9.13 29 Mar 2016

*) Change: non-idempotent requests (POST, LOCK, PATCH) are no longer

passed to the next server by default if a request has been sent to a
backend; the "non_idempotent" parameter of the "proxy_next_upstream"
directive explicitly allows retrying such requests.

​http://nginx.org/en/docs/http/ngx_http_proxy_module.html#non_idempotent