when nginx proxy to tomcat, sometimes nginx changes method(POST -> 01POST)

[jangys9510@…]

Hello.
I have a problem that nginx returns 501 error or 200 OK for same requests.
They are same url, parameters and hosts.
But sometimes nginx returns 501 error.
I use nginx 1.16.0 as the web server, tomcat 7.0.54 and java spring.

I tried to figure out why it happens.
First, I looked at nginx and tomcat's access log.

example)
nginx access.log
10.33.xxx.xxx - [25/Oct/2019:16:21:19 +0900] "POST /ajax/test?ts=1571988079124 HTTP/1.1" [501] 1147 0.006 "​https://test.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

tomcat access.log
127.0.0.1 - - [25/Oct/2019:16:21:19 +0900] "01POST /ajax/test?ts=1571988079124 HTTP/1.1" 501 1147 "​https://test.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

As you can see, client requested POST but tomcat received 01POST.
So I tried to set proxy_method to request method like this.

nginx.conf
proxy_method $request_method;

But it doesn't work. Tomcat still receives strange method. (01POST, 01GET, 15GET, 7DPOST..)

Last, I looked at tcpdump by the wireshark.

Before tomcat returns 501 error, Continuation packet sends strange data.

There are Reassembled TCP segments in the continuation packet.
There are two different frame in treassembled TCP segments.
[Frame:67781, payload: 0-1 (2bytes)]
[Frame: 67833, payload:2-191 (190bytes)]
(Frame 67833 is the packet which returns 501 error)
First payload is "01" and second payload is "GET ~" (http header, body contents)
I think this is why nginx returns 501 error.

I looked tcpdump of other 501 responses, I found out it makes invalid continuation packet because of specific response. According to the example,Frame 67781's response is the specific response.(67783)

But still I don't know why it makes method weird. When I use apache, it doesn't return 501 error.

I will try to downgrade nginx 1.16.0 to nginx 1.15.9. I don't know if it can be fixed.

Please let me know how to fix it.

Thank you.

[Maxim Dounin]

Please check if you are seeing the problem without any 3rd party modules (the nginx -V output suggests there is at least one).

If the problem persists, please provide full configuration (preferably minimal) you are seeing the problem with, as well as a full packet capture of a connection between nginx and your backend where an error can be seen.

[jangys9510@…]

Here is nginx.conf.

---

user irteam;
worker_processes auto;

pid /home1/irteam/apps/nginx/nginx.pid;

events {

worker_connections 16000;
use epoll;

}

http {

include mime.types;
default_type application/octet-stream;

#default on
server_tokens off;

tcp_nopush on;
tcp_nodelay on;

access_log logs/access.log combined;
error_log logs/error.log error;

sendfile on;
etag on;

keepalive_timeout 25s; #default 75s
keepalive_requests 150; #default 100

client_max_body_size 20M; # default 1M
client_body_timeout 60s; # default 60s
client_body_buffer_size 512k; #default 8k|16k
client_header_timeout 60s; # default 60s
client_header_buffer_size 8k; #default 1k
send_timeout 60s; #default 60s
large_client_header_buffers 20 32k; #default 4 8k

gzip on;
gzip_types text/html application/javascript text/css application/json text/javascript;
gzip_disable "MSIE[4-6]\.";

root /home1/irteam/test;

# upstream
upstream tomcat {

server 127.0.0.1:8080 max_fails=0;
keepalive 30;

}

proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_method $request_method;
proxy_http_version 1.1;
proxy_set_header Connection "";

proxy_buffering off; #default on
proxy_request_buffering off; #default on
proxy_buffer_size 32k; #default 4k|8k
proxy_buffers 20 32k; #default 8 4k|8k
proxy_connect_timeout 60s; #default 60s
proxy_read_timeout 60s; #default 60s
proxy_send_timeout 60s; #default 60s

#HTTP Server
server {

listen 80;
server_name test.com;

access_log logs/access.log combined;
error_log logs/error.log error;

rewrite ^{((?!http_stub_status).)*$ ​https://$host$uri permanent;}

location / {

allow all;
index index.html index.jsp;
proxy_pass ​http://tomcat;

}

#gzip static
location ~ .*\.(css|js|js.gz|css.gz)$ {

gzip_vary on;

expires 1w;

}

#static
location ~ .*\.(swf|jpe?g|png|gif|bmp|ico)$ {

expires 1w;

}

location /http_stub_status {

stub_status on;
access_log off;
allow 127.0.0.1;
deny all;

}

}

#HTTPS Server
server {

listen 443 ssl;
server_name test.com;

access_log logs/access.log combined;
error_log logs/error.log error;

ssl_certificate /home1/irteam/apps/nginx/conf/test.cert.pem;
ssl_certificate_key /home1/irteam/apps/nginx/conf/test.key.pem;

ssl_session_cache shared:SSL:600m;
ssl_session_timeout 5m;

ssl_ciphers EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:RSA+AESGCM:RC4:!LOW:!EXP:!PSK:!KRB5:!MD5:!aNULL:!eNULL:ALL;
ssl_prefer_server_ciphers on;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

rewrite ^{/$ /main permanent;}

location / {

allow all;
index index.html index.jsp;
proxy_pass ​http://tomcat;

}

#gzip static
location ~ .*\.(css|js|js.gz|css.gz)$ {

gzip_vary on;

expires 1w;

}

#static
location ~ .*\.(swf|jpe?g|png|gif|bmp|ico)$ {

expires 1w;

}

location /http_stub_status {

stub_status on;
access_log off;
allow 127.0.0.1;
deny all;

}

}

}

[Maxim Dounin]

So, as expected, your configuration uses keepalive connections to upstream servers:

upstream tomcat {
    server 127.0.0.1:8080 max_fails=0;
    keepalive 30;
}

As previously suggested, a full packet capture of a connection is needed to further analyze what's going on here. Most likely, digits before the method name you are seeing are from the previous request, and with full packet capture will'll be able to find out if the problem is on the nginx side (that is, it somehow misreports request length) or on the Tomcat side.

Also, could you please confirm you are still seeing the problem without any 3rd party modules? Thanks.

[Maxim Dounin]

Feedback timeout.