Opened 4 years ago

Last modified 4 years ago

#2060 accepted defect

Nginx doesn't take case http_502 as unsuccessful attempt in ngx_http_grpc_module

Reported by: AbacusHu@… Owned by:
Priority: minor Milestone:
Component: nginx-module Version: 1.15.x
Keywords: ngx_http_grpc_module, unsuccessful attempt, http_502 Cc:
uname -a: Linux tradecenter-01 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.15.5
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --user=nginx --group=nginx --prefix=/opt/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_v2_module --with-stream

Description

From the nginx document ​http://nginx.org/en/docs/http/ngx_http_grpc_module.html, syntax "grpc_next_upstream error timeout http_502;" is valid and case http_502 will take as unsuccessful attempt. However, Nginx doesn't take case http_502 as unsuccessful attempt in fact.

Below is an example. A grpc client sent request to nginx server every seconds, and nginx kept sending request to the upstream server which returned 502 and the other one in round-robin way. Nginx didn't take case http_502 as unsuccessful attempt.

nginx config file:

upstream testserver {
  server 10.46.46.161:9999 max_fails=1 fail_timeout=60; # another nginx server which can retrun responses with error code 502.
  server 10.46.46.160:9999; # a server which can retrun normal responses with status code 200.
}

server {
  listen 8888 http2;
  location /com.company.test {
    grpc_pass grpc://testserver;
    grpc_next_upstream error timeout http_504 http_502 non_idempotent;
  }
}

access log file:

[11:24:40 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 502|| 150|| "-"|| grpc-java-netty/1.17.2|| -|| 0.000|| 0.001|| 10.46.46.161:9999|| 502
[11:24:41 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 200|| 28|| "-"|| grpc-java-netty/1.17.2|| -|| 0.003|| 0.003|| 10.46.46.160:9999|| 200
[11:24:42 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 502|| 150|| "-"|| grpc-java-netty/1.17.2|| -|| 0.000|| 0.001|| 10.46.46.161:9999|| 502
[11:24:43 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 200|| 28|| "-"|| grpc-java-netty/1.17.2|| -|| 0.005|| 0.005|| 10.46.46.160:9999|| 200
[11:24:44 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 502|| 150|| "-"|| grpc-java-netty/1.17.2|| -|| 0.001|| 0.000|| 10.46.46.161:9999|| 502
[11:24:45 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 200|| 28|| "-"|| grpc-java-netty/1.17.2|| -|| 0.005|| 0.004|| 10.46.46.160:9999|| 200
[11:24:46 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 502|| 150|| "-"|| grpc-java-netty/1.17.2|| -|| 0.000|| 0.001|| 10.46.46.161:9999|| 502
[11:24:47 +0000]|| "POST /com.company.test/order HTTP/2.0"|| 200|| 28|| "-"|| grpc-java-netty/1.17.2|| -|| 0.003|| 0.003|| 10.46.46.160:9999|| 200

Change History (1)

comment:1 by Maxim Dounin, 4 years ago

Milestone: nginx-1.19
Priority: major → minor
Status: new → accepted

This seems to happen because switching to the next upstream server is not possible due to request body buffering switched off (which is the only option for gRPC proxying, as there can be client-to-server streaming RPCs in gRPC).

Even if switching to the next upstream server is not possible, we should still mark the server as failed, which does not happen now.

Note: See TracTickets for help on using tickets.