Opened 6 years ago

Closed 6 years ago

#1590 closed defect (invalid)

grpc nginx connection issue (draining and no connection available)

Reported by: stephanus.tedy@… Owned by:
Priority: blocker Milestone:
Component: nginx-module Version: 1.14.x
Keywords: grpc connection failure, the connection is draining, no connection available Cc:
uname -a: Linux 4.4.0-93-generic #116~14.04.1-Ubuntu SMP Mon Aug 14 16:07:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.14.0


grpc client
golang, grpc-go/1.10.0

Hi, I am implementing grpc and nginx as load balancer.
setup is :

  • use service discovery, it will resolve to multiple server
  • each server will have dedicated nginx and do grpc_pass to localhost

on normal traffic, everything works fine.
but on high traffic, start getting error on the client.

it produce 2 types of error :

  • there is no connection available
  • the connection is draining

but there is no error on access and error log.
these error gone when I disable nginx and connect directly to servers.

what is actually happens and how to fix this ?


config :
server {

server_name ..;
listen 57046 http2;
default_type application/grpc;

access_log /var/log/nginx/grpc.access.log logformat;
error_log /var/log/nginx/grpc.error.log;

location / {

grpc_pass grpc://;
error_page 502 500 504 = /grpc_error_page;


location = /grpc_error_page {


add_header grpc-status 14;
add_header grpc-message "upstream service failure";

return 204;



Change History (1)

comment:1 by Maxim Dounin, 6 years ago

Resolution: invalid
Status: newclosed

These errors seems to come from grpc-go client code:

It seems that grpc-go isn't able to handle GOAWAY frames and properly retry not-yet-completed requests.

As a workaround, you may try using a large http2_max_requests value to prevent nginx from closing client connections periodically.

Note: See TracTickets for help on using tickets.