Opened 6 years ago

Closed 6 years ago

#1590 closed defect (invalid)

grpc nginx connection issue (draining and no connection available)

Reported by: stephanus.tedy@… Owned by:
Priority: blocker Milestone:
Component: nginx-module Version: 1.14.x
Keywords: grpc connection failure, the connection is draining, no connection available Cc:
uname -a: Linux 4.4.0-93-generic #116~14.04.1-Ubuntu SMP Mon Aug 14 16:07:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.14.0

Description

grpc client
golang, grpc-go/1.10.0

Hi, I am implementing grpc and nginx as load balancer.
setup is :

  • use service discovery, it will resolve to multiple server
  • each server will have dedicated nginx and do grpc_pass to localhost

on normal traffic, everything works fine.
but on high traffic, start getting error on the client.

it produce 2 types of error :

  • there is no connection available
  • the connection is draining

but there is no error on access and error log.
these error gone when I disable nginx and connect directly to servers.

what is actually happens and how to fix this ?

Thanks!

config :
server {

server_name ..;
listen 57046 http2;
default_type application/grpc;

access_log /var/log/nginx/grpc.access.log logformat;
error_log /var/log/nginx/grpc.error.log;

location / {

grpc_pass grpc://127.0.0.1:57045;
error_page 502 500 504 = /grpc_error_page;

}

location = /grpc_error_page {

internal;

add_header grpc-status 14;
add_header grpc-message "upstream service failure";

return 204;

}

}

Change History (1)

comment:1 by Maxim Dounin, 6 years ago

Resolution: invalid
Status: newclosed

These errors seems to come from grpc-go client code:

https://github.com/grpc/grpc-go/blob/4344c204c9562c9e78083a742ee4634cd556b596/balancer_v1_wrapper.go#L318
https://github.com/grpc/grpc-go/blob/40a879c23a0dc77234d17e0699d074d5fd151bd0/transport/http2_client.go#L530
https://github.com/grpc/grpc-go/blob/40a879c23a0dc77234d17e0699d074d5fd151bd0/transport/http2_client.go#L605

It seems that grpc-go isn't able to handle GOAWAY frames and properly retry not-yet-completed requests.

As a workaround, you may try using a large http2_max_requests value to prevent nginx from closing client connections periodically.

Note: See TracTickets for help on using tickets.