grpc nginx connection issue (draining and no connection available)

[stephanus.tedy@…]

grpc client
golang, grpc-go/1.10.0

Hi, I am implementing grpc and nginx as load balancer.
setup is :

• use service discovery, it will resolve to multiple server
• each server will have dedicated nginx and do grpc_pass to localhost

on normal traffic, everything works fine.
but on high traffic, start getting error on the client.

it produce 2 types of error :

• there is no connection available
• the connection is draining

but there is no error on access and error log.
these error gone when I disable nginx and connect directly to servers.

what is actually happens and how to fix this ?

Thanks!

config :
server {

server_name ..;
listen 57046 http2;
default_type application/grpc;

access_log /var/log/nginx/grpc.access.log logformat;
error_log /var/log/nginx/grpc.error.log;

location / {

grpc_pass grpc://127.0.0.1:57045;
error_page 502 500 504 = /grpc_error_page;

}

location = /grpc_error_page {

internal;

add_header grpc-status 14;
add_header grpc-message "upstream service failure";

return 204;

}

}

[Maxim Dounin]

These errors seems to come from grpc-go client code:

​https://github.com/grpc/grpc-go/blob/4344c204c9562c9e78083a742ee4634cd556b596/balancer_v1_wrapper.go#L318
​https://github.com/grpc/grpc-go/blob/40a879c23a0dc77234d17e0699d074d5fd151bd0/transport/http2_client.go#L530
​https://github.com/grpc/grpc-go/blob/40a879c23a0dc77234d17e0699d074d5fd151bd0/transport/http2_client.go#L605

It seems that grpc-go isn't able to handle GOAWAY frames and properly retry not-yet-completed requests.

As a workaround, you may try using a large ​http2_max_requests value to prevent nginx from closing client connections periodically.