nginx with grpc is poor performance

[kinglion811@…]

$nginx -v
nginx version: nginx/1.13.12
$uname -a
Linux bjlt-hg58.sy 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
I delopy one nginx by docker which network mode is host,then I deploy fifteen tensorflow serving,the nginx upstream to them
I make reques,the qps is 1000,but the ack has too many error,if I don't use nginx,make request director to serving hte error is 0. the request error is:
grpcerror err:Error 14 unavailable tcp write failed

the config is:

daemon off;
worker_processes 56;

pid /run/nginx.pid;

worker_rlimit_nofile 1024;

worker_shutdown_timeout 10s ;

events {
multi_accept on;
worker_connections 16384;
use epoll;
}

http {

#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent"';
log_format upstreaminfo '$remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';
keepalive_timeout 120s;
keepalive_requests 8192;
upstream ingress-nginx-serving-svc2-9000 {

least_conn;
keepalive 32;

server 10.48.147.52:9000;
server 10.48.147.28:9000;
server 10.48.146.35:9000;
server 10.48.144.27:9000;
server 10.48.153.43:9000;
server 10.48.147.52:9002;
server 10.48.147.28:9002;
server 10.48.146.35:9002;
server 10.48.144.27:9002;
server 10.48.153.43:9002;
server 10.48.147.52:9003;
server 10.48.147.28:9003;
server 10.48.146.35:9003;
server 10.48.144.27:9003;
server 10.48.153.43:9003;

}

server {

listen 8088 backlog=8192 http2;

#access_log logs/access.log upstreaminfo;
access_log /var/log/nginx/access.log upstreaminfo;

location / {

#grpc_pass grpc://10.48.144.28:31579;
grpc_pass grpc://ingress-nginx-serving-svc2-9000;

}

}

}

[Maxim Dounin]

You may want to take a look into nginx error log to find out if something is going wrong. Looking into system logs and status might be helpful too. Also, you may dig further into your grpc client to understand why it reports errors and what exactly these errors mean.

You may also want to define "too many" and elaborate on why you think that the performance you observe is low.

Note well that with HTTP/2 the keepalive_requests directive you've used in the configuration is ignored. Consider ​http2_max_requests instead.

Note well that worker_processes 56; might be too many unless you really have 56 or more CPUs dedicated to nginx processes and appropriate amount of memory. It is certainly much more than needed for proxying of 1000 requests per second, and more likely to cause problems if there are not enough resources than to do anything good. You may want to start with worker_processes 1; instead.

[Maxim Dounin]

Feedback timeout.