Opened 6 months ago

Closed 5 months ago

#1794 closed defect (invalid)

Difference Between Upstream Response Time And Nginx Response Time

Reported by: Sagararora2492@… Owned by:
Priority: blocker Milestone:
Component: other Version: 1.15.x
Keywords: Cc:
uname -a: Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.14.2 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

HI Folks,

From past few months we are facing issue in nginx , wherein we can see huge difference between upstream response time and nginx response time , this difference lies between 1-6seconds and this pattern occur's at random interval of time .

This is happening at random interval but for every 100th connection request at nginx, have checked there is no load on our system and network wide resources are fine.

Below are the snippets of our conf

user nginx;
worker_processes 22;
pid /var/run/nginx.pid;
worker_rlimit_nofile 60000;

events {

worker_connections 30000;
multi_accept on;
use epoll;

}

http {

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 30;
types_hash_max_size 2048;
server_tokens off;

Please help us out to figure this

Change History (5)

comment:1 follow-up: Changed 6 months ago by mdounin

Please define "nginx response time" - how do you measure it? What are the numbers you observe, and why you think this is the problem in nginx?

comment:2 in reply to: ↑ 1 Changed 6 months ago by Sagararora2492@…

Replying to mdounin:

Please define "nginx response time" - how do you measure it? What are the numbers you observe, and why you think this is the problem in nginx?

For an instance we have an API (Supposingly A) , we are capturing upstream response time of A in api log file, response time of that api lies between 1-50ms where in nginx response time is is close to 4-6seconds
We are getting nginxResponsetime from nginx logpattern which is mentioned below

log_format kibana '$http_host '

'|$http_true_client_ip [$time_local] '
'$remote_addr [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time '
'$upstream_response_time '
'"$connection|$connection_requests" '
'"$upstream_addr" $ssl_protocol/"$ssl_cipher"'
' $upstream_status $siteshield $http_TRACK_TLS1_0';


comment:3 follow-up: Changed 6 months ago by mdounin

There is no "nginxResponsetime" in the log pattern provided. Do you mean $request_time variable?

comment:4 in reply to: ↑ 3 Changed 5 months ago by sagar1.arora.paytm.com@…

Replying to mdounin:

There is no "nginxResponsetime" in the log pattern provided. Do you mean $request_time variable?

Yes we are considering nginxResponsetime from log value $request_time.

comment:5 Changed 5 months ago by mdounin

  • Resolution set to invalid
  • Status changed from new to closed

The $request_time variable measures time between reading first bytes from the client and sending last bytes to the client. It can be larger than $upstream_response_time if, for example, sending the response to the client takes additional time, or when receiving the request from the client takes some non-zero time. Large differences are likely to indicate network problems / packet loss somewhere, causing TCP timeouts and retransmits. Either way, this doesn't look like an nginx issue.

Note: See TracTickets for help on using tickets.