Opened 4 years ago

Closed 4 years ago

#989 closed enhancement (duplicate)

Use monotonic time for event handlers

Reported by: i.ponetaev.ndmsystems.com@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.10.x
Keywords: Cc:
uname -a: Linux Keenetic_Start 3.4.112 #1 Mon May 30 16:36:55 MSK 2016 mips GNU/Linux
nginx -V: nginx version: nginx/1.10.0
built by gcc 4.9.3 (crosstool-NG crosstool-ng-1.22.0-ndm-5 - NDMS Toolkit 1.0.0)
configure arguments: --with-libfatalsig --crossbuild=Linux::mipsel --prefix=/usr --without-select_module --without-poll_module --without-http_charset_module --without-http_gzip_module --without-http_ssi_module --without-http_userid_module --without-http_access_module --without-http_auth_basic_module --without-http_geo_module --without-http_map_module --without-http_split_clients_module --without-http_uwsgi_module --without-http_fastcgi_module --without-http_memcached_module --without-http_limit_conn_module --without-http_limit_req_module --without-http_empty_gif_module --without-http_browser_module --without-http_upstream_hash_module --without-http_upstream_ip_hash_module --without-http_upstream_least_conn_module --without-http_upstream_keepalive_module --without-http_upstream_zone_module --without-http-cache --with-pcre=/mnt/build/build/ipsec/kl_rg/build_dir/target-mipsel-linux-uclibc/nginx-1.10.0-3/ndm/pcre-8.38 --with-pcre-opt='-Os -pipe -mips32r2 -mtune=mips32r2 -fhonour-copts -msoft-float -fasynchronous-unwind-tables -fno-omit-frame-pointer -fcaller-saves' --with-pcre-host=mipsel-linux-uclibc --add-module=/mnt/build/build/ipsec/kl_rg/build_dir/target-mipsel-linux-uclibc/nginx-1.10.0-3/ndm/nginx-http-auth-digest --conf-path=/tmp/nginx/nginx.conf --error-log-path=/tmp/nginx/error.log --pid-path=/tmp/nginx/nginx.pid --lock-path=/tmp/nginx/nginx.lock --http-log-path=/tmp/nginx/access.log --http-client-body-temp-path=/tmp/nginx/body --http-proxy-temp-path=/tmp/nginx/proxy --http-scgi-temp-path=/tmp/nginx/scgi --with-cc=/mnt/build/build/ipsec/kl_rg/staging_dir/toolchain-mipsel-linux-uclibc/bin/mipsel-ndms-linux-uclibc-gcc --with-cc-opt='-Os -pipe -mips32r2 -mtune=mips32r2 -fhonour-copts -msoft-float -fasynchronous-unwind-tables -fno-omit-frame-pointer -fcaller-saves' --with-ld-opt='-L/mnt/build/build/ipsec/kl_rg/staging_dir/target-mipsel-linux-uclibc/usr/lib -L/mnt/build/build/ipsec/kl_rg/staging_dir/target-mipsel-linux-uclibc/lib'

Description

Many embedded systems don't have RTC, so they need to use external time source like NTP.
When large jump in system time happens (for example after ntpdate or settimedate() execution), nginx closes all connections with timeout error.

Just a few lines from device's log when we change system time periodically by 5-10 minutes jumps and http/1.1 client performs requests in infinite cycle:

[I] May 31 15:10:21 ndm: Core::System::Clock: system time has been changed.
[E] May 31 15:10:22 keenetic_start nginx: 2016/05/31 15:10:22 [error] 273#0: *433 upstream timed out (145: Unknown error) while reading response header from upstream, client: 192.168.1.237, server: my.keenetic.net, request: "POST /ci HTTP/1.1", upstream: "scgi://127.0.0"
[I] May 31 15:11:21 ndm: Core::System::Clock: system time has been changed.
[I] May 31 15:20:21 ndm: Core::Syslog: last message repeated 3 times.
[E] May 31 15:20:22 keenetic_start nginx: 2016/05/31 15:20:22 [error] 273#0: *457 upstream timed out (145: Unknown error) while reading response header from upstream, client: 192.168.1.237, server: my.keenetic.net, request: "POST /ci HTTP/1.1", upstream: "scgi://127.0.0"
[I] May 31 15:25:21 ndm: Core::System::Clock: system time has been changed.
[E] May 31 15:25:22 keenetic_start nginx: 2016/05/31 15:25:22 [error] 273#0: *461 upstream timed out (145: Unknown error) while reading response header from upstream, client: 192.168.1.237, server: my.keenetic.net, request: "POST /ci HTTP/1.1", upstream: "scgi://127.0.0"
[I] May 31 15:30:21 ndm: Core::System::Clock: system time has been changed.
[I] May 31 15:35:21 ndm: Core::System::Clock: system time has been changed.
[E] May 31 15:35:22 keenetic_start nginx: 2016/05/31 15:35:22 [error] 273#0: *469 upstream timed out (145: Unknown error) while reading response header from upstream, client: 192.168.1.237, server: my.keenetic.net, request: "POST /ci HTTP/1.1", upstream: "scgi://127.0.0"
[I] May 31 15:40:21 ndm: Core::System::Clock: system time has been changed.
[E] May 31 15:40:22 keenetic_start nginx: 2016/05/31 15:40:22 [error] 273#0: *471 upstream timed out (145: Unknown error) while reading response header from upstream, client: 192.168.1.237, server: my.keenetic.net, request: "POST /ci HTTP/1.1", upstream: "scgi://127.0.0"

To mitigate the problem we can use monotonic clock source (not gettimeofday(), but clock_gettime(CLOCK_MONOTONIC,) for variable ngx_current_msec in src/core/ngx_times.c. As I understand from sources, it is used only for event handlers and this change should not affect logs, http headers, e.t.c.

Patch that we use to mitigate the problem is attached.

Attachments (1)

1.diff (1.3 KB ) - added by i.ponetaev.ndmsystems.com@… 4 years ago.
patch for replacing clocksource for ngx_current_msec

Download all attachments as: .zip

Change History (3)

by i.ponetaev.ndmsystems.com@…, 4 years ago

Attachment: 1.diff added

patch for replacing clocksource for ngx_current_msec

comment:1 by Maxim Dounin, 4 years ago

comment:2 by Maxim Dounin, 4 years ago

Resolution: duplicate
Status: newclosed
Note: See TracTickets for help on using tickets.