Client disconnect issue lasts for 24 hours
|Reported by:||Andrew Benson||Owned by:||somebody|
|uname -a:||SunOS hostname 5.10 Generic_147440-10 sun4v sparc SUNW,Sun-Blade-T6340|
nginx version: nginx/0.8.55
built by Sun C 5.11 SunOS_sparc 145354-05 2011/09/13
TLS SNI support disabled
configure arguments: --with-cc=/opt/solstudio12.2/bin/cc --with-cpp=/opt/solstudio12.2/bin/cc --with-cc-opt='-xtarget=ultraT2plus -xO5 -I /usr/sfw/include' --with-ld-opt='-R/usr/sfw/lib -L/usr/sfw/lib' --without-http --with-mail --with-mail_ssl_module --error-log-path=logs/nginx.log --pid-path=/var/run/nginx.pid --with-debug
Running nginx-0.8.55 (also occurs with 0.7.67 and 1.0.14) as an imap/pop proxy only, we have experienced a very strange bug that seems to occur every 50 days or so and last just under 24 hours at each occurrence. It has occurred 4 times now.
Symptoms: imap/pop clients are frequently disconnected from the nginx proxy with the following message printed in the nginx.log:
2012/04/03 21:25:11 [info] 29999#0: *23561178 client timed out (145: Connection timed out) while proxying, client: redacted, server: 0.0.0.0:993, login: "redacted", upstream: 127.0.0.1:14300
First occurrence (Eastern Daylight Time EDT):
Begin: 2011/11/06 17:16:33
End: 2011/11/07 16:46:51
Second (Eastern Standard Time EST):
Begin: 2011/12/26 10:17:55
End: 2011/12/27 09:49:57
Begin: 2012/02/14 03:21:09
End: 2012/02/15 02:52:01
Begin: 2012/04/03 21:25:11
End: 2012/04/04 20:55:26
The only way we've been able to resolve the issue has been to set the system clock to a point in time outside of the affected window. We believe the network is not at fault as we were able to reproduce the problem consistently between client and server on the same network segment with no firewall or load balancer in between.
We can reproduce the issue at any time by setting the system clock to a point within any one of the last occurrences. I can provide packet captures if it is useful but nothing seems out of the ordinary in them.