Opened 7 years ago

Closed 7 years ago

Last modified 3 years ago

#390 closed defect (fixed)

Master nginx process segfaults after ~10 config reloads (amazon ami)

Reported by: gansbrest Owned by:
Priority: major Milestone:
Component: nginx-core Version:
Keywords: Cc:
uname -a: Linux sergey-nginx-segfaults 3.2.22-35.60.amzn1.x86_64 #1 SMP Thu Jul 5 14:07:24 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
nginx -V: nginx version: nginx/1.4.2
built by gcc 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx-1.4.2 --conf-path=/etc/nginx/nginx.conf --sbin-path=/opt/nginx-1.4.2/sbin/nginx --with-http_perl_module --with-debug --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_realip_module --with-http_geoip_module --with-ld-opt='-Wl,-R,/usr/local/lib -L /usr/local/lib'

Description

Affected versions 1.4.2 and 1.5.2

Master nginx process segfaults after ~10 config reloads (amazon ami). Not using 3rd party modules.

The problem happens with no traffic, to reproduce the problem I had to execute /etc/init.d/nginx reload about 10 times and then master process would crash. Worker processes would continue to work and serve content. I tried 1.5.2 and noticed same segfault problems during reload.

I have backtrace and debug log attached. Would be great to figure out what's causing the problem, since reload is very useful feature. Let me know if I can provide any other info.

Attachments (3)

nginx-1.4.2.segfault.backtrace.log (21.3 KB ) - added by gansbrest 7 years ago.
backtrace
nginx-1.4.2-debug.logae.tar.gz (102.3 KB ) - added by gansbrest 7 years ago.
nginx_conf.tar.gz (8.3 KB ) - added by gansbrest 7 years ago.
Full nginx config to reproduce the problem

Download all attachments as: .zip

Change History (15)

by gansbrest, 7 years ago

backtrace

by gansbrest, 7 years ago

comment:1 by Maxim Dounin, 7 years ago

Could you please provide configuration file(s) to trigger the problem? Thank you.

comment:2 by gansbrest, 7 years ago

Hi Maxim,

Is there a way to share config files securely, since they may expose some company specific info?

comment:3 by Maxim Dounin, 7 years ago

Not really (unless you have a support contract). If in doubt, please check all try_files directives in the configuration - from the backtract provided I suspect there are try_files with empty arguments.

Alternatively, try the following patch:

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -4758,7 +4758,9 @@ ngx_http_core_try_files(ngx_conf_t *cf,
 
         tf[i].name = value[i + 1];
 
-        if (tf[i].name.data[tf[i].name.len - 1] == '/') {
+        if (tf[i].name.len > 0 
+            && tf[i].name.data[tf[i].name.len - 1] == '/')
+        {
             tf[i].test_dir = 1;
             tf[i].name.len--;
             tf[i].name.data[tf[i].name.len] = '\0';

It should help.

comment:4 by gansbrest, 7 years ago

Thanks for the patch, will give it a shot. I was able to strip down our configs to just couple files - one in includes and one in sites-enabled.

We do use try_files like this:

location ~* /(api) {

try_files "" @colabs_nocache;

}

extensively to direct traffic to different location throughout configs, but I recall Igor mentioned it couple times as a good soulution on the nginx forum. Maybe something had changed in more recent versions of nginx.

Anyways, I attach our full configs where you can reproduce a problem by doing nginx reaload a bunch of times ( with one config it takes more reloads to trigger the problem ). Maybe you could point out specific parts of the config which could cause segfaults.

Thanks again Maxim!

by gansbrest, 7 years ago

Attachment: nginx_conf.tar.gz added

Full nginx config to reproduce the problem

comment:5 by gansbrest, 7 years ago

Hi, Maxim,

Did you have a chance to look at the configs I attached, just so we could understand what we doing wrong and refactor.

comment:6 by Maxim Dounin, 7 years ago

As suggested earlier, try_files with an empty argument most likely is what causes your problems, try the patch above.

comment:7 by gansbrest, 7 years ago

Seems like your patch worked perfectly. Reloaded a bunch of times and it still works. Would be nice if you could propose that for the mainline.

comment:8 by gansbrest, 7 years ago

THANKS A LOT!

comment:9 by Maxim Dounin <mdounin@…>, 7 years ago

In ee2a4c68fb35187906fbaed1331f1faa46ed1286/nginx:

Fixed try_files with empty argument (ticket #390).

comment:10 by Maxim Dounin, 7 years ago

Resolution: fixed
Status: newclosed

Fix committed and will be available in nginx 1.5.4, thank you for report and testing.

comment:11 by Maxim Dounin <mdounin@…>, 7 years ago

In ddf740b9fdf69942e9587749bc29916f7268ee57/nginx:

Fixed try_files with empty argument (ticket #390).

comment:12 by Maxim Dounin, 3 years ago

sensitive: 10
Note: See TracTickets for help on using tickets.