Master nginx process segfaults after ~10 config reloads (amazon ami)

[gansbrest]

Affected versions 1.4.2 and 1.5.2

Master nginx process segfaults after ~10 config reloads (amazon ami). Not using 3rd party modules.

The problem happens with no traffic, to reproduce the problem I had to execute /etc/init.d/nginx reload about 10 times and then master process would crash. Worker processes would continue to work and serve content. I tried 1.5.2 and noticed same segfault problems during reload.

I have backtrace and debug log attached. Would be great to figure out what's causing the problem, since reload is very useful feature. Let me know if I can provide any other info.

[gansbrest]

backtrace

[Maxim Dounin]

Could you please provide configuration file(s) to trigger the problem? Thank you.

[gansbrest]

Hi Maxim,

Is there a way to share config files securely, since they may expose some company specific info?

[Maxim Dounin]

Not really (unless you have a ​support contract). If in doubt, please check all try_files directives in the configuration - from the backtract provided I suspect there are try_files with empty arguments.

Alternatively, try the following patch:

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -4758,7 +4758,9 @@ ngx_http_core_try_files(ngx_conf_t *cf,
 
         tf[i].name = value[i + 1];
 
-        if (tf[i].name.data[tf[i].name.len - 1] == '/') {
+        if (tf[i].name.len > 0 
+            && tf[i].name.data[tf[i].name.len - 1] == '/')
+        {
             tf[i].test_dir = 1;
             tf[i].name.len--;
             tf[i].name.data[tf[i].name.len] = '\0';

It should help.

[gansbrest]

Thanks for the patch, will give it a shot. I was able to strip down our configs to just couple files - one in includes and one in sites-enabled.

We do use try_files like this:

location ~* ^{/(api) {}

try_files "" @colabs_nocache;

}

extensively to direct traffic to different location throughout configs, but I recall Igor mentioned it couple times as a good soulution on the nginx forum. Maybe something had changed in more recent versions of nginx.

Anyways, I attach our full configs where you can reproduce a problem by doing nginx reaload a bunch of times ( with one config it takes more reloads to trigger the problem ). Maybe you could point out specific parts of the config which could cause segfaults.

Thanks again Maxim!

[gansbrest]

Full nginx config to reproduce the problem

[gansbrest]

Hi, Maxim,

Did you have a chance to look at the configs I attached, just so we could understand what we doing wrong and refactor.

[Maxim Dounin]

As suggested earlier, try_files with an empty argument most likely is what causes your problems, try the patch above.

[gansbrest]

Seems like your patch worked perfectly. Reloaded a bunch of times and it still works. Would be nice if you could propose that for the mainline.

[gansbrest]

THANKS A LOT!

[Maxim Dounin <mdounin@…>]

In ee2a4c68fb35187906fbaed1331f1faa46ed1286/nginx:

Fixed try_files with empty argument (ticket #390).

[Maxim Dounin]

Fix committed and will be available in nginx 1.5.4, thank you for report and testing.

[Maxim Dounin <mdounin@…>]

In ddf740b9fdf69942e9587749bc29916f7268ee57/nginx:

Fixed try_files with empty argument (ticket #390).