nginx 1.13.0/Solaris 11.3/amd64; coredump when using ssl_session_cache

[Slarky@…]

Hi, nginx 1.13 dumps core on Solaris 11.3 with Oracle Developer Studio 12.5 when compiled with -m64:

$ ./configure --with-cc=/opt/developerstudio12.5/bin/cc --with-cpu-opt=amd64 --prefix=/tmp/nginx13 --with-http_ssl_module

Remove 2x -fast -xipo from objs/Makefile !

$ gmake
...
/opt/developerstudio12.5/bin/cc -c -m64 -errwarn=%all -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include/pcre -I objs \
        -o objs/src/core/ngx_times.o \
        src/core/ngx_times.c src/os/unix/ngx_sunpro_amd64.il
"src/core/ngx_times.c", line 87: argument #2 is incompatible with prototype:
        prototype: pointer to void : "src/core/ngx_times.c", line 0
        argument : pointer to volatile unsigned long
"src/core/ngx_times.c", line 203: argument #2 is incompatible with prototype:
        prototype: pointer to void : "src/core/ngx_times.c", line 0
        argument : pointer to volatile unsigned long
cc: acomp failed for src/core/ngx_times.c

Remove -errwarn=%all from objs/Makefile as well...

$ gmake
...
$ gmake install

Add "ssl_session_cache shared:SSL:10m;" to /tmp/nginx13/conf/nginx.conf

$ /tmp/nginx13/sbin/nginx -t
nginx: the configuration file /tmp/nginx13/conf/nginx.conf syntax is ok
Memory fault(coredump)

$ dbx objs/nginx core
program terminated by signal SEGV (no mapping at the fault address)
Current function is ngx_slab_alloc
  176       return p;
(dbx) where                                                                  
=>[1] ngx_slab_alloc(pool = <bad address 0xffff80feffffffd8>, size = <bad address 0xffff80feffffffd0>), line 176 in "ngx_slab.c"

(dbx) list 165 177
  165   void *
  166   ngx_slab_alloc(ngx_slab_pool_t *pool, size_t size)
  167   {
  168       void  *p;
  169   
  170       ngx_shmtx_lock(&pool->mutex);
  171   
  172       p = ngx_slab_alloc_locked(pool, size);
  173   
  174       ngx_shmtx_unlock(&pool->mutex);
  175   
  176       return p;
  177   }

[skrueger8@…]

The root cause seems to be that nginx thinks it can use GCC atomics even though the sun compiler is used...

checking for gcc builtin atomic operations ... found

I don't get any coredumps or compile time errors when "NGX_HAVE_GCC_ATOMIC" is disabled in objs/ngx_auto_config.h (i.e. delete it or just comment it out).

[Maxim Dounin]

It looks like there is more than one problem here. Oracle Developer Studio 12.5 introduced ​GCC-compatible __sync builtins, but:

1. These builtins aren't really compatible with GCC, and will generate warnings when used with volatile pointers.
2. These builtins seems to be broken at least with some compiler flags / optimizations.

The (1) causes warnings during compilation, and (2) causes segmentation faults.

I was able to reproduce broken behaviour using the following test code:

$ cat t.c
#include <stdio.h>

long  n;

int
main(void)
{
    if (!__sync_bool_compare_and_swap(&n, 0, 1)) {
        fprintf(stderr, "__sync_bool_compare_and_swap() failed\n");
        return 1;
    }

    fprintf(stderr, "all clear\n");
}
$ /opt/developerstudio12.5/bin/cc t.c
$ ./a.out
all clear
$ /opt/developerstudio12.5/bin/cc -m64 t.c
$ ./a.out
__sync_bool_compare_and_swap() failed

Exact compiler options needed to trigger a failure seems to depend heavily on the particular code layout. For example, if long n = 0; is used in main, it only fails with -m64 -O0.

Given the above, the only sensible solution seems to explicitly disable use of GCC atomic operations when compiling with Sun C, patch below. Quick fix would be to use --with-cc-opt="-DNGX_HAVE_GCC_ATOMIC=0".

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1493918641 -10800
#      Thu May 04 20:24:01 2017 +0300
# Node ID 758f3bb7b84aa8937ea6a3dd27f54b3313cefc50
# Parent  f38647c651a8d5c884b5aacc9f9a5b1af196309b
Configure: disabled gcc atomics with Sun C (ticket #1261).

Oracle Developer Studio 12.5 introduced GCC-compatible __sync builtins.
Unfortunately, these builtins are neither GCC-compatible (they generate
warnings when used with volatile), nor working (unexpectedly fail on
unpredictable combinations of code layout and compiler flags).  As such,
the gcc builtin atomic operations configure test explicitly disabled when
compiling with Sun C.

diff --git a/auto/cc/conf b/auto/cc/conf
--- a/auto/cc/conf
+++ b/auto/cc/conf
@@ -178,21 +178,25 @@ if [ "$NGX_PLATFORM" != win32 ]; then
     fi
 
 
-    ngx_feature="gcc builtin atomic operations"
-    ngx_feature_name=NGX_HAVE_GCC_ATOMIC
-    ngx_feature_run=yes
-    ngx_feature_incs=
-    ngx_feature_path=
-    ngx_feature_libs=
-    ngx_feature_test="long  n = 0;
-                      if (!__sync_bool_compare_and_swap(&n, 0, 1))
-                          return 1;
-                      if (__sync_fetch_and_add(&n, 1) != 1)
-                          return 1;
-                      if (n != 2)
-                          return 1;
-                      __sync_synchronize();"
-    . auto/feature
+    if [ "$NGX_CC_NAME" = "sunc" ]; then
+        echo "checking for gcc builtin atomic operations ... disabled"
+    else
+        ngx_feature="gcc builtin atomic operations"
+        ngx_feature_name=NGX_HAVE_GCC_ATOMIC
+        ngx_feature_run=yes
+        ngx_feature_incs=
+        ngx_feature_path=
+        ngx_feature_libs=
+        ngx_feature_test="long  n = 0;
+                          if (!__sync_bool_compare_and_swap(&n, 0, 1))
+                              return 1;
+                          if (__sync_fetch_and_add(&n, 1) != 1)
+                              return 1;
+                          if (n != 2)
+                              return 1;
+                          __sync_synchronize();"
+        . auto/feature
+    fi
 
 
     if [ "$NGX_CC_NAME" = "ccc" ]; then

[Maxim Dounin <mdounin@…>]

In 6999:4925a6d13746/nginx:

Configure: disabled gcc atomics with Sun C (ticket #1261).

Oracle Developer Studio 12.5 introduced GCC-compatible sync builtins.
Unfortunately, these builtins are neither GCC-compatible (they generate
warnings when used with volatile), nor working (unexpectedly fail on
unpredictable combinations of code layout and compiler flags). As such,
the gcc builtin atomic operations configure test explicitly disabled when
compiling with Sun C.

[Maxim Dounin]

Patch committed, thanks for reporting this.