Custom Query (2297 matches)

We are running nginx 1.0.10.

The type of traffic is large posts (more than 4K - 16K post messages, hundreds to thousands per second) We kept crashing within seconds, while free: #4 0x000000000041ac41 in ngx_destroy_pool (pool=0x1f5e3f0) at core/ngx_palloc.c:64

Please see the steps below we run to solve this crash:

1. Commenting our code and instead responding with a dummy predefined message.

After this step I notice the nginx is crashing easily and generated cores easily. (e.g. started at Nov 29 08:06:08, 1st core was created 2011/11/29 08:06:30, 2nd 2011/11/29 08:06:33, 3rd 08:07:16, 4th 2011/11/29 08:07:22) What I also noticed it that it always crashes on the same function/line: ngx_destroy_pool, while it frees a large pool (same large pool for all the cores at the same run).

#4 0x000000000041ac41 in ngx_destroy_pool (pool=0x1f5e3f0) at core/ngx_palloc.c:64

2. After running strlen on the allocated buffer I noticed that the size that is allocated is the same as the size returns by strlen.

Which means the allocation size is for content_ln and not content_ln+1 (+1 for the end_of string), the allocation doesn’t includes the null char. ngx_alloc 0x125e8a0 size 4376

(gdb) p strlen((char *)(l->alloc)) $1 = 4376 (gdb) p l->alloc $2 = (void *) 0x125e8a0

3. I added +1 to the allocation in ngx_palloc_large and rerun again:

static void * ngx_palloc_large(ngx_pool_t *pool, size_t size) {

void *p; ngx_uint_t n; ngx_pool_large_t *large;

p = ngx_alloc(size+1, pool->log); THE CHANGE IS HERE

.... }

it runs for a while, no cores were generated while before I could generate cores easily within seconds.

4. I returned back our original call and rerun again.

So far no cores were created, and it runs for almost 12 hours.

There is probably a better/cleaner fix for that .....

The reason I put it as major is because this crash can lead to a dead lock when all the workers are waiting for a malloc lock while a thread that was killed due to "double-free" false diagnostic (memory corruption) did not release this lock.

Please let me know if you need any core files

Thanks, Rafit

We've set up Nginx as load balancer, but sometimes it returns 502 instead of page contents. We've found the problem is happening when some of backends is going down for maintenance (while others is alive and looks good). Looks like Nginx sometimes decides that alive servers is down, while they aren't.

Everything is tested under very low load (32 requests was performed from a single client in a few threads).

The problem is easily reproducible and affects all Nginx versions since 0.9.4 up to 1.1.9. It's harder to reproduce it in 0.8.55, but looks like it's affected too.

I'm attaching sample minimized configuration with debug logs for latest stable version, which should help to reproduce and figure out the problem.

Also, the problem was reproduced on Centos 6.0 with backends running apache, Nginx and IIS on both Windows and Linux (whenever it was possible).

Quoting ​http://forum.nginx.org/read.php?29,65907,65907

"at the moment [March 20, 2010], linkage on Solaris with SunPRO will fail, if CFLAGS is set in the environment. This is a consequenec of skipping auto/cc/sunc, which sets NGX_AUX to the assembler routines implementing ngx_atomic_fetch_add and co. Shouldn't the compiler specific routines generally be run, but CPU_OPT and NGX_CPU_CACHE_LINE ignored if CFLAGS was already set?"