Memory overrun due to alignment issues when cross-compiling mingw on linux

[Orgad Shaneh]

Hi,

When I cross-compile nginx for Windows on linux host using mxe, NGX_ALIGNMENT is set to 16. This is set in auto/os/conf:

Error: Failed to load processor bash

No macro or processor named 'bash' found

But the pool is allocated using plain malloc (on Windows target), which is *not* aligned to 16 bytes (it's aligned to 4), so pool->d.end is 4-byte aligned.

Now, in ngx_palloc_small, the following code causes integer underflow, and the comparison doesn't work as expected:

        m = p->d.last;

        if (align) {
            m = ngx_align_ptr(m, NGX_ALIGNMENT);
        }

        if ((size_t) (p->d.end - m) >= size) { /* <------- This condition */
            p->d.last = m + size;

            return m;
        }

Since m is aligned, and end is not, end - m can be a negative number. Then it is cast to size_t, which is unsigned, and so it just keeps writing past the memory pool, and overruns the next pool.

A minimal patch that solves this issue is:

src/core/ngx_palloc.c

@@ -160 +160 @@
             m = ngx_align_ptr(m, NGX_ALIGNMENT);
         }
 
-        if ((size_t) (p->d.end - m) >= size) {
+        if (m + size <= p->d.end) {
             p->d.last = m + size;
 
             return m;

But it would be better to assign the right alignment for Windows, even when cross-building. And also ensure that end is aligned correctly.

I also noticed that the default alignment when not set is sizeof(long), which is wrong for Win64. It should be sizeof(intptr_t).

Build command:

Error: Failed to load processor bash

No macro or processor named 'bash' found