Minimum zone size for limit_req strangely high

[xim@…]

Hi,

I'm running nginx on an embedded platform, namely a Tilera Tile-Gx36. The embedded http server is not intended for any heavy lifting, but for API calls that are possibly expensive. I tried using this config:

limit_req_zone 1 zone=global:32k rate=10r/s;

This works on my x86 computer, but not on the Tilera. I checked the zone requirements in the source code, and found this code:

if (size < (ssize_t) (8 * ngx_pagesize)) { ... "zone \"%V\" is too small" ... }

The Tilera has a 64k page size. This limit means I have to allocate at least 512k memory for each zone I want to define, which seems excessive on an embedded system which will run with a very low rate and a very low burst.

Is there any reason why this limit needs to be "at least 8 pages"?

[Maxim Dounin]

This is because shared memory uses slab allocator, which allocates memory by dividing a memory page into small fixed-size chunks. Absolute minimum for the slab allocator to work is 2 pages - one page will be used for internal slab allocator structures, and another one for real allocations. This is, however, is only enough for very simple cases, when all allocations from the memory zone will use the same size.

With limit_req, absolute minimum is 3 pages - with only 2 pages nginx won't be able to allocate limit_req global structures, because there are two allocations, and they happen to use distinct slabs. This is, however, do not take into account actual allocations of limit states, and, depending on key size used, 3 pages may not be enough to store even one state.

Moreover, things become worse when different key sizes are used. When there are only a few pages available, this means that only a few distinct states with differently sized keys can be stored. For example, if there are only 4 pages (with one of them used for slab allocator internal data), you'll be able to store a 64-byte state, 128-bytes state, but trying to store a 256-bytes state will result in an error.

The "at least 8 pages" limit is a safety limit used by (almost) all modules which work with shared memory. It ensures that the specified shared memory size is big enough to store various global allocations, including slab allocator's own data, and there is some room to make allocations from.

Note well that in most real-world cases you actually need some states to be stored, and for things to work you actually need reasonable shared memory size, in most cases more than 1 megabyte, which only enough to store ​16 thousands of 64-byte states.

While changing this limit is possible, given the above the smallest practical value is about 4 pages. So it will be about 256k on your platform, and I don't really think it makes a difference. On the other hand, 8 pages limit as used now seems to be a better limit from the user experience point of view as it ensures that the shared memory zone can be used for allocations of various sizes.

[xim@…]

Thanks for the detailed answer. As this is a web server running an admin interface on an embedded device, with a very simple limit_req config, it sounds like we could safely patch nginx to reduce the limit to 4 pages. Thanks.

[Maxim Dounin]

Closing this, it doesn't look like there are practical reasons to try to change the limit to 4 pages instead.

[joshua.mccullough.teslagovernment.com@…]

Could the docs be updated regarding this? It's very confusing. In my case, I'm trying to find the magic number and am stepping up the size in base2 e.g. 32, 64, 128, etc. I'm on 4096 and it's still "too small".

[joshua.mccullough.teslagovernment.com@…]

The above comment was my mistake, I did not include the unit in the size parameter, so my zone=abc:10 and such were being interpreted as bytes, I assume. Fixed with zone=abc:10m, as the docs show in the example.