Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#1403 closed defect (wontfix)

nginx module cannot use libpcre

Reported by: andwei@… Owned by:
Priority: minor Milestone:
Component: nginx-core Version: 1.10.x
Keywords: Cc:
uname -a: Linux <somename> 4.9.45-yocto-custom #3 SMP Wed Sep 27 14:10:54 CEST 2017 i686 i686 i386 GNU/Linux
nginx -V: configure arguments: --with-ipv6 --with-http_ssl_module --with-http_auth_request_module --with-http_gzip_static_module --with-http_v2_module --with-cc='i586-poky-linux-gcc -Wl,--hash-style=gnu -DNGX_ENABLE_SYSLOG' --with-pcre --sbin-path=/usr/sbin/nginx --pid-path=/var/run/ --lock-path=/var/lock/nginx.lock --error-log-path=/var/log/nginx_error --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx_access --http-client-body-temp-path=/var/lib/nginx/client_body_temp --http-proxy-temp-path=/var/lib/nginx/proxy_temp --http-fastcgi-temp-path=/var/lib/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/lib/nginx/uwsgi_temp --http-scgi-temp-path=/var/lib/nginx/scgi_temp --add-module=../rrdgraph --add-module=../echomod --with-cc='i586-poky-linux-gcc -m32 -march=i586 --sysroot=<some-sysroot> -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed'


Using the rrdgraph module, and with glib having switched to using the libpcre system library, I found that libpcre becomes unusable for that module (and theoretically for all other modules, too), because src/core/ngx_regex sets pcre_malloc = ngx_regex_malloc. This, of course works for ngx_regex_compile, which initializes the pool used by ngx_regex_malloc and clears it after regex compilation.

libpcre error message:

cannot compile regular expression: Error while compiling regular expression ^(?:[^%]+|%%)*%[-+ 0#]?[0-9]*(?:[.][0-9]+)?l[eEfFgG](?:[^%]+|%%)*(?:%s)?(?:[^%]+|%%)*$ at char 86: failed to get memory (^(?:[^%]+|%%)*%[-+ 0#]?[0-9]*(?:[.][0-9]+)?l[eEfFgG](?:[^%]+|%%)*(?:%s)?(?:[^%]+|%%)*$)

For an nginx module (in our case compiled statically into nginx), this means, however, that using libpcre regex compilation is no longer possible, because ngx_regex_malloc does not have any buffer available when called from somewhere else.

What first came to my mind was to just exchange the allocation mechanism before calling the rrdlibrary and change it back afterwards, but I'm quite sure this would get me into serious multithreading trouble.

Globally changing the pcre_malloc to standard malloc is probably even worse due to the allocated memory not being cleaned up, because this is usually done by the pools (right?) and also probably decreasing performance.

Any ideas on how to solve this without doing any damage?

A working alternative is to use the glib's embedded libpcre, but this is quite outdated and judging from will be removed soon.

Change History (2)

comment:1 by Maxim Dounin, 7 years ago

Resolution: wontfix
Status: newclosed

In general nginx assumes that any regex compilation should be done via ngx_regex_compile() which takes care of setting appropriate global variables, notably pool to allocate memory from. Once finished, it is intentionally does ngx_pcre_pool = NULL to ensure no additional allocations will be done. This expectedly causes problems when PCRE functions are used directly.

Obviously enough, in the case of the particular module it is not possible to switch to using nginx infrastructure (which would be best possible solution), as PCRE is used deeply in the rrd library. A possible solution would be to set pcre_malloc / pcre_free before the library call, and restore them afterwards. This should work at least as long as the library does not try to create threads and do some background work (and certainly there will be other problems if a library will try to create threads).

I doubt there is a better solution since memory allocation functions in PCRE are global. PCRE2 might be better here as it allows to specify private allocation functions via pcre2_general_context_create() instead of via global variables, though it's yet to be adopted.

comment:2 by andwei@…, 7 years ago

Thanks for confirming.

Note: See TracTickets for help on using tickets.