Opened 7 years ago

Closed 7 years ago

#1321 closed enhancement (wontfix)

False positives in map $http_user_agent if regex does not use word boundaries.

Reported by: mitchellkrogza@… Owned by:
Priority: minor Milestone:
Component: documentation Version: 1.10.x
Keywords: regex Cc:
uname -a:
nginx -V: nginx version: nginx/1.10.3 (Ubuntu)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_mp4_module --with-http_perl_module --with-http_random_index_module --with-http_secure_link_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-threads --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/headers-more-nginx-module --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-auth-pam --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-cache-purge --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-dav-ext-module --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-development-kit --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-echo --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/ngx-fancyindex --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-http-push --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-lua --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-upload-progress --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/nginx-upstream-fair --add-module=/build/nginx-ce2jao/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_module

Description

This is not really a bug, merely something to point out to people who may be using

map $http_user_agent

If the regex pattern is not escaped by word boundaries it leads to false positives.

For example:

map $http_user_agent $bad_bot {
"~*Disco"		1;
}

Anything starting with Disco is detected, so an innocent user-agent like Discourse is detected as a positive hit.

But adding word boundaries like this prevents the false positive match.

map $http_user_agent $bad_bot {
   "~*\bDisco\b"		1;
}

The same occurs with

map $http_referer

If dots in referrer domains are not escaped it leads to false positives.

Example:

map $http_referer $bad_referer {
"~*ico.re" 1;
}

Will detect ico.re and also locatellicorretor.com

So dots and special characters need to be escaped

map $http_referer $bad_referer {
"~*ico\.re" 1;
}

Change History (1)

in reply to:  description comment:1 by Maxim Dounin, 7 years ago

Resolution: wontfix
Status: newclosed

Replying to mitchellkrogza@…:

For example:

map $http_user_agent $bad_bot {
"~*Disco"		1;
}

Anything starting with Disco is detected, so an innocent user-agent like Discourse is detected as a positive hit.

Not really. Anything containing Disco is detected, including something like undiscoverable or superdiscount.

These are regular expression basics tough, and I don't think there is a room for such basic things even in introductory articles like How nginx processes a request (it provides some examples of properly escaped regular expressions though, as well as some other places in the documentation).

Note well that any such checks, even properly escaped, are subject to false positives. For example, an unrelated user agent may mention Disco word for some reason. Or a referer may include ico.re string somewhere in the path, like in http://example.com/favico.reference.html. It is generally a good idea to think twice before applying any restrictions based on such checks.

Note: See TracTickets for help on using tickets.