Nginx not detecting malformed percent encoding in query string

[Colin Matthias]

When serving a request with bad percent encoding in the path portion of the URL, nginx returns a 400 Bad Request error. Example:

$ curl -v "http://localhost/%RRfoo.html"

* About to connect() to localhost port 80 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /%RRfoo.html HTTP/1.1
> User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 NSS/3.13.3.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: localhost
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Server: nginx/1.2.6
< Date: Tue, 11 Jun 2013 16:50:39 GMT
< Content-Type: text/html
< Content-Length: 172
< Connection: close

However, when the bad percent encoding is in the query portion of the URL, nginx fails to detect this condition and passes the bad URL along to our web framework/stack (in this case, Passenger/Rack/Rails), where it throws a 500 Internal Server Error:

$ curl -v "http://localhost/?q=%RRfoo.html"
* About to connect() to localhost port 80 (#0)
*   Trying ::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /?q=%RRfoo.html HTTP/1.1
> User-Agent: curl/7.21.7 (x86_64-redhat-linux-gnu) libcurl/7.21.7 NSS/3.13.3.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: localhost
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/html; charset=utf-8
< Content-Length: 728
< Connection: keep-alive
< Status: 500
< X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.19
< X-Request-Id: 4a9549a1fa73eedf39c4264ceed95fff
< X-Runtime: 0.148903
< Date: Tue, 11 Jun 2013 16:52:29 GMT
< X-Rack-Cache: pass
< Server: nginx/1.2.6 + Phusion Passenger 3.0.19

This inconsistency is causing issues in our app. Per RFC 3986, section 3.4 (​http://tools.ietf.org/html/rfc3986#section-3.4) I would expect the query portion of the URL to be treated the same as the path for the purposes of percent encoding.

Note that this occurs on our 1.2.6 installation of nginx, but making the same requests to ​http://nginx.org, which appears to be running 1.5.0, produces the same results.

[Maxim Dounin]

As nginx doesn't decode query string, it's more or less expected that it doesn't care if there are any invalid percent encoding in the query string and "fails to detect this condition". If it causes problems in your app - fix your app.

[Colin Matthias]

OK, thank you for the info. Out of curiosity, is there a particular reason why nginx does not follow the RFC in this case? Is it for performance/simplicity reasons or something else?

[Maxim Dounin]

What makes you think that nginx doesn't follow the RFC? The fact that it doesn't return 400 doesn't mean it doesn't follow - it means it don't care if some client doesn't follow the RFC.

As for query string encoding, there is at least one widely used in the past in query strings which isn't RFC-complaint - %uXXXX, still seen sometimes from a javascript code in old browsers. Rejecting such query strings would be a very bad idea.

[Colin Matthias]

I see. I wasn't aware of the %uXXXX issue... that definitely makes sense. It's just frustrating to me because at every major level of my app's stack (nginx, Passenger, Rack, Rails) there seems to be a different idea about who is responsible for determining when a query string is malformed. Not your fault though -- I will just have to hack a solution into my app for now. Thank you for the information and your help!