Space in URL %20 is decoded space causing an invalid URL

[cawoodm@…]

We are seeing nginx 1.15 and the latest 1.17.8 wrongly decoding %20 to " " (space) in URLs before forwarding them upstream. Some upstreams do not accept this and cause a 400. Nginx should not IMHO be unencoding URLs!

Consider the following Nginx running on 8080 with fiddler running as the upstream server on port 8888 to view traffic:

location ~ "^/yo/(.*)" {
  proxy_pass http://127.0.0.1:8888/$1$is_args$args;
}

Next we hit NginX with:

GET http://localhost:8080/yo/foo%20bar

Fiddler (upstream) shows the incoming request as:

GET /foo bar

Correct would be

GET /foo%20bar

The following workaround is available:

location ~ "^/yo/(.*)" {
  set $allowspace1 $1;
  proxy_pass http://127.0.0.1:8888/$allowspace1$is_args$args;

[Maxim Dounin]

In your configuration snippet:

location ~ "^/yo/(.*)" {
  proxy_pass http://127.0.0.1:8888/$1$is_args$args;
}

you are using $1$is_args$args, where $1 comes from unescaped URI as matched by the location. Given that proxy_pass expects properly escaped URI and uses it as is, this results in arbitrary invalid requests being generated.

If you are not prepared to deal with it and do not want to correctly escape all the variables used, consider using proxy_pass without variables instead. The same configuration can be correctly rewritten as:

location /yo/ {
    proxy_pass http://127.0.0.1:8888/;
}

Note the trailing / in proxy_pass. Alternatively, you can use proxy_pass without a URI and rewrite, though in this particular case using proxy_pass with a URI is much cleaner solution. See ​proxy_pass docs for additional examples.

(Note well that the "workaround" you've mentioned is rather a bug, see #348.)

[marc.cawood.buhlergroup.com@…]

you are using $1$is_args$args, where $1 comes from unescaped URI as matched by the location.

Do you mean by "unescaped" that the URL is decoded from /foo%20bar to /foo bar? If so then you are confirming the incorrect behaviour. Why would anyone want the URL unencoded??

I want to pass /foo%20bar and not /foo bar downstream. Why is $1 unencoded and which variable can I use to get the URL *as is*?

[marc.cawood.buhlergroup.com@…]

Note: in our scenario we are rewriting /foo%20bar to /subdir/foo%20bar downstream and not to root / and thus we are using proxy_pass with a URL and running into this documented but IMHO incorrect behaviour.

Documentation

If the proxy_pass directive is specified with a URI, then when a request is passed to the server, the part of a normalized request URI matching the location is replaced by a URI specified in the directive:

location /name/ {
    proxy_pass http://127.0.0.1/remote/;
}

If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI:

location /some/path/ {
    proxy_pass http://127.0.0.1;
}

[marc.cawood.buhlergroup.com@…]

See above.

[Maxim Dounin]

Do you mean by "unescaped" that the URL is decoded from /foo%20bar to /foo bar? If so then you are confirming the incorrect behaviour. Why would anyone want the URL unencoded??

Locations are matched on _unescaped_ URIs, so "location /foo" matches both "GET /foo" and "GET /%66oo". See the ​location directive description for details.

Why is $1 unencoded and which variable can I use to get the URL *as is*?

The $1 variable contains unescaped data as you've obtained it from the unescaped URI. If you want original escaped URI as got from the client, consider using ​$request_uri instead. On the other hand, as explained above, most likely you don't need it at all. There are much simpler and more natural solutions in nginx to do what you want to do.

Note: in our scenario we are rewriting /foo%20bar to /subdir/foo%20bar downstream and not to root / and thus we are using proxy_pass with a URL and running into this documented but IMHO incorrect behaviour.

You are not using "proxy_pass with a URL", you are instead using "proxy_pass with an URL explicitly specified using variables". This is a special mode designed to be used when you know in advance the URL you want to proxy the request to. As explained above, using this mode to _modify_ existing URI might be tricky. And it certainly does not make sense to use it to remove or add a subdirectory.

If you want to add a subdirectory, consider using proxy_pass with an URI component, _without_ variables, as suggested by the documentation:

location / {
    proxy_pass http://127.0.0.1/subdir/;
}

This way requests to /foo%20bar will be passed as requests to /subdir/foo%20bar on the proxied server.

[cawoodm@…]

I think I see the complication which I forgot to explain:

I am trying to proxy /yo/lo/foo%20bar to any upstream server and path e.g. http://127.0.0.1:8888/here/foo%20bar where lo and foo%bar are variable.

Note lo is a variable which I want to IGNORE. It could be any value but foo%bar is a variable I want to pass on INCLUDING SPACES so it could be:

• /yo/lo/foobar => /here/foobar
• /yo/baz/foobar => /here/foobar
• /yo/baz/foo-Bar => /here/foo-Bar
• /yo/nginxiscool/foo bar => /here/foo%20bar * This breaks if I use $1

So the dilemma is: I have to use RegEx to ignore lo but if I use RegEx $1 causes broken URLs.

Again I ask why is $1 decoded to foo bar and how can we get it unencoded foo%20bar?

[Maxim Dounin]

Again I ask why is $1 decoded to foo bar and how can we get it unencoded foo%20bar?

Again: locations are matched on unescaped URIs. Since $1 in your case comes from location matching, you get unescaped foo bar in the $1 variable. And you use it in the proxy_pass directive, which needs properly escaped URL.

As already said in comment:5, the original escaped URI is available in the $request_uri variable, and you can obtain a part of it instead of using location matching. This might not be trivial to do properly though, as you still need to unescape things to properly remove the /yo/.../ part.

As already mentioned in comment:1, for cases when you need to apply complex modifications to URIs, most simple solution would be to use ​rewrite, which is specifically designed to modify URIs. In your particular case something like this should work:

location /yo/ {
    rewrite ^/yo/[^/]+/(.*) /here/$1 break;
    rewrite ^ /here/ break;
    proxy_pass http://127.0.0.1:8888;
}

If you have further questions on how to configure nginx, consider using ​support options available.

[cawoodm@…]

I have asked where anyone needs an unescaped $1 and you have explained where it comes from technically (because location directives work on unescaped URIs) not why it is unescaped. Who needs an unescaped variable which they cannot use in URLs?

Just because a user wants to work with unescaped URIs in the configuration does not mean that they want invalid values in a variable. I say "invalid" because an unescaped URL is an invalid URL - hardly anyone needs $1 unescaped and most people need it escaped since they are using it in a URL. Every Nginx example on the official Docs site and elsewhere uses $1 in a URL.

If you think it is right that $1 is unescaped then what can we use for an escaped version?

$_1?

Your solution provided rewrite ^/yo/[^/]+/(.*) /here/$1 break; does not work with spaces in URLs. Indeed $1 is always invalid in a URL if the URL contains any %__ escaping.

[Maxim Dounin]

Who needs an unescaped variable which they cannot use in URLs?

This is more or less philosophical question. In an ideal world where no one needs an unescaped variable there is no need for escaping, as everyone uses escaped variables instead.

In the real world escaping is needed because one needs to use characters not allowed in URLs, for example, in file names. And nginx has to work with file names. Further, variables are often used for various access control tasks - and this needs unescaped form, much like for location matching.

If you think that the real world web server might be written without using the unescaped variables, consider writing one. It would be interesting to take a look you'll solve the problems which appear when you have to work with characters which need escaping.

If you think it is right that $1 is unescaped then what can we use for an escaped version?

As of now, there is no easy way to obtain an escaped version of a particular variable (you can either use 3rd party modules such as "set-misc" or scripting modules such as embedded perl or njs; there is a feature request to simply this, #52). The recommended approach is check the context: that is, use escaped ones when you need escaped strings, and unescaped ones when you need unescaped strings.

Your solution provided rewrite ^/yo/[^/]+/(.*) /here/$1 break; does not work with spaces in URLs. Indeed $1 is always invalid in a URL if the URL contains any %__ escaping.

It looks you've failed to properly copy the example configuration provided. The rewrite directive knows that $1 is from the regular expression in the rewrite itself, and takes care of proper escaping. As previously suggested, if you have further questions on how to configure nginx, consider using ​support options available.

[JoshMcCullough@…]

I think there is a disconnect here. "cawoodm" is talking about "why is my variable unescaped" when it comes from a regex location match. Well, it should be unescaped, because you wouldn't want to have your regex include escaped characters. You'd run into constant issues with that!

So it is correct that $1 is unsecaped. Good.

However, the part that is missing here and which wasn't discussed (unless I missed it) is this: why does proxy_pass not ensure that the generated request-line is properly formatted?

Given the example in this ticket, the request-line sent to the proxy would look like this:

GET /foo bar HTTP/1.1

This is in direct violation of ​RFC 2616, which states the following:

The Request-Line begins with a method token, followed by the Request-URI and the protocol version, and ending with CRLF. The elements are separated by SP characters. No CR or LF is allowed except in the final CRLF sequence.

Request-Line = Method SP Request-URI SP HTTP-Version CRLF

The delimiter used in the request-line is a space, so "in what world" is it correct that proxy_pass doesn't at least encode spaces in the URL parameter to ensure that the request-line is HTTP-compliant?

[Maxim Dounin]

Replying to JoshMcCullough@…:

However, the part that is missing here and which wasn't discussed (unless I missed it) is this: why does proxy_pass not ensure that the generated request-line is properly formatted?

The proxy_pass expects properly escaped URI. As per the "garbage in, garbage out" concept, an incorrect URI could result in anything. So, it's certainly not a bug that an incorrect URI results in a request-line which is not RFC-compliant.

The real question is: wouldn't it be better from usability point of view to additionally scan resulting URI to make sure it doesn't contain some obvious protocol violations, such as spaces. In most cases nginx does not try to do something like this, especially at runtime, and rather does what the configuration says. See, for example, ticket #2258.

[...]

The delimiter used in the request-line is a space, so "in what world" is it correct that proxy_pass doesn't at least encode spaces in the URL parameter to ensure that the request-line is HTTP-compliant?

I would rather say that "at least encode spaces" is the worst thing that nginx can do here, since configuration mistakes as the one in the ticket will be unnoticed, resulting in incorrect handling of some other requests, such as ones with %.