proxy_protocol in virtual hosts

[Gaurav Purandare]

When multiple virtual hosts listen to proxy_protocol as follows:

listen 80 proxy_protocol; #Port 80 may differ per virtual host

Its been observed that, very first virtual host defined in configuration file throws "400 - Bad Request" error for all of the requests received. And, consecutive virtual hosts tend to work normally.

A temporary solution for this issue is, to define a separate default virtual host which listens to proxy_protocol, like below:

server {
listen       80 proxy_protocol;
server_name _; #Default virtual host
}

Its not clear why only very first virtual host throws "400 - Bad Request" error.

[Roman Arutyunyan]

Could you provide more details or even a simple small config for the issue?
With my simple configuration I don't see this problem.
Please watch error.log for errors as well.

[Gaurav Purandare]

@arut, did you check the configuration with proxy_protocol enabled on AWS ELB and nginx running behind it?

[Roman Arutyunyan]

No, I tested it manually. Does this issue occur only behind AWS ELB?

[Gaurav Purandare]

I tested it several times with nginx + AWS ELB - got this issue.

However, you need to enable proxy protocol on ELB explicitly.
Reference: ​http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html

You can find the complete use-case here:
​http://www.raweng.com/blog/2014/11/11/websockets-on-aws-elb

Thanks!

[Roman Arutyunyan]

I tried nginx+aws/elb, but it's still fine. Could you provide error log?

error_log logs/error.log debug;

[Andrei Belov]

We've just checked with a similar configuration using live ELB with proxy_protocol turned on, and haven't noticed anything wrong.

We've seen a couple of 400 errors shortly after applying proxy_protocol policy to the ELB, while nginx has been using old configuration (listen on 80/tcp without "proxy_protocol" parameter turned on) and ELB was using HTTP health checks, but this was expected.

After adding "proxy_protocol" to the listen directive, reloading nginx configuration and switching ELB health checks to use TCP we were not able to see 400 errors.

Is there a reliable way to reproduce your issue?

How your ELB health checks are configured?

[Gaurav Purandare]

@arut, @Andrei - Have you guys tried with configuration that involves default virtual host? The problem I faced was totally specific to very first virtual host. I believe it can be reproduced if you do not define default virtual host, and check the logs for very first virtual host listening to proxy_protocol on nginx instance running behind ELB.

@Andrei, you might want to try this out in non-production environment, instead of production.

When default virtual host is defined, all '400 - Bad requests' errors are seen in that default virtual host. i.e. when default virtual host is defined, I can use rest of the virtual hosts just fine.

I'll also check on my side, and share the error logs once I get chance.

@Andrei, yes - TCP health checks for ELB.

Thanks for your help!

[Andrei Belov]

I tried with a number of virtual hosts defined with valid server names - wasn't able to reproduce.

One note though - in the post you've mentioned there is an error in the following configuration snippet:

#Virtual host accepting SSL traffic

  server {
  listen       81 proxy_protocol;
  ...

If you're going to handle SSL on 81/tcp port (I assume you are, looking at the ELB policies), you should add the "ssl" parameter to the "listen" directive, and also specify paths to certificate and key, e.g.:

#Virtual host accepting SSL traffic

  server {
  listen       81 proxy_protocol ssl;
  ssl_certificate your_cert.pem;
  ssl_certificate_key your_cert.key;
  ...

Otherwise, you'll get 400 error and some garbage in access log on every HTTPS request to ELB endpoint.

To summarize - we still do not have a reliable way to reproduce the issue you described.

Please share your relevant error log entries and the whole configuration to make us able to see what happen.

[Gaurav Purandare]

@Andrei, sure, will share the error log and configuration.

No, port 81 doesn't use SSL. SSL termination occurs at ELB on port 443 and requests are forwarded to instance port 81.

[Andrei Belov]

Unable to reproduce without the entire configuration from the requestor.

[Felix Bünemann]

I can reproduce the problem. It happens if there are multiple SSL vHosts using SNI and not all of them have proxy_protocol enabled.