Allow proxy_http_version 2.0

[mmarkus@…]

We are working on migrating the service to service calls to gRPC. We use nginx as an gateway that proxies to a pool of gRPC servers (HTTP2.0).
Is there a plan to support this feature and if so an ETA for it? This would help Square to move forward but also seems like a pretty common setting for the gRPC enabled deployments.

[Maxim Dounin]

There are no plans to implement HTTP/2 support in the proxy module in the foreseeable future, see ​detailed answer here. If you want to use nginx to balance multiple servers, consider using ​the stream module to do this.

[mmarkus@…]

Replying to mdounin:

There are no plans to implement HTTP/2 support in the proxy module in the foreseeable future, see ​detailed answer here. If you want to use nginx to balance multiple servers, consider using ​the stream module to do this.

Thanks for the quick response.
I don't think level 4 proxying would work for us, specifically in situations where a small number of clients are connected to a large pool of backend servers: each client would issue all its requests over the same TCP connection to the same backend server, potentially exhausting it.
Regarding [[[​http://mailman.nginx.org/pipermail/nginx/2015-December/049445.html|your concern on the limit on number of
simalteneous requests]]:the way I was thinking about load balancing is to have nginx dispatch different stream to different backend servers, which I think should solve the problem. (not very familiar with nginx/http2 so the idea might be way off)

[cicerocomp@…]

For those who come here and still need to make a proxy HTTP2 protocol:

​https://nghttp2.org/blog/2015/03/24/proxying-grpc-with-nghttpx/

[RoiEXLab@…]

While it might be worse when using HTTP/2 instead of HTTP/1.1 between nginx and your server, not implementing such a feature has a huge downside.
On of the features of HTTP/2 is Server-Side pushes which are therefor not supported at all when using nginx as reverse proxy.
There is no workaround for this other than simply not using nginx and connect to the server directly.

[Maxim Dounin]

HTTP/2 push has nothing to do with HTTP/2 support to upstreams, it can be easily (and likely more efficiently) implemented without using HTTP/2 to upstreams, see ​here.

[cicerocomp@…]

I do not know if it should be critical, but it should be at least major

[AlekSi@…]

Any update on this?

[cicerocomp@…]

any

[Valentin V. Bartenev]

The development of gRPC proxy is in progress. See: https://trac.nginx.org/nginx/roadmap

[mcgrawia@…]

Hi, does anyone know if the gRPC proxy will support context based routing rules with the hostname/path? I am working on a use case using Kubernetes and trying to expose many different gRPC services through an nginx ingress controller. I see it is on the roadmap but can't find out any more details of what features will be included.

Thanks!

[Maxim Dounin]

The gRPC proxy module has been committed and will be available in nginx 1.13.10.

[tonyaw@…]

Hi, now nginx can support gRPC proxy( ​https://www.nginx.com/blog/nginx-1-13-10-grpc ), but looks like it still can't support HTTP2 proxy. Since gRPC is based on HTTP2, is there any reason not to implement it, and is there any plan to implement it in the future?

[tonyaw@…]

I realized that nginx can work as HTTP2 reversed proxy actually. :-)
Looks like the grpc proxy function can work as a HTTP2 proxy.

1. The following is my configuration:

error_log stderr debug;


events {
       worker_connections  1024;
}


http {
     upstream grpcservers {
         server nghttp2servernginx-0.nghttp2-server-svc.tonyaw.svc.cluster.local.:5555;
         server nghttp2servernginx-1.nghttp2-server-svc.tonyaw.svc.cluster.local.:5555;
         server nghttp2servernginx-2.nghttp2-server-svc.tonyaw.svc.cluster.local.:5555;
     }


     server {
         listen 7777 http2;

         access_log /dev/stdout;

         location / {
             grpc_pass grpc://grpcservers;
             error_page 502 = /error502grpc;
         }

         location = /error502grpc {
             internal;
             default_type application/grpc;
             add_header grpc-status 14;
             add_header grpc-message "unavailable";
             return 204;
         }
     }
}

2. h2load works well:

[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# ./h2load  -n 1000 -c 50 -d tls.cc http://127.0.0.1:7777/
starting benchmark...
spawning thread #0: 50 total client(s). 1000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 181.99ms, 5494.75 req/s, 850.50KB/s
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 154.79KB (158500) total, 93.80KB (96050) headers (space savings 33.76%), 32.23KB (33000) data
					min         max         mean         sd        +/- sd
time for request:     3.78ms     12.20ms      8.83ms      1.33ms    76.40%
time for connect:      158us      1.74ms       895us       437us    62.00%
time to 1st byte:     4.09ms     14.98ms     11.69ms      2.07ms    70.00%
req/s           :     110.49      116.52      112.20        1.22    68.00%
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]#    

3. So can I say nginx can work as a http/2 reversed proxy? :-)
4. Also, I found an issue: nginx establishes a dedicated tcp connection to backend server for every HTTP/2 stream:

[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.249 | head -n1
tcp        0      0 10.244.0.252:53382      10.244.0.249:5555       TIME_WAIT   -
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.249 | wc -l
334
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.250 | head -n1
tcp        0      0 10.244.0.252:52096      10.244.0.250:5555       TIME_WAIT   -
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.250 | wc -l
333
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.251 | head -n1
tcp        0      0 10.244.0.252:48100      10.244.0.251:5555       TIME_WAIT   -
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# netstat -anp | grep 10.244.0.251 | wc -l
333
[root@nghttp2-client-deploy-nginx-8566b9468d-fffgw src]# 

Is it by design or it is caused by bad configuration?
If it is by design, it doesn't leverage http2 benefit to reuse a single tcp connection to transport multiple HTTP/2 streams. Will it be enhanced in the future?

[Maxim Dounin]

So can I say nginx can work as a http/2 reversed proxy? :-)

The gRPC proxy module is specifically designed to work with gRPC servers. While it can handle generic HTTP/2 traffic just fine in most cases, I wouldn't recommend using it for generic HTTP/2 proxying - because gRPC is picky/different in various subtle details. For example, gRPC can only work properly if neither request nor response buffering is used, and requires trailer header fields to be passed through, and so on. For general HTTP proxying, consider using the proxy module with HTTP/1.x instead, it is believed to be better option for this task.

If it is by design, it doesn't leverage http2 benefit to reuse a single tcp connection to transport multiple HTTP/2 streams. Will it be enhanced in the future?

As of now, multiplexing different streams in a single upstream connection is not supported. Mostly because of two reasons - a) this is a bad feature from complexity point of view, and b) it is believed to cause more harm than good when talking to backend servers, see comment:1. This is unlikely to change in the near future.

[tonyaw@…]

Thanks for your quick reply. So, nginx still doesn't have plan to support a generic HTTP/2 proxy. Right? :-)
In my case, I need a proxy to accept HTTP/2 connection, and forward the HTTP/2 traffic via another HTTP/2 connection to one backend server. So, the proxy module with HTTP/1.x can't meet the requirement. :-(

[Maxim Dounin]

See also #1875.

[dominic-p@…]

With the advent of ​HTTP desync attacks (request smuggling), it seems as though this feature could become much more useful. One of the simplest mitigations suggested for these kinds of attacks is to use HTTP/2 for proxied requests:

you can resolve all variants of this vulnerability by configuring the front-end server to exclusively use HTTP/2 to communicate to back-end systems

I realize that the gPRC module is available now, but in light of the comment below, can this issue be reopened to add full HTTP/2 support to proxy_pass?

The gRPC proxy module is specifically designed to work with gRPC servers. While it can handle generic HTTP/2 traffic just fine in most cases, I wouldn't recommend using it for generic HTTP/2 proxying...

[Maxim Dounin]

Replying to dominic-p@…:

With the advent of ​HTTP desync attacks (request smuggling), it seems as though this feature could become much more useful. One of the simplest mitigations suggested for these kinds of attacks is to use HTTP/2 for proxied requests:

you can resolve all variants of this vulnerability by configuring the front-end server to exclusively use HTTP/2 to communicate to back-end systems

While HTTP/2 might appear to mitigate some of the attacks due to using different framing, it is important to understand that HTTP/2 introduces a wide range of his own vulnerabilities, and these might be worse that the ones you are trying to mitigate by using HTTP/2. Moreover, switching to HTTP/2 is certainly not a solution as long as HTTP/1.x connections remain: there is still possibility of HTTP desync / request smuggling vulnerabilities being present. That is, while switching to HTTP/2 might work in some isolated environments, this is not going to be a large-scale solution.

Further, it is believed that nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so). If there are concerns that such attacks might still be an issue, improving this mitigations might be a better idea, at least from the "large-scale" perspective.

[dominic-p@…]

Replying to Maxim Dounin:

If there are concerns that such attacks might still be an issue, improving this mitigations might be a better idea, at least from the "large-scale" perspective.

Thanks for the quick, and in-depth reply. I appreciate it. It sounds like nginx is already doing a lot to defeat desync attacks and adding HTTP/2 may not be the best solution. So, it's probably best to leave this closed.