nginx does not pass metadata to grpc server

[Jürgen De Commer]

nginx is configured with the most basic grpc proxy config. This is the only configuration in use:

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    server {
        listen 9080 http2;
        location / {
            grpc_pass localhost:5000;
        }
    }
}

I have a grpc client that adds metadata to its outgoing context:

func testDataInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
	traceID := uuid.New().String()
	ctx = metadata.AppendToOutgoingContext(ctx,
		"trace.id", traceID,
		"trace.application", "test-client",
		"auth.jwt", xxxxx)

	return invoker(ctx, method, req, reply, cc, opts...)
}

On the server side, this metadata is logged by an interceptor.

	md, ok := metadata.FromIncomingContext(ctx)
	if !ok {
		log.Printf("Attempt to execute %s without providing metadata", info.FullMethod)
		return nil, status.Error(codes.Unauthenticated, "Expected metadata but it was not present in the context")
	}
	log.Println("Context metadata", md)

If the client is communicating directly with the server, the server has access to the metadata, producing following log:

2018/05/22 16:56:36 map[:authority:[localhost:5000] content-type:[application/grpc] user-agent:[grpc-go/1.12.0] trace.id:[7f0d8e18-654d-4b75-bae5-0e3f4ea33058] trace.application:[test-client] auth.jwt:[xxxxxxxx]]

If the client is communicating with nginx, then this information is no longer present when the request reaches the grpc server:

2018/05/22 16:57:10 Context metadata map[:authority:[localhost:5000] content-type:[application/grpc] user-agent:[grpc-go/1.12.0]]

Same behaviour on:

vagrant@scaler-1:~$ uname -a
Linux scaler-1 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
vagrant@scaler-1:~$ nginx -V
nginx version: nginx/1.14.0
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

[Maxim Dounin]

The problem is that headers you add in your code have dots (.) in them. As such, these headers are ignored by nginx by default. You may either change header names to something without dots, or instruct nginx to accept arbitrary header names by using the ​ignore_invalid_headers directive:

ignore_invalid_headers off;

Note though that allowing arbitrary header names might result security issues.

[Jürgen De Commer]

That was fast, thanks!
Changed the metadata keys and it works like a charm.