Odd variable sets with proxy_pass

[Shinrai@…]

nginx.conf:

	server {
        listen       ****************:80;
        server_name  cdn.cldmv.net;

		set $fastcgihost ****************;
		set $fastcgiport ****************;
		set $fastcgihostname $fastcgihost:$fastcgiport;
		set $chrootdir ****************/cdn.cldmv.net/html;
		set $globalchroot ****************;
		fastcgi_intercept_errors on;
		
		include ****************/templates/base/setup.conf;

		root   $globalchroot****************/cdn.cldmv.net/html;

        access_log  ****************/cdn.cldmv.net/logs/access.log  main;
        error_log  ****************/cdn.cldmv.net/logs/error.log;

		index  index.php index.html index.htm;
		autoindex on;
		
		#fancyindex on;              # Enable fancy indexes.
		#fancyindex_exact_size off;  # Output human-readable file sizes.
		#fancyindex_ignore
		#concat on;
		#concat_max_files 30;
		
		include ****************/templates/software/vhost.conf;
    }
	
	server {
        listen       ****************:80;
        server_name  cdn2.cldmv.net;

		set $fastcgihost ****************;
		set $fastcgiport ****************;
		set $fastcgihostname $fastcgihost:$fastcgiport;
		set $chrootdir ****************/cdn.cldmv.net/html;
		set $globalchroot ****************;
		fastcgi_intercept_errors on;
		
		include ****************/templates/base/setup.conf;

		root   $globalchroot****************/cdn.cldmv.net/html;

        access_log  ****************/cdn.cldmv.net/logs/access.log  main;
        error_log  ****************/cdn.cldmv.net/logs/error.log;

		index  index.php index.html index.htm;
		autoindex on;
		
		#fancyindex on;              # Enable fancy indexes.
		#fancyindex_exact_size off;  # Output human-readable file sizes.
		#fancyindex_ignore
		#concat on;
		#concat_max_files 30;
		set $_robots_index "disabled";
		set $_seo_process_trail "disabled";
		
		location /cldmv/jquery/ {
			add_header X-DIR-Test "success";
			set $_robots_index "enabled";
			set $_seo_process_trail "enabled";
		}
		
		include ****************/templates/software/vhost.conf;
    }
	
	server {
        listen       ****************:80;
        server_name  cdn3.cldmv.net;

		set $fastcgihost ****************;
		set $fastcgiport ****************;
		set $fastcgihostname $fastcgihost:$fastcgiport;
		set $chrootdir ****************/cdn.cldmv.net/html;
		set $globalchroot ****************;
		fastcgi_intercept_errors on;
		
		include ****************/templates/base/setup.conf;

		root   $globalchroot****************/cdn.cldmv.net/html;

        access_log  ****************/cdn.cldmv.net/logs/access.log  main;
        error_log  ****************/cdn.cldmv.net/logs/error.log;

		index  index.php index.html index.htm;
		autoindex on;
		
		#fancyindex on;              # Enable fancy indexes.
		#fancyindex_exact_size off;  # Output human-readable file sizes.
		#fancyindex_ignore
		#concat on;
		#concat_max_files 30;
		set $_robots_index "disabled";
		set $_seo_process_trail "disabled";
		
		location /cldmv/jquery/ {
			add_header X-DIR-Test "success";
			set $_robots_index "enabled";
			set $_seo_process_trail "enabled";
			include ****************/templates/base/base/seo.conf;
		}
		
		include ****************/templates/software/vhost.conf;
    }

All of the above configs works as intended.

​http://cdn3.cldmv.net/cldmv/jquery/ produces the headers:

X-DIR-Test:success
X-Robots-Parse:index
X-Robots-Tag:all
X-SEO-Test:get
X-SEO-Trail:get_process_seo_trail

and include /templates/base/base/seo.conf; is not needed.

​http://cdn2.cldmv.net/cldmv/jquery/ produces the headers:

X-DIR-Test:success
X-Robots-Parse:index
X-Robots-Tag:all
X-SEO-Test:get
X-SEO-Trail:get_process_seo_trail

​http://cdn2.cldmv.net/cldmv/ produces the headers:

X-Robots-Parse:no_index
X-Robots-Tag:noindex, noarchive, nosnippet, nofollow
X-SEO-Test:get
X-SEO-Trail:get_skip_seo_trail

However when the above cdn.cldmv.net nginx.conf block is paired with a proxy_pass block like so:

		location ~ ****************/nas1(.*)$ {
			set $_seo_process_trail "disabled";
			set $_robots_index "disabled";
			access_log  ****************/logs/nas1-access.log  main;
			error_log  ****************/logs/nas1-error.log;
			#proxy_buffering on;
			proxy_connect_timeout 3600;
			proxy_send_timeout 3600;
			proxy_read_timeout 3600;
			proxy_redirect off;
			proxy_max_temp_file_size 0;
			proxy_buffer_size 16k;
			proxy_buffers 64 16k;
			proxy_busy_buffers_size 16k;
			#proxy_max_temp_file_size 2048m;
			proxy_temp_file_write_size 32k;
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-Proto $scheme;
			proxy_set_header X-Url-Scheme $scheme;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Port $server_port;
			proxy_set_header X-Forwarded-Server $host;
			proxy_set_header X-Forwarded-Host $host;
			
			proxy_buffering off;
			proxy_request_buffering off;
			proxy_http_version 1.1;
			proxy_set_header Connection "";
			
			proxy_pass http://****************$1;
			#include ****************/templates/base/base/seo.conf;
			#include ****************/templates/base/base/seo/x-robots.conf;
		}

Both

	set $_seo_process_trail "disabled";
	set $_robots_index "disabled";

do not apply and I'm left with the following headers:

x-robots-parse:index
x-robots-tag:all
x-seo-test:get
x-seo-trail:get_process_seo_trail

However if I add

add_header X-Robots-Tag "noindex, noarchive, nosnippet, nofollow";

to the proxy_pass block than none of the headers apply, not even my debug ones.

Though when accessing /nas1 instead of /nas1/ I do get:

x-robots-parse:index
x-robots-tag:all
x-seo-test:get
x-seo-trail:get_process_seo_trail

and a redirect to /nas1/ via the code in the /templates/base/base/seo.conf

So this bug is two fold.

One being that if I access /nas1/url/admin none of my proxy pass block is applied and it skips over it and applies the base server block config. Thus redirecting me to /nas1/url/admin/. Once at /nas1/url/admin/ I get the expected headers IF and only IF I specify an add_header directive in the proxy_pass block. If add_header is NOT specified the headers which should have been parsed through the variables set in the proxy_pass block are not parsed correctly. Even though the variables being specified in the proxy_pass block are in fact set and sent out via add_header.

Expected behavior is that the URL matches the proxy_pass block and thus the set variables are applied. Then the remaining config is ran and the headers are sent out as the conditional headers. As shown to work in the cdn* configs.

[Maxim Dounin]

In the provided configuration neither $_seo_process_trail nor $_robots_index are used anywhere. As such, claiming that these variables are not set based on some returned headers is not correct. Moreover, the only add_header directives in the provided configuration are add_header X-DIR-Test "success"; in /cldmv/jquery/ locations.

Likely your configuration tries to add additional headers based on the variables mentioned in include files which seems to be used in some parts of the configuration. You haven't showed what these includes contain though, and most likely the problem is that you've failed to properly write these include files and/or failed to properly include them when needed.

If you still think there is a bug in nginx, please provide full self-contained configuration example which demonstrates the problem. If you need help with configuring nginx, consider using ​support options available.

[Shinrai@…]

Likely you did not read the post. I clearly show that when the proxy_pass block is given a add_header directive no other headers will be applied. However when it's not it will process everything else inside include /templates/software/vhost.conf;

and most likely the problem is that you've failed to properly write these include files and/or failed to properly include them when needed.

Sure, there may be an issue with my config. However the problem at hand is not caused by it. The two problems exist. One set variables is not being honored in a consistent fashion and proxy_pass is not honoring server block level directives consistently.

A proxy_pass should operate in one of two fashions. Either execute all other server level directives or not. I clearly showed by adding an add_header directive in the proxy_pass block it causes nginx to differ in how it handles a proxy_pass request.

To re-itterated, when the block is left alone and does not have an add_header directive the directives in the server block will execute. However variables used to create other variables and send headers do not operate as expected. IE _robots_index set to disabled will send that header through showing disabled. However the variable _robots_noindex_parse will generate as if _robots_index was set to enabled.

					###include ******/templates/base/base/seo/x-robots.conf;

			
						set $xrobots "all";
						set $_robots_noindex_parse "index";
						
						if ($_robots_index = disabled) {
							set $_robots_noindex_parse "no_index";
						}
						
						if ($server_name ~* "(projects|dev)\..*\..*$") {
							set $_robots_noindex_parse "${_robots_noindex_parse}_dev";
						}
						
						if ($_robots_noindex_parse = "no_index") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "no_index_dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "index_dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						
						add_header X-Robots-Tag "${xrobots}";
						add_header X-Robots-INDEX "${_robots_index}";
						add_header X-Robots-Parse "${_robots_noindex_parse}";

Now when you put an add_header into the proxy_pass block no other headers will come through. IE all the test headers set in my configuration no longer are being sent to the end user.

Here is the relevant FULL self contained config for your easy copy pasting.

	server {
        listen       ***.***.***.***:80;
        listen       ***.***.***.***:443 ssl http2;
        server_name  company.net www.company.net *.company.net;

		set $fastcgihost 127.0.0.1;
		set $fastcgiport ****;
		set $fastcgihostname $fastcgihost:$fastcgiport;
		set $chrootdir /www/company.net/html;
		set $globalchroot /data/clients/****;
		#fastcgi_intercept_errors on;
		fastcgi_intercept_errors off;
		
		#include ******/templates/base/setup.conf;
			charset utf-8;
			client_max_body_size 150m;
			client_body_buffer_size 128k;

			index  index.php index.html index.htm;

			set $fastcgi_psuedo_script_name false;
			set $_robots_index enabled;
			set $_seo_process_trail  enabled;

		root   $globalchroot/www/company.net/html;

        access_log  ******/company.net/logs/access.log  main;
        error_log  ******/company.net/logs/error.log;
		
		location ~/tools/nas1(.*)$ {
			set $_seo_process_trail  disabled;
			set $_robots_index disabled;
			access_log  ******/company.net/logs/nas1-access.log  main;
			error_log  ******/company.net/logs/nas1-error.log;
			#add_header X-Robots-Tag none;
			proxy_connect_timeout 3600;
			proxy_send_timeout 3600;
			proxy_read_timeout 3600;
			proxy_redirect off;
			proxy_max_temp_file_size 0;
			proxy_buffer_size 16k;
			proxy_buffers 64 16k;
			proxy_busy_buffers_size 16k;
			proxy_temp_file_write_size 32k;
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-Proto $scheme;
			proxy_set_header X-Url-Scheme $scheme;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Port $server_port;
			proxy_set_header X-Forwarded-Server $host;
			proxy_set_header X-Forwarded-Host $host;
			
			proxy_buffering off;
			proxy_request_buffering off;
			proxy_http_version 1.1;
			proxy_set_header Connection "";
			
			proxy_pass http://10.4.0.1$1;
		}

		###include ******/templates/software/vhost.conf;
			###include ******/templates/base/base.conf;
				###include ******/templates/base/base/seo.conf;
					###include ******/templates/base/base/seo/index-trailing-slash.conf;
						set $_seo_test  "";
						if ($request_method = GET) {
							set $_seo_test "${_seo_test}get";
						}
						if ($request_uri ~* "^(.+)?/index.(php|html)(.+)?$") {
							set $_seo_test  "${_seo_test}_index";
						}
						if ($request_uri ~* "^(.+)?/index.(php|html)/(.+)?$") {
							set $_seo_test  "${_seo_test}_index_trail";
						}
						if ($ajaxrequest) {
							set $_seo_test  "${_seo_test}_ajax";
						}
						if ($_seo_test = "get_index") {
							rewrite ^(.+)index.(php|html)(.+)?$ $1$3 redirect;
						}
						if ($_seo_test = "get_index_trail") {
							rewrite ^(.+)index.(php|html)/(.+)?$ $1$3 redirect;
						}
						add_header X-SEO-Test "${_seo_test}";
					###include ******/templates/base/base/seo/trailing-slash.conf;
						set $_seo_trail  "";
						if ($request_method = GET) {
							set $_seo_trail "${_seo_trail}get";
						}
						if ($_seo_process_trail  = disabled) {
							set $_seo_trail "${_seo_trail}_skip_seo_trail";
						}
						if ($_seo_process_trail  = enabled) {
							set $_seo_trail "${_seo_trail}_process_seo_trail";
						}
						
						add_header X-SEO-Skip-Trail "${_seo_process_trail }";
						add_header X-SEO-Trail "${_seo_trail}";
						
						if ($_seo_trail = "get_process_seo_trail") {
							rewrite ^([^.\?]*[^/])$ $1/ permanent;
						}
					###include ******/templates/base/base/seo/x-robots.conf;

			
						set $xrobots "all";
						set $_robots_noindex_parse "index";
						
						if ($_robots_index = disabled) {
							set $_robots_noindex_parse "no_index";
						}
						
						if ($server_name ~* "(projects|dev)\..*\..*$") {
							set $_robots_noindex_parse "${_robots_noindex_parse}_dev";
						}
						
						if ($_robots_noindex_parse = "no_index") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "no_index_dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "index_dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						if ($_robots_noindex_parse = "dev") {
							set $xrobots "noindex, noarchive, nosnippet, nofollow";
						}
						
						add_header X-Robots-Tag "${xrobots}";
						add_header X-Robots-INDEX "${_robots_index}";
						add_header X-Robots-Parse "${_robots_noindex_parse}";


			location / {
				###include ******/templates/base/routes/routeindex.conf;
					if (!-e $request_filename) {
						rewrite ^([^?]*)$ /?_route_=$1 last;
					}
					
					try_files $uri $uri/ $uri.php =404;
			}

			###include ******/templates/base/php-basic.conf;
				# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
				#
				location ~ \.php$ {
					#try_files $uri /index.php =404;
					try_files $uri $uri/ $uri.php =404;
					###include ******/templates/base/php.conf;
						fastcgi_bind $fastcgihost;
						fastcgi_pass $fastcgihostname;
						fastcgi_index index.php;
						set $fastcgi_real_script_name $fastcgi_script_name;
						if ($fastcgi_psuedo_script_name != false) {
							set $fastcgi_real_script_name $fastcgi_psuedo_script_name;
						}
						set $fastcgi_full_script_name $chrootdir$fastcgi_real_script_name;
						fastcgi_param SCRIPT_FILENAME $fastcgi_full_script_name;
						fastcgi_param PHP_SELF $fastcgi_real_script_name;
						fastcgi_param PATH_INFO $fastcgi_real_script_name;
						fastcgi_param USER www;
						fastcgi_param HOME /www;
						include fastcgi_params;
						#fastcgi_intercept_errors on;
						fastcgi_next_upstream error timeout invalid_header http_500;
						fastcgi_next_upstream_tries 2;
						fastcgi_param DOCUMENT_ROOT $chrootdir;
						fastcgi_param SERVER_SOFTWARE    "nginx/$nginx_version - COMPANY";
						fastcgi_param COMPANY_REQUEST_BASEURI $scheme://$server_name$request_dir;
						fastcgi_param GEOIP_COUNTRY_CODE $geoip_city_country_code; 
						fastcgi_param GEOIP_COUNTRY_CODE3 $geoip_city_country_code3;
						fastcgi_param GEOIP_COUNTRY_NAME $geoip_city_country_name;
						fastcgi_param GEOIP_REGION $geoip_region;
						fastcgi_param GEOIP_CITY $geoip_city;
						fastcgi_param GEOIP_POSTAL_CODE $geoip_postal_code;
						fastcgi_param GEOIP_CITY_CONTINENT_CODE $geoip_city_continent_code;
						fastcgi_param GEOIP_LATITUDE $geoip_latitude;
						fastcgi_param GEOIP_LONGITUDE $geoip_longitude;
						fastcgi_param GEOIP_DMA_CODE $geoip_dma_code;
						fastcgi_param GEOIP_AREA_CODE $geoip_area_code;
						#fastcgi_param X-Real-IP       $remote_addr;
						fastcgi_param X-Real-IP $remote_addr;
						fastcgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
						set $opcache 1;
						if ($server_name ~ ^dev.) {
							set $opcache 0;
						}
						set $phpvalue "";
						if ($opcache = 0) {
							set $phpvalue "opcache.enable=$opcache";
						}
						fastcgi_param PHP_VALUE $phpvalue;
						
						# Fix for https://httpoxy.org/ vulnerability
						fastcgi_param HTTP_PROXY "";
						
						if ($cache_server = false) {
							set $cache_server "127.0.0.1:8080";
						}
						if ($cache_appid = false) {
							set $cache_appid "false";
						}
						
						fastcgi_param COMPANY_MEMCACHE_SERVER $cache_server;
						fastcgi_param COMPANY_MEMCACHE_APPID $cache_appid;
				}
    }

[Maxim Dounin]

Replying to Shinrai@…:

A proxy_pass should operate in one of two fashions. Either execute all other server level directives or not. I clearly showed by adding an add_header directive in the proxy_pass block it causes nginx to differ in how it handles a proxy_pass request.

Adding an add_header directive to the proxy_pass location block will cause all add_header directives defined at previous levels to not be inherited into this location block. Quoting ​docs:

These directives are inherited from the previous level if and only if there are no add_header directives defined on the current level.

This is how inheritance in nginx configuration works: if do not define a directive at a given level, it's inherited from previous levels. And if you do define it - it is not inherited from previous levels.

If you want to add additional headers to the add_header directives defined at previous levels, you have to repeat these add_header directives in the location in question.

To re-itterated, when the block is left alone and does not have an add_header directive the directives in the server block will execute. However variables used to create other variables and send headers do not operate as expected. IE _robots_index set to disabled will send that header through showing disabled. However the variable _robots_noindex_parse will generate as if _robots_index was set to enabled.

When "the block is left alone", you redefine $_seo_process_trail and $_robots_index variables, but other variables are already set during execution of the rewrite module directives define at the server level. See ​docs for information on how rewrite module directives are executed.

Hint: what you are trying to do can be better done using ​the map module. If you need help with configuring nginx, consider using ​​support options available.