Custom Query (2296 matches)

Actually, this defect is existed from v1.4.4 to v1.5.19

This defect is easy to reproduce. I use Nginx as Websocket reverse-proxy, when the ngnix connected upstream server, and upstream sever reject the connection after sending the error message because some conditions not satisfied, then, Ngnix worker process crashed.

gdb shows the crash information as below:

2628        if (upstream->read->timedout || upstream->write->timedout) {

(gdb) p upstream
$1 = (ngx_connection_t *) 0x0 <---- NULL Pointer

(gdb) bt
#0  ngx_http_upstream_process_upgraded (r=0x1808f90, from_upstream=0, do_write=1) at src/http/ngx_http_upstream.c:2628
#1  0x000000000044a37f in ngx_http_upstream_upgrade (r=0x1808f90, u=0x180a1b0) at src/http/ngx_http_upstream.c:2566
#2  ngx_http_upstream_send_response (r=0x1808f90, u=0x180a1b0) at src/http/ngx_http_upstream.c:2213
#3  ngx_http_upstream_process_header (r=0x1808f90, u=0x180a1b0) at src/http/ngx_http_upstream.c:1735
#4  0x00000000004474fa in ngx_http_upstream_handler (ev=<value optimized out>) at src/http/ngx_http_upstream.c:977
#5  0x000000000042c547 in ngx_epoll_process_events (cycle=<value optimized out>, timer=<value optimized out>, flags=<value optimized out>)
    at src/event/modules/ngx_epoll_module.c:691
#6  0x0000000000425593 in ngx_process_events_and_timers (cycle=0x1804f80) at src/event/ngx_event.c:248
#7  0x000000000042b1c9 in ngx_worker_process_cycle (cycle=0x1804f80, data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:816
#8  0x0000000000429ab4 in ngx_spawn_process (cycle=0x1804f80, proc=0x42b0f6 <ngx_worker_process_cycle>, data=0x0, name=0x49b766 "worker process", respawn=-3)
    at src/os/unix/ngx_process.c:198
#9  0x000000000042a700 in ngx_start_worker_processes (cycle=0x1804f80, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:364
#10 0x000000000042b707 in ngx_master_process_cycle (cycle=0x1804f80) at src/os/unix/ngx_process_cycle.c:136
#11 0x0000000000410cbc in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:407
(gdb)

after debugging and looked into the source code, I found the following code: (we can see the ngx_http_upstream_process_upgraded() will be invoked twice )

​http://trac.nginx.org/nginx/browser/nginx/src/http/ngx_http_upstream.c#L2557

2557	    if (u->peer.connection->read->ready
2558	        || u->buffer.pos != u->buffer.last)
2559	    {
2560	        ngx_http_upstream_process_upgraded(r, 1, 1);
2561	    }
2562	
2563	    ngx_http_upstream_process_upgraded(r, 0, 1);
2564	}

Take a look the function ngx_http_upstream_process_upgraded(), we can see if the upstream connection closed, then the ngx_http_upstream_finalize_request() will be invoked.

​http://trac.nginx.org/nginx/browser/nginx/src/http/ngx_http_upstream.c#L2738

2738	    if (ngx_handle_read_event(upstream->read, 0) != NGX_OK) {
2739	        ngx_http_upstream_finalize_request(r, u, NGX_ERROR); <--- Free the memory
2740	        return;
2741	    }

The ngx_http_upstream_finalize_request() function will shutdown, close and free the memory of peer connection.

So, we can see the ngx_http_upstream_process_upgraded() will be called twice, but the second one hasn't any guard to check NULL pointer.

After check the source code change, found the following message:

​http://trac.nginx.org/nginx/changeset/5532/nginx

r5500	 r5532	 
2561	2561	    } 
2562	2562	 
2563	 	    if (c->read->ready 
2564	 	        || r->header_in->pos != r->header_in->last) 
2565	 	    { 
2566	 	        ngx_http_upstream_process_upgraded(r, 0, 1); 
2567	 	    } 
 	2563	    ngx_http_upstream_process_upgraded(r, 0, 1); 
2568	2564	} 
2569	2565	

I think we should rollback that change and reconsider it.

Thanks, Hao Chen

changeset 6181:6893a1007a7c ("OCSP stapling: avoid sending expired responses (ticket #425)"), included in 1.9.2, broke stapling with ssl_stapling_file (= no OCSP response is being stapled, despite that configuration directive being present).

The problems seems to lie here: 1.24 - if (staple->staple.len) { 1.25 + if (staple->staple.len 1.26 + && staple->valid >= ngx_time()) 1.27 + { That's from ngx_ssl_certificate_status_callback().

However, staple->valid is only set by ngx_ssl_stapling_ocsp_handler(), which is clearly only called when online stapling is being used.

Hi,

I have a very simple proxy config:

http {

proxy_cache_path /var/www/cache levels=1:2 keys_zone=s3-images-cache:50m inactive=1M max_size=1000m; proxy_temp_path /var/www/cache/tmp;

server {

listen 80; server_name images.example.com;

location / {

proxy_cache s3-images-cache; proxy_cache_key $scheme$proxy_host$uri$is_args$args; proxy_cache_bypass $http_purge_cache; proxy_cache_valid any 1y; proxy_pass ​http://images-example.s3.amazonaws.com; add_header X-Cache $upstream_cache_status;

proxy_intercept_errors on; error_page 404 = @no_image;

}

location @no_image {

return 403;

}

}

}

Now follow me here:

1. Let's request /image.jpg.
2. Request is sent to proxy for /image.jpg (does not exists yet).
3. Backend responds with 404.
4. "proxy_intercept_errors on" kicks in and "error_page 404 = @no_image" is called.
5. Nginx returns 403.
6. Do another request for same image and see that "X-Cache: HIT" is set. We are clearly hitting cache.

But, if we check /var/www/cache/ folder at this time we will see that there is no cache item created for this request. So does it mean Nginx keeps the cache for it in memory and forgot to write to file?

7. Let's upload /image.jpg to backend.
8. Now do "PURGE-CACHE: 1" request to that image. We see that now we get the image instead of 403. Good.

If we check /var/www/cache/ we will see that the cache file is now finally created for this request. Also good.

9. Now here is the problem: lets request /image.jpg again.
10. Nginx returns 403 with "X-Cache: HIT". Why? So it's hitting the cache but returning something else not what is in /var/www/cache folder?? How?

My only explanation to this is it seems that Nginx is caching the response in memory and doesn't write to file when we are hitting error with our custom error_page in the proxied responses. Furthermore when using proxy_cache_bypass it does not overwrite in-memory cache, so that subsequent requests to the same item will be using old cache which is stored in memory and not the new one created in the cache folder.