Opened 4 years ago
Last modified 4 years ago
#2248 closed defect
Stalled connection when using `aio_write on` — at Version 1
| Reported by: | Owned by: | ||
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | nginx-core | Version: | 1.19.x |
| Keywords: | aio aio_write proxy | Cc: | aminvakil@… |
| uname -a: | Linux hostname 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux | ||
| nginx -V: |
nginx version: nginx/1.20.1
built by gcc 8.3.0 (Debian 8.3.0-6) built with OpenSSL 1.1.1d 10 Sep 2019 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.20.1/debian/debuild-base/nginx-1.20.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' |
||
Description (last modified by )
Connection gets stalled most times and does not continue downloading files most of times, here is the configuration files:
/etc/nginx/nginx.conf
worker_rlimit_nofile 40000;
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log error;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
}
http {
server_tokens off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log main buffer=2048k flush=10s;
open_file_cache max=10000 inactive=5m;
open_file_cache_valid 2m;
open_file_cache_min_uses 2;
open_file_cache_errors off;
sendfile on;
sendfile_max_chunk 512k;
tcp_nopush on;
tcp_nodelay on;
output_buffers 2 8k;
keepalive_timeout 15;
http2_idle_timeout 1m;
# SSL
ssl_session_timeout 8h;
ssl_session_cache shared:SSL:100m; # one megabyte can store about 4000 sessions
ssl_session_tickets off;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
include /etc/nginx/conf.d/*.conf;
}
/etc/nginx/conf.d/example.org.conf
upstream cdns {
server cdn1.example.org;
keepalive 20;
}
server {
listen 80 reuseport;
listen 443 ssl http2 reuseport;
server_name cache.example.org;
ssl_certificate /etc/nginx/ssl/example.org.pem;
ssl_certificate_key /etc/nginx/ssl/example.org.key;
root /usr/share/nginx/html/;
aio threads;
aio_write on;
location /apk-files/ {
proxy_pass http://cdns;
}
}
Then using http2 to connect to server gets stalled like this about 90% of times.
curl -LO "https://example.org/samplefile" -vvvvv
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 1.1.1.1:443...
* Connected to example.org (1.1.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4766 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.example.org
* start date: Dec 14 11:59:52 2020 GMT
* expire date: Dec 14 09:32:52 2021 GMT
* subjectAltName: host "cache.example.org" matched cert's "*.example.org"
* issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Domain Validation CA SHA2
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x801cbac360)
} [5 bytes data]
> GET /samplefile HTTP/2
> Host: example.org
> user-agent: curl/7.79.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200
< server: nginx
< date: Mon, 20 Sep 2021 08:53:41 GMT
< content-type: application/octet-stream
< content-length: 40703269
< last-modified: Mon, 08 Feb 2021 08:13:22 GMT
< etag: "6020f2a2-26d1525"
< accept-ranges: bytes
<
{ [7942 bytes data]
0 38.8M 0 73478 0 0 5165 0 2:11:20 0:00:14 2:11:06 0
I couldn't reproduce the issue with curl --http1.1 or after commenting aio_write on.
Note:
See TracTickets
for help on using tickets.
