Stalled connection when using `aio_write on`

[aminvakil@…]

/etc/nginx/nginx.conf

worker_rlimit_nofile 40000;

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    use epoll;
}

http {
    server_tokens off;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log  /var/log/nginx/access.log main buffer=2048k flush=10s;

    open_file_cache          max=10000 inactive=5m;
    open_file_cache_valid    2m;
    open_file_cache_min_uses 2;
    open_file_cache_errors   off;

    sendfile on;
    sendfile_max_chunk 512k;
    tcp_nopush  on;
    tcp_nodelay on;
    output_buffers 2 8k;
    keepalive_timeout  15;
    http2_idle_timeout 1m;

    # SSL
    ssl_session_timeout 8h;
    ssl_session_cache shared:SSL:100m;   # one megabyte can store about 4000 sessions
    ssl_session_tickets off;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/example.org.conf

upstream cdns {
    server cdn1.example.org;
    keepalive 20;
}


server {
    listen 80 reuseport;
    listen 443 ssl http2 reuseport;
    server_name cache.example.org;

    ssl_certificate /etc/nginx/ssl/example.org.pem;
    ssl_certificate_key /etc/nginx/ssl/example.org.key;
        
    root /usr/share/nginx/html/;
    aio threads;
    aio_write on;

    location /apk-files/ {

        proxy_pass http://cdns;

    }
}

Then using http2 to connect to server gets stalled like this about 90% of times.

curl -LO "https://example.org/samplefile" -vvvvv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 1.1.1.1:443...
* Connected to example.org (1.1.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4766 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.example.org
*  start date: Dec 14 11:59:52 2020 GMT
*  expire date: Dec 14 09:32:52 2021 GMT
*  subjectAltName: host "cache.example.org" matched cert's "*.example.org"
*  issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Domain Validation CA SHA2
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x801cbac360)
} [5 bytes data]
> GET /samplefile HTTP/2
> Host: example.org
> user-agent: curl/7.79.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200 
< server: nginx
< date: Mon, 20 Sep 2021 08:53:41 GMT
< content-type: application/octet-stream
< content-length: 40703269
< last-modified: Mon, 08 Feb 2021 08:13:22 GMT
< etag: "6020f2a2-26d1525"
< accept-ranges: bytes
< 
{ [7942 bytes data]
  0 38.8M    0 73478    0     0   5165      0  2:11:20  0:00:14  2:11:06     0

I couldn't reproduce the issue with curl --http1.1 or after commenting aio_write on.