limit_rate and proxy_limit_rate broken from 1.14

[Ondřej Nový]

Hello,

I have a problem with directives limit_rate and proxy_limit_rate in nginx version 1.14.1. Rate limited speed is much lower than configured one. These directives works fine with older version of nginx (1.12.2). Problem looks related to usage of proxy_pass and proxy_buffering.

I made some tests with nginx versions 1.12.2 and 1.14.1.

I prepared the following configuration for my nginx tests with limit_rate (proxy_limit_rate) 5000K.

/root/nginx_minimal/data is just huge file with dummy content.

nginx configuration file:

user  root;
worker_processes  1;

events {
    worker_connections  1024;
}

http {

    server {
        server_name .;
        listen  8001;
        location / {
            root /root/nginx_minimal;
        }
    }

    server {
        server_name .;
        listen  8000;

        location /limit_rate_buffering {
            rewrite /(.*) /data break;
            limit_rate 5000k;
            proxy_cache off;
            proxy_buffering on;
            proxy_pass http://localhost:8001;
        }

        location /limit_rate_buffering_plus_zero_temp {
            rewrite /(.*) /data break;
            limit_rate 5000k;
            proxy_cache off;
            proxy_buffering on;
            proxy_max_temp_file_size 0;
            proxy_pass http://localhost:8001;
        }

        location /proxy_limit_rate_buffering {
            rewrite /(.*) /data break;
            proxy_limit_rate 5000k;
            proxy_cache off;
            proxy_buffering on;
            proxy_pass http://localhost:8001;
        }

        location /proxy_limit_buffering_plus_zero_temp {
            rewrite /(.*) /data break;
            proxy_limit_rate 5000k;
            proxy_cache off;
            proxy_buffering on;
            proxy_max_temp_file_size 0;
            proxy_pass http://localhost:8001;
        }
    }
}

Test with nginx 1.12.2:

/www/nginx/sbin/nginx -V
nginx version: nginx/1.12.2
built with OpenSSL 1.1.0g 2 Nov 2017
TLS SNI support enabled
configure arguments: –with-debug –with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -fPIC -D_FORTIFY_SOURCE=2 -I /builds/nginx/szn-nginx/openssl/.openssl/include -pthread' –with-ld-opt='-fPIE -pie -Wl,-z,relro -Wl,-z,now -fPIC -L /builds/nginx/szn-nginx/openssl/.openssl/lib -pthread' –prefix=/www/nginx –conf-path=/www/nginx/doc/nginx.conf –http-log-path=/www/nginx/log/access.log –error-log-path=/www/nginx/log/error.log –lock-path=/www/nginx/lock/nginx.lock –pid-path=/var/run/nginx.pid –modules-path=/www/nginx/modules –http-client-body-temp-path=/www/nginx/var/lib/body –http-fastcgi-temp-path=/www/nginx/var/lib/fastcgi –http-proxy-temp-path=/www/nginx/var/lib/proxy –http-scgi-temp-path=/www/nginx/var/lib/scgi –http-uwsgi-temp-path=/www/nginx/var/lib/uwsgi –with-debug –with-pcre-jit –with-http_ssl_module –with-http_stub_status_module –with-http_realip_module –with-http_auth_request_module –with-http_v2_module –with-http_dav_module –with-http_slice_module –with-threads –with-http_addition_module –with-http_flv_module –with-http_geoip_module=dynamic –with-http_gunzip_module –with-http_gzip_static_module –with-http_image_filter_module=dynamic –with-http_mp4_module –with-http_perl_module=dynamic –with-http_random_index_module –with-http_secure_link_module –with-http_sub_module –with-http_xslt_module=dynamic –with-mail=dynamic –with-mail_ssl_module –with-stream=dynamic –with-stream_geoip_module=dynamic –with-stream_ssl_module –with-stream_ssl_preread_module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/headers-more-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/ngx_devel_kit –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/echo-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-lua –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/memc-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-push-stream-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-rtmp-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-auth-ldap –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/set-misc-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/srcache-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/ap_frpc –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/ap_mod –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/logap_mod –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/cd-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-dav-ext-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/ngx_brotli

curl -v -X GET "http://localhost:8000/limit_rate_buffering" >/dev/null
 22 97.6M   22 21.6M    0     0  5244k      0  0:00:19  0:00:04  0:00:15 5243k

curl -v -X GET "http://localhost:8000/limit_rate_buffering_plus_zero_temp" >/dev/null
 40 97.6M   40 39.3M    0     0  5256k      0  0:00:19  0:00:07  0:00:12 4998k

curl -v -X GET "http://localhost:8000/proxy_limit_rate_buffering" >/dev/null
 40 97.6M   40 39.8M    0     0  5044k      0  0:00:19  0:00:08  0:00:11 5041k

curl -v -X GET "http://localhost:8000/proxy_limit_buffering_plus_zero_temp" >/dev/null
 39 97.6M   39 38.8M    0     0  5040k      0  0:00:19  0:00:07  0:00:12 5040k

The limitations works fine. Average download speeds 5244k, 5256k, 5044k and 5040k corresponds with nginx configuration value 5000K.

Test of nginx 1.14.1:

/www/nginx/sbin/nginx -V

nginx version: nginx/1.14.1
built with OpenSSL 1.1.0i 14 Aug 2018
TLS SNI support enabled
configure arguments: –with-debug –with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -fPIC -D_FORTIFY_SOURCE=2 -I /builds/nginx/szn-nginx/openssl/.openssl/include -pthread' –with-ld-opt='-fPIE -pie -Wl,-z,relro -Wl,-z,now -fPIC -L /builds/nginx/szn-nginx/openssl/.openssl/lib -pthread' –prefix=/www/nginx –conf-path=/www/nginx/doc/nginx.conf –http-log-path=/www/nginx/log/access.log –error-log-path=/www/nginx/log/error.log –lock-path=/www/nginx/lock/nginx.lock –pid-path=/var/run/nginx.pid –modules-path=/www/nginx/modules –http-client-body-temp-path=/www/nginx/var/lib/body –http-fastcgi-temp-path=/www/nginx/var/lib/fastcgi –http-proxy-temp-path=/www/nginx/var/lib/proxy –http-scgi-temp-path=/www/nginx/var/lib/scgi –http-uwsgi-temp-path=/www/nginx/var/lib/uwsgi –with-debug –with-pcre-jit –with-http_ssl_module –with-http_stub_status_module –with-http_realip_module –with-http_auth_request_module –with-http_v2_module –with-http_dav_module –with-http_slice_module –with-threads –with-http_addition_module –with-http_flv_module –with-http_geoip_module=dynamic –with-http_gunzip_module –with-http_gzip_static_module –with-http_image_filter_module=dynamic –with-http_mp4_module –with-http_perl_module=dynamic –with-http_random_index_module –with-http_secure_link_module –with-http_sub_module –with-http_xslt_module=dynamic –with-mail=dynamic –with-mail_ssl_module –with-stream=dynamic –with-stream_geoip_module=dynamic –with-stream_ssl_module –with-stream_ssl_preread_module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/headers-more-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/ngx_devel_kit –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/echo-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-lua –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/memc-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-push-stream-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-rtmp-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-auth-ldap –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/set-misc-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/srcache-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/ap_frpc –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/ap_mod –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/datacollect-nginx-modules/logap_mod –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/cd-nginx-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/nginx-dav-ext-module –add-dynamic-module=/builds/nginx/szn-nginx/debian/modules/ngx_brotli

curl -v -X GET "http://localhost:8000/limit_rate_buffering" >/dev/null
 18 97.6M   18 18.2M    0     0  3201k      0  0:00:31  0:00:05  0:00:26 3200k

curl -v -X GET "http://localhost:8000/limit_rate_buffering_plus_zero_temp" >/dev/null
  3 97.6M    3 3023k    0     0   795k      0  0:02:05  0:00:03  0:02:02  795k

curl -v -X GET "http://localhost:8000/proxy_limit_rate_buffering" >/dev/null
 25 97.6M   25 24.9M    0     0  3605k      0  0:00:27  0:00:07  0:00:20 3603k

curl -v -X GET "http://localhost:8000/proxy_limit_buffering_plus_zero_temp" >/dev/null
 29 97.6M   29 29.0M    0     0  3603k      0  0:00:27  0:00:08  0:00:19 3600k

Average download speeds doesn't match nginx configuration value 5000K. They are much lower.

Thanks.

[Maxim Dounin]

Could you please provide details on the system you are testing with, in particular, which CONFIG_HZ is used in the kernel (grep CONFIG_HZ= /boot/config-`uname -r`)? Also, could you please check if building nginx with --with-cc-opt="-DNGX_HAVE_CLOCK_MONOTONIC=0" helps in your case? Since 1.13.10 nginx uses clock_gettime(CLOCK_MONOTONIC_COARSE) if available, and this may have low resolution if CONFIG_HZ is set to a low value.

I can't reproduce any difference between with limit_rate and proxy_limit_rate behaviour on nginx 1.12.2 and 1.14.1, neither Linux nor on FreeBSD. I'm however able to reproduce inaccurate limits being applied with the settings in question on a virtual machine with Linux, both with nginx 1.14.1 and nginx 1.12.2.

It seems to be an unfortunate effect of the limits configured and the fact that virtual machine uses non-precise timers. This is because limit_rate limits both average and immediate download rate, and while limiting immediate download rate it uses timers to delay further sending. If the delay time calculated is only several milliseconds (12ms in the configuration provided, as sending of 64k output_buffers is expected take at least 12ms with the speed configured), and the timer is fired later than specified, then sending in small chunks may result in smaller speed than specified.

To improve accuracy of limit_rate with high rates, you may consider using larger buffers (​proxy_buffers, ​output_buffers, and/or using ​sendfile with disk buffering.

(Please also note that average download rate limiting uses time with seconds resolution, and may be very inaccurate on short downloads, such as 3 seconds in your second /limit_rate_buffering_plus_zero_temp test.)

[Ondřej Nový]

It's LXC container on bare metal.

CONFIG_HZ=100

I will try -DNGX_HAVE_CLOCK_MONOTONIC=0 and/or larger buffers. Will report results after weekend.

Thanks for suggestions.

[Maxim Dounin]

Well, CONFIG_HZ=100 explains what you observe, as CLOCK_MONOTONIC_COARSE time will be only updated every 10 milliseconds. As such, delaying sending for 12ms as happens with default buffers and the rate configured will be very inaccurate. Using larger buffers should help.

[Ondřej Nový]

Using larger buffers helps. Setting higher CONFIG_HZ or disabling NGX_HAVE_CLOCK_MONOTONIC works too. Thanks for suggestions.

Maybe it's good idea to write this into documentation?

Thx.

[Maxim Dounin]

Thanks for testing.

I'm not sure there is a good way to document all the details which may affect limit_rate accuracy, and also its accuracy is good enough when limiting to reasonable rates with default buffers, even on Linux with default CONFIG_HZ. We'll consider this if there will be more questions about this. Alternatively, we can also consider not using CLOCK_MONOTONIC_COARSE on Linux, it seems to be much worse than CLOCK_MONOTONIC_FAST as available on FreeBSD due to very low accuracy, which also varies with kernel configuration.

[Maxim Dounin]

See also #1965.

[Maxim Dounin <mdounin@…>]

In 7939:9e7de0547f09/nginx:

Removed CLOCK_MONOTONIC_COARSE support.

While clock_gettime(CLOCK_MONOTONIC_COARSE) is faster than
clock_gettime(CLOCK_MONOTONIC), the latter is fast enough on Linux for
practical usage, and the difference is negligible compared to other costs
at each event loop iteration. On the other hand, CLOCK_MONOTONIC_COARSE
causes various issues with typical CONFIG_HZ=250, notably very inaccurate
limit_rate handling in some edge cases (ticket #1678) and negative difference
between $request_time and $upstream_response_time (ticket #1965).