Include larger speed units for HttpLimitReqModule

[incognito2.myopenid.com]

I have a huge problem on my site with site scrapers, spam and spider bots. While having per second and per minute request limits helps mitigate a DDOS it doesn't help with scrapers & spiders.

Most of these bots fetch pages at a similar rate to regular users ie. 2-3 per minute. The difference between these bots and users is the consistency over time;they can fetch hundreds or thousands or pages a day. I can't use iptables either because they usually use one connection and sometimes users with real browsers establish more connections than these robots.

My request is to increase the allowable measured speed unit in the limit_req_zone directive to something larger than per minute; ie per hour, per day.

[Sam Kottler]

This seems sensible, but I think the sensible thing to do here is to make the time frame an optional parameter of limit_req_zone directive. That way existing users can continue to use the directive with the timeframe of a minute, but users with more complex needs (like yours) can choose a larger increment when necessary. Does that sound like it would satisfy your use case?

[k.cherenkov@…]

This problem with site scrapers is actual for me too. Please improve HttpLimitReqModule.
The time frame as an optional parameter of limit_req_zone directive may solve problem.

[Maxim Dounin]

See also #1060.

[xdmitry@…]

can we give it a go please?

change is straightforward:

--- ngx_http_limit_req_module.c.orig    2018-12-12 10:29:08.434882008 +0000
+++ ngx_http_limit_req_module.c 2018-12-12 10:30:29.313061685 +0000
@@ -801,6 +801,12 @@
             } else if (ngx_strncmp(p, "r/m", 3) == 0) {
                 scale = 60;
                 len -= 3;
+            } else if (ngx_strncmp(p, "r/h", 3) == 0 {
+                scale = 3600;
+                len -= 3;
+            } else if (ngx_strncmp(p, "r/d", 3) == 0 {
+                scale = 86400;
+                len -= 3;
             }
 
             rate = ngx_atoi(value[i].data + 5, len - 5);

[Maxim Dounin]

The patch is wrong, it does not take into account that rates use fixed point with 3 digits, and 1r/h rate simply cannot be represent.

Previous attempts to introduce smaller rates can be found here:

​http://mailman.nginx.org/pipermail/nginx-devel/2018-February/010856.html

Review can be found here:

​http://mailman.nginx.org/pipermail/nginx-devel/2018-February/010858.html