Opened 12 years ago

Last modified 5 years ago

#68 new enhancement

Include larger speed units for HttpLimitReqModule

Reported by: Owned by: somebody
Priority: major Milestone:
Component: nginx-module Version: 1.0.x
Keywords: Cc:
uname -a: not applicable
nginx -V: nginx 1.0.10


I have a huge problem on my site with site scrapers, spam and spider bots. While having per second and per minute request limits helps mitigate a DDOS it doesn't help with scrapers & spiders.

Most of these bots fetch pages at a similar rate to regular users ie. 2-3 per minute. The difference between these bots and users is the consistency over time;they can fetch hundreds or thousands or pages a day. I can't use iptables either because they usually use one connection and sometimes users with real browsers establish more connections than these robots.

My request is to increase the allowable measured speed unit in the limit_req_zone directive to something larger than per minute; ie per hour, per day.

Change History (5)

comment:1 by Sam Kottler, 11 years ago

This seems sensible, but I think the sensible thing to do here is to make the time frame an optional parameter of limit_req_zone directive. That way existing users can continue to use the directive with the timeframe of a minute, but users with more complex needs (like yours) can choose a larger increment when necessary. Does that sound like it would satisfy your use case?

comment:2 by k.cherenkov@…, 8 years ago

This problem with site scrapers is actual for me too. Please improve HttpLimitReqModule.
The time frame as an optional parameter of limit_req_zone directive may solve problem.

comment:3 by Maxim Dounin, 6 years ago

sensitive: 0

See also #1060.

comment:4 by xdmitry@…, 5 years ago

can we give it a go please?

change is straightforward:

--- ngx_http_limit_req_module.c.orig    2018-12-12 10:29:08.434882008 +0000
+++ ngx_http_limit_req_module.c 2018-12-12 10:30:29.313061685 +0000
@@ -801,6 +801,12 @@
             } else if (ngx_strncmp(p, "r/m", 3) == 0) {
                 scale = 60;
                 len -= 3;
+            } else if (ngx_strncmp(p, "r/h", 3) == 0 {
+                scale = 3600;
+                len -= 3;
+            } else if (ngx_strncmp(p, "r/d", 3) == 0 {
+                scale = 86400;
+                len -= 3;
             rate = ngx_atoi(value[i].data + 5, len - 5);

comment:5 by Maxim Dounin, 5 years ago

The patch is wrong, it does not take into account that rates use fixed point with 3 digits, and 1r/h rate simply cannot be represent.

Previous attempts to introduce smaller rates can be found here:

Review can be found here:

Note: See TracTickets for help on using tickets.