Opened 6 years ago

Last modified 20 months ago

#68 new enhancement

Include larger speed units for HttpLimitReqModule

Reported by: incognito2.myopenid.com Owned by: somebody
Priority: major Milestone:
Component: nginx-module Version: 1.0.x
Keywords: Cc:
Sensitive:
uname -a: not applicable
nginx -V: nginx 1.0.10

Description

I have a huge problem on my site with site scrapers, spam and spider bots. While having per second and per minute request limits helps mitigate a DDOS it doesn't help with scrapers & spiders.

Most of these bots fetch pages at a similar rate to regular users ie. 2-3 per minute. The difference between these bots and users is the consistency over time;they can fetch hundreds or thousands or pages a day. I can't use iptables either because they usually use one connection and sometimes users with real browsers establish more connections than these robots.

My request is to increase the allowable measured speed unit in the limit_req_zone directive to something larger than per minute; ie per hour, per day.

Change History (2)

comment:1 Changed 5 years ago by www.google.com/accounts/o8/id?id=AItOawlw3IJys0xaqUB1gRJ55tN_bZbiTAuCZnc

This seems sensible, but I think the sensible thing to do here is to make the time frame an optional parameter of limit_req_zone directive. That way existing users can continue to use the directive with the timeframe of a minute, but users with more complex needs (like yours) can choose a larger increment when necessary. Does that sound like it would satisfy your use case?

comment:2 Changed 20 months ago by k.cherenkov@…

This problem with site scrapers is actual for me too. Please improve HttpLimitReqModule?.
The time frame as an optional parameter of limit_req_zone directive may solve problem.

Note: See TracTickets for help on using tickets.