Comments (14)
Hi Mitchell
Thanks for quick reply !
It seems to work so far :)
Please just change the 1r/s to 1r/m 👍
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
Thanks !
from nginx-ultimate-bad-bot-blocker.
Hi Mitchell,
So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" 😊
Your modification to globalblacklist.conf was perfect, so we keep that:
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
We only forgot the bot_iplimit2 declaration into blockbots.conf !
limit_conn bot2_connlimit 10;
limit_req zone=bot2_reqlimitip burst=10;
limit_conn bot4_connlimit 10;
limit_req zone=bot4_reqlimitip burst=10;
if ($bad_bot = '3') {
return 444;
This lead to a working validation with new test-blocker-rate-limiting.sh:
- Googlebot (2r/s) got responses in less than 30 sec (2 sec)
- Applebot (1r/m) got responses in more than 30 sec (90 sec)
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - Applebot was SUPER RATE LIMITED
Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf 😊
from nginx-ultimate-bad-bot-blocker.
Hi @bigrck64
I'm a bit tied down with some other work. Can you possibly do a test, by manually modifying your globalblacklist.conf as follows
# 1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 $binary_remote_addr;
}
# --------------------------
# 2. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/s;
Let me know if this works
from nginx-ultimate-bad-bot-blocker.
Hi Mitchell
Thanks for quick reply ! It seems to work so far :)
Please just change the 1r/s to 1r/m 👍
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
Thanks !
@bigrck64
Done, feature added - docs updated and tests added into build tests.
from nginx-ultimate-bad-bot-blocker.
Awesome ! 😊
from nginx-ultimate-bad-bot-blocker.
Maybe you should change
### 1 = allowed or rate limited less restrictive
### 2 = rate limited more
### 3 = block completely
### 4 = super rate limited
to
### 1 = allowed
### 2 = rate limited
### 3 = block completely
### 4 = super rate limited
Because for me, 1 does not perform any rate limited task, no ?
from nginx-ultimate-bad-bot-blocker.
@bigrck64 correct 1
essentially does nothing, same as 0
bit too late to change that.
Will update docs.
from nginx-ultimate-bad-bot-blocker.
Maybe you should change
### 1 = allowed or rate limited less restrictive ### 2 = rate limited more ### 3 = block completely ### 4 = super rate limited
to
### 1 = allowed ### 2 = rate limited ### 3 = block completely ### 4 = super rate limited
Because for me, 1 does not perform any rate limited task, no ?
Have done - but explained what 1 actually is/was for and may still be implemented if anyone wants an even less agressive rate limiting zone.
from nginx-ultimate-bad-bot-blocker.
Hum, I download your latest version and I feel that "Claudebot" is going into the rate=2r/s instead of rate=1r/m zone :(
Maybe with 2 or 4 it just send the binary_remote_addr into $bot_iplimit:
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 $binary_remote_addr;
}
Which just finish into
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
So the solution may be something like
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
and
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
?
from nginx-ultimate-bad-bot-blocker.
Please add this manually to your local globalblacklist.conf and see if nginx throws any errors - I'm a bit tied down with other stuff
# ============================================
# BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS
# ============================================
# --------------------------------------------
# 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION
# --------------------------------------------
map $bad_bot $bot_iplimit {
0 "";
1 "";
2 $binary_remote_addr;
4 "";
}
# --------------------------------------------------
# 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION
# --------------------------------------------------
map $bad_bot $bot_iplimit2 {
0 "";
1 "";
2 "";
4 $binary_remote_addr;
}
# --------------------------
# 4.3. SET RATE LIMITING ZONES
# --------------------------
# BAD BOT RATE LIMITING ZONE
# Rate limiting will only take effect if on any User-Agents with a value of 2
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
# BAD BOT SUPER RATE LIMITING ZONE
# Super Rate limiting will only take effect if on any User-Agents with a value of 4
limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m;
limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
from nginx-ultimate-bad-bot-blocker.
Thanks !
No problem with the new nginx config :)
The rate limiting seems fine. (edit: it is not 😢)
I can not run your test-blocker-rate-limiting.sh on my prod server, but maybe you will be able to confirm the super rate limiting is effective with it ?
from nginx-ultimate-bad-bot-blocker.
Hello Mitchell
I worked on the problem 😊
So I setup a new server to be able to understand your "test-blocker-rate-limiting.sh"
And I improved it with the following change:
- Add the duration and change from 80 to 60 curl requests with the response code:
truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
- Wait for all the curl to finish, then display the duration and the number of response code 200 or 503
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "GoogleBot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}
echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait
echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"
if [ $SECONDS -lt 30 ]
then
echo "${bold}${green}PASSED - ${green}GoogleBot was ${bold}${green}RATE LIMITED"
else
echo "${bold}${red}FAILED - ${red}GoogleBot was ${bold}${red}NOT RATE LIMITED"
fi
printf "\n"
}
- Same things for Applebot
truncate -s 0 ${ratelimittestfile}
SECONDS=0
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
And
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile} &
curl -s -m 90 -A "Applebot" -w "%{http_code}\n" -o /dev/null http://localhost:80 >> ${ratelimittestfile}
echo "${bold}${yellow}------------------------------"
echo "${bold}${yellow}Wait for all 60 curl to finish"
echo "${bold}${yellow}------------------------------"
printf "\n"
wait
echo "Duration = $SECONDS seconds"
nb200=`cat ${ratelimittestfile} |grep 200 |wc -l`
echo "Count the number of 200 (response code) = $nb200";
nb503=`cat ${ratelimittestfile} |grep 503 |wc -l`
echo "Count the number of 503 (response code) = $nb503";
printf "\n"
if [ $SECONDS -gt 30 ]
then
echo "${bold}${green}PASSED - ${green}Applebot was ${bold}${green}SUPER RATE LIMITED"
else
echo "${bold}${red}FAILED - ${red}Applebot was ${bold}${red}NOT SUPER RATE LIMITED"
fi
printf "\n"
Conclusion
This new version of test-blocker-rate-limiting.sh show us the duration time of all curl and the % of request ok/rejected 👍
It also confirm that the current implementation of "super rate limited" have the same result than "rate limited".
TEST WITH 2r/s =
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=2r/s;
RESULT
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 11
Count the number of 503 (response code) = 49
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 5 seconds
Count the number of 200 (response code) = 10
Count the number of 503 (response code) = 50
FAILED - Applebot was NOT SUPER RATE LIMITED
TEST with 1r/m
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=1r/m;
limit_req_zone $bot_iplimit zone=bot4_reqlimitip:16m rate=1r/m;
RESULT
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - GoogleBot was RATE LIMITED
------------------------------
Wait for all 60 curl to finish
------------------------------
Duration = 90 seconds
Count the number of 200 (response code) = 0
Count the number of 503 (response code) = 58
PASSED - Applebot was SUPER RATE LIMITED
edit1: make the curl wait 90 sec max to not freeze completly the script with nginx super rate limited (1r/m)
edit2: add the total duration of 60 curl tests (BASH native)
edit3: slow down from 80 to 60 curl request to not slow the test at 2 minuts with normal "rate limited" settings (2r/s)
edit4: change [if] from number of 503 to if duration of replied (more stable)
Latest test-blocker-rate-limiting.sh (v2) :
test-blocker-rate-limiting.zip
I will now work into nginx globalblacklist.conf to see if I can find a way to make the "super rate limited" to work along with "rate limited" 😊
from nginx-ultimate-bad-bot-blocker.
Hi Mitchell,
So with the updated test-blocker-rate-limiting.sh, I was able to test and find a solution for nginx and "super rate limiter" 😊
Your modification to globalblacklist.conf was perfect, so we keep that:
# ============================================ # BEGIN SECTION 4 - ACTIVATE BLOCKER FUNCTIONS # ============================================ # -------------------------------------------- # 4.1. MAP BAD BOTS TO OUR RATE LIMITER FUNCTION # -------------------------------------------- map $bad_bot $bot_iplimit { 0 ""; 1 ""; 2 $binary_remote_addr; 4 ""; } # -------------------------------------------------- # 4.2. MAP BAD BOTS TO OUR SUPER RATE LIMITER FUNCTION # -------------------------------------------------- map $bad_bot $bot_iplimit2 { 0 ""; 1 ""; 2 ""; 4 $binary_remote_addr; } # -------------------------- # 4.3. SET RATE LIMITING ZONES # -------------------------- # BAD BOT RATE LIMITING ZONE # Rate limiting will only take effect if on any User-Agents with a value of 2 limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m; limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s; # BAD BOT SUPER RATE LIMITING ZONE # Super Rate limiting will only take effect if on any User-Agents with a value of 4 limit_conn_zone $bot_iplimit2 zone=bot4_connlimit:16m; limit_req_zone $bot_iplimit2 zone=bot4_reqlimitip:16m rate=1r/m;
We only forgot the bot_iplimit2 declaration into blockbots.conf !
limit_conn bot2_connlimit 10; limit_req zone=bot2_reqlimitip burst=10; limit_conn bot4_connlimit 10; limit_req zone=bot4_reqlimitip burst=10; if ($bad_bot = '3') { return 444;
This lead to a working validation with new test-blocker-rate-limiting.sh:
- Googlebot (2r/s) got responses in less than 30 sec (2 sec)
- Applebot (1r/m) got responses in more than 30 sec (90 sec)
------------------------------ Wait for all 60 curl to finish ------------------------------ Duration = 5 seconds Count the number of 200 (response code) = 11 Count the number of 503 (response code) = 49 PASSED - GoogleBot was RATE LIMITED ------------------------------ Wait for all 60 curl to finish ------------------------------ Duration = 90 seconds Count the number of 200 (response code) = 0 Count the number of 503 (response code) = 58 PASSED - Applebot was SUPER RATE LIMITED
Feel free to update test-blocker-rate-limiting.sh, and please patch globalblacklist.conf + blockbots.conf 😊
Done, please grab latest of both files and confirm.
Tested on my server and working OK - I still need to reintroduce the BETA tests in this repo which broke some time ago.
By default I have left the new super rate limiting zone commented out in blockbots.conf users can decide to enable themselves. It threw an error during the build so I need to as i said reintroduce the beta tests again which test stuff like this without causing any potential breaking changes.
from nginx-ultimate-bad-bot-blocker.
Hi Mitchell,
Thanks for the patch to the master branch !
I uninstall / reinstall UBBB to my test and production server (Linux Debian), and it's working as expected, nice ! 👍
I have no error with the uncommented following lines in blockbots.conf, but it's fine like this, user can decide themself to active the "super rate limiter" zone or not 😊
# Uncomment below lines for super rate limiting feature
limit_conn bot4_connlimit 10;
limit_req zone=bot4_reqlimitip burst=10;
Thanks for all !
from nginx-ultimate-bad-bot-blocker.
Related Issues (20)
- [User-Agent] BrandVerity HOT 1
- [User-Agent] AwarioBot HOT 3
- [User-Agent] Bytespider, Bytedance didn't get to block after install bad bot blocker
- linkfluence HOT 4
- AwarioBot Addition HOT 1
- [INSTALLATION] nginx -t return warn HOT 1
- Whitelist Yahoo HOT 2
- [User-Agent] InternetMeasurement
- [BUG] Nginx logs shows permissions denied error to /etc/nginx/conf.d/globalblacklist.conf HOT 1
- blacklist-ips.conf does not do anything for me
- [User-Agent] GeedoProductSearch
- [User-Agent] Information Security Team InfraSec Scanner
- [INSTALLATION] Fail2ban/crowdsec and their interaction with this project, worth it? HOT 1
- [BUG] (Please add a way to uninstall completely) HOT 2
- [BUG] iptables-common.conf not in fail2ban v1.1.0
- [BUG] Certbot fails
- [BUG] facebookexternalhit is NOT a good bot HOT 7
- [User-Agent] ClaudeBot - Bad AI user-agent HOT 1
- OAI-SearchBot Whitelisting
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nginx-ultimate-bad-bot-blocker.