coreruleset / fake-bot-plugin Goto Github PK
View Code? Open in Web Editor NEWThis is a plugin that brings blocking of bots faking User-Agent to CRS.
License: Apache License 2.0
This is a plugin that brings blocking of bots faking User-Agent to CRS.
License: Apache License 2.0
Fake Bot plugin blocks valid bots and allows fake bots.
Sample googlebot transaction:
{
"transaction": {
"client_ip": "66.249.64.219",
"time_stamp": "Tue Jul 12 15:54:19 2022",
"server_id": "9e7bc6878f15155f664887f5952d257c0d032745",
"client_port": 61702,
"host_ip": "255.255.255.255",
"host_port": 443,
"unique_id": "1657641259",
"request": {
"method": "GET",
"http_version": 1.1,
"uri": "/",
"headers": {
"Host": "redacted.com",
"AMP-Cache-Transform": "google;v=\"1..8\"",
"Connection": "keep-alive",
"Accept": "text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"From": "googlebot(at)googlebot.com",
"Accept-Encoding": "gzip, deflate, br",
"If-Modified-Since": "Tue, 24 May 2022 23:58:16 GMT"
}
},
"response": {
"body": "",
"http_code": 403,
"headers": {}
},
"producer": {
"modsecurity": "ModSecurity v3.0.6 (FreeBSD)",
"connector": "ModSecurity-nginx v1.0.3",
"secrules_engine": "Enabled",
"components": [
"OWASP_CRS/4.0.0-rc1\""
]
},
"messages": [
{
"message": "Fake bot detected: ",
"details": {
"match": "Matched \"Operator `Pm' with parameter `applebot bingbot facebookbot facebookcatalog facebookexternalhit googlebot twitterbot' against variable `REQUEST_HEADERS:User-Agent' (Value: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chr (100 characters omitted)' )",
"reference": "o153,9v265,200",
"ruleId": "9504110",
"file": "/usr/local/share/modsecurity-crs/plugins/fake-bot-after.conf",
"lineNumber": "19",
"data": "Matched Data: if-modified-since found within REQUEST_HEADERS:User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"severity": "2",
"ver": "fake-bot-plugin/1.0.0",
"rev": "",
"tags": [
"application-multi",
"language-multi",
"platform-multi",
"attack-bot",
"capec/1000/225/22/77/13",
"PCI/6.5.10",
"paranoia-level/1"
],
"maturity": "0",
"accuracy": "0"
}
}
]
}
}
Sample fake transaction curl "https://redacted.com/" -H "User-Agent: asd googlebot asd"
:
{
"transaction": {
"client_ip": "1.2.3.4",
"time_stamp": "Tue Jul 12 15:59:08 2022",
"server_id": "9e7bc6878f15155f664887f5952d257c0d032745",
"client_port": 64532,
"host_ip": "255.255.255.255",
"host_port": 443,
"unique_id": "1657641548",
"request": {
"method": "GET",
"http_version": 2,
"uri": "/",
"headers": {
"host": "redacted.com",
"accept": "*/*",
"user-agent": "asd googlebot asd"
}
},
"response": {
"body": "redacted",
"http_code": 200,
"headers": {}
},
"producer": {
"modsecurity": "ModSecurity v3.0.6 (FreeBSD)",
"connector": "ModSecurity-nginx v1.0.3",
"secrules_engine": "Enabled",
"components": [
"OWASP_CRS/4.0.0-rc1\""
]
},
"messages": [
{
"message": "Fake bot detected: Googlebot",
"details": {
"match": "Matched \"Operator `InspectFile' with parameter `fake-bot.lua' against variable `TX:0' (Value: `googlebot' )",
"reference": "o0,9v69,9",
"ruleId": "9504110",
"file": "/usr/local/share/modsecurity-crs/plugins/fake-bot-after.conf",
"lineNumber": "19",
"data": "Matched Data: googlebot found within REQUEST_HEADERS:User-Agent: Googlebot",
"severity": "2",
"ver": "fake-bot-plugin/1.0.0",
"rev": "",
"tags": [
"application-multi",
"language-multi",
"platform-multi",
"attack-bot",
"capec/1000/225/22/77/13",
"PCI/6.5.10",
"paranoia-level/1"
],
"maturity": "0",
"accuracy": "0"
}
}
]
}
}
Using fb1381f, modsecurity 3.0.6, modsecurity-nginx 1.0.2, nginx 1.22.0.
really useful plugin @azurit !
Is it possible to use "Impersonator" instead of "fake bot"? According to other vendors, "Impersonator" should be the right category name of this kind of "bad bots".
https://support.apple.com/en-us/HT204683
$ grep -c "Applebot" /tmp/tmp.access
45
You may find some additional ideas for the plugin at https://github.com/Flameeyes/modsec-flameeyes
Hi there,
apple devices with iMessages using:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Will need to disable this plugin and/or modify lua/rule on my own to take care of these requests! (lots of)
Thanks for your work!
After configuration, antivirus protection should be tested, for example, using:
curl http://localhost --header "User-Agent: Googlebot"
It would also be useful if you would indicate the expected behaviour and log entry.
Hello, i use CDN on my wensite and somes request are redirected from.
Possible to add exeption ?
Exemple:
[Mon Feb 21 04:15:43.895269 2022] [:error] [pid 759926:tid 140629216577280] [client 137.74.122.3:28024] [client 137.74.122.3] ModSecurity: Warning. Fake Bot Plugin: Detected fake Bingbot. [file "/etc/modsecurity/plugins/fake-bot-after.conf"] [line "27"] [id "9504110"] [msg "Fake bot detected: Bingbot"] [data "Matched Data: bingbot found within REQUEST_HEADERS:User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"] [severity "CRITICAL"] [ver "fake-bot-plugin/1.0.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-bot"] [tag "capec/1000/225/22/77/13"] [tag "PCI/6.5.10"] [tag "paranoia-level/1"] [hostname "cdn.achatpc.com"] [uri "/media/catalog/product/cache/7d38c09cea45996b9bdc3949df76cebd/icecatconnect/48/1053924807500_6877864685.jpg"] [unique_id "YhMD3xtye7aMD6dFGT_JYAAAFAY"]
Thank you.
I saw the following UA faking twitterbot:
twitterbot/1.0
Do not know if a real twitterbot is scanning the net.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.