coreruleset / fake-bot-plugin Goto Github PK

This is a plugin that brings blocking of bots faking User-Agent to CRS.

License: Apache License 2.0

Lua 80.21% Shell 19.79%

fake-bot-plugin's Introduction

OWASP CRS

The OWASP CRS is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts.

CRS Resources

Please see the OWASP CRS page to get introduced to the CRS and view resources on installation, configuration, and working with the CRS.

Contributing to the CRS

We strive to make the OWASP ModSecurity CRS accessible to a wide audience of beginner and experienced users. We are interested in hearing any bug reports, false-positive alert reports, evasions, usability issues, and suggestions for new detections.

Create an issue on GitHub to report a false positive or false negative (evasion). Please include your installed version and the relevant portions of your ModSecurity audit log. We will try and address your issue and potentially ask for additional information to reproduce your problem. Please also note that stale issues will be flagged and closed after 120 days. You can search for stale issues with the following search query.

Sign up for our Google Group to ask general usage questions and participate in discussions on the CRS. Also here you can find the archives for the previous mailing list.

Join the #coreruleset channel on OWASP Slack to chat about the CRS. (Click here to get an invitation if you are not yet registered on the OWASP slack. It's open to non-members too.)

License

The OWASP CRS is distributed under Apache Software License (ASL) version 2. Please see the enclosed LICENSE file for full details.

fake-bot-plugin's People

Contributors

Stargazers

Watchers

Forkers

redxanadu theseion ics-forks yigitbasalma

fake-bot-plugin's Issues

README: explain how to install "LuaSocket library" on standard systems

CDN request false positive

Hello, i use CDN on my wensite and somes request are redirected from.

Possible to add exeption ?

Exemple:

[Mon Feb 21 04:15:43.895269 2022] [:error] [pid 759926:tid 140629216577280] [client 137.74.122.3:28024] [client 137.74.122.3] ModSecurity: Warning. Fake Bot Plugin: Detected fake Bingbot. [file "/etc/modsecurity/plugins/fake-bot-after.conf"] [line "27"] [id "9504110"] [msg "Fake bot detected: Bingbot"] [data "Matched Data: bingbot found within REQUEST_HEADERS:User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"] [severity "CRITICAL"] [ver "fake-bot-plugin/1.0.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-bot"] [tag "capec/1000/225/22/77/13"] [tag "PCI/6.5.10"] [tag "paranoia-level/1"] [hostname "cdn.achatpc.com"] [uri "/media/catalog/product/cache/7d38c09cea45996b9bdc3949df76cebd/icecatconnect/48/1053924807500_6877864685.jpg"] [unique_id "YhMD3xtye7aMD6dFGT_JYAAAFAY"]

Thank you.

Support for Applebot

https://support.apple.com/en-us/HT204683

$ grep -c "Applebot" /tmp/tmp.access 
45

Support for Twitterbot.

I saw the following UA faking twitterbot:

twitterbot/1.0

Do not know if a real twitterbot is scanning the net.

Use "Impersonator"

really useful plugin @azurit !

Is it possible to use "Impersonator" instead of "fake bot"? According to other vendors, "Impersonator" should be the right category name of this kind of "bad bots".

Inverted block

Fake Bot plugin blocks valid bots and allows fake bots.

Sample googlebot transaction:

{
  "transaction": {
    "client_ip": "66.249.64.219",
    "time_stamp": "Tue Jul 12 15:54:19 2022",
    "server_id": "9e7bc6878f15155f664887f5952d257c0d032745",
    "client_port": 61702,
    "host_ip": "255.255.255.255",
    "host_port": 443,
    "unique_id": "1657641259",
    "request": {
      "method": "GET",
      "http_version": 1.1,
      "uri": "/",
      "headers": {
        "Host": "redacted.com",
        "AMP-Cache-Transform": "google;v=\"1..8\"",
        "Connection": "keep-alive",
        "Accept": "text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8",
        "User-Agent": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
        "From": "googlebot(at)googlebot.com",
        "Accept-Encoding": "gzip, deflate, br",
        "If-Modified-Since": "Tue, 24 May 2022 23:58:16 GMT"
      }
    },
    "response": {
      "body": "",
      "http_code": 403,
      "headers": {}
    },
    "producer": {
      "modsecurity": "ModSecurity v3.0.6 (FreeBSD)",
      "connector": "ModSecurity-nginx v1.0.3",
      "secrules_engine": "Enabled",
      "components": [
        "OWASP_CRS/4.0.0-rc1\""
      ]
    },
    "messages": [
      {
        "message": "Fake bot detected: ",
        "details": {
          "match": "Matched \"Operator `Pm' with parameter `applebot bingbot facebookbot facebookcatalog facebookexternalhit googlebot twitterbot' against variable `REQUEST_HEADERS:User-Agent' (Value: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chr (100 characters omitted)' )",
          "reference": "o153,9v265,200",
          "ruleId": "9504110",
          "file": "/usr/local/share/modsecurity-crs/plugins/fake-bot-after.conf",
          "lineNumber": "19",
          "data": "Matched Data: if-modified-since found within REQUEST_HEADERS:User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
          "severity": "2",
          "ver": "fake-bot-plugin/1.0.0",
          "rev": "",
          "tags": [
            "application-multi",
            "language-multi",
            "platform-multi",
            "attack-bot",
            "capec/1000/225/22/77/13",
            "PCI/6.5.10",
            "paranoia-level/1"
          ],
          "maturity": "0",
          "accuracy": "0"
        }
      }
    ]
  }
}

Sample fake transaction curl "https://redacted.com/" -H "User-Agent: asd googlebot asd":

{
  "transaction": {
    "client_ip": "1.2.3.4",
    "time_stamp": "Tue Jul 12 15:59:08 2022",
    "server_id": "9e7bc6878f15155f664887f5952d257c0d032745",
    "client_port": 64532,
    "host_ip": "255.255.255.255",
    "host_port": 443,
    "unique_id": "1657641548",
    "request": {
      "method": "GET",
      "http_version": 2,
      "uri": "/",
      "headers": {
        "host": "redacted.com",
        "accept": "*/*",
        "user-agent": "asd googlebot asd"
      }
    },
    "response": {
      "body": "redacted",
      "http_code": 200,
      "headers": {}
    },
    "producer": {
      "modsecurity": "ModSecurity v3.0.6 (FreeBSD)",
      "connector": "ModSecurity-nginx v1.0.3",
      "secrules_engine": "Enabled",
      "components": [
        "OWASP_CRS/4.0.0-rc1\""
      ]
    },
    "messages": [
      {
        "message": "Fake bot detected: Googlebot",
        "details": {
          "match": "Matched \"Operator `InspectFile' with parameter `fake-bot.lua' against variable `TX:0' (Value: `googlebot' )",
          "reference": "o0,9v69,9",
          "ruleId": "9504110",
          "file": "/usr/local/share/modsecurity-crs/plugins/fake-bot-after.conf",
          "lineNumber": "19",
          "data": "Matched Data: googlebot found within REQUEST_HEADERS:User-Agent: Googlebot",
          "severity": "2",
          "ver": "fake-bot-plugin/1.0.0",
          "rev": "",
          "tags": [
            "application-multi",
            "language-multi",
            "platform-multi",
            "attack-bot",
            "capec/1000/225/22/77/13",
            "PCI/6.5.10",
            "paranoia-level/1"
          ],
          "maturity": "0",
          "accuracy": "0"
        }
      }
    ]
  }
}

Using fb1381f, modsecurity 3.0.6, modsecurity-nginx 1.0.2, nginx 1.22.0.

Look at Flameeyes rule set for inspiration

You may find some additional ideas for the plugin at https://github.com/Flameeyes/modsec-flameeyes

Fix wording in README

Testing

After configuration, antivirus protection should be tested, for example, using:
curl http://localhost --header "User-Agent: Googlebot"

It would also be useful if you would indicate the expected behaviour and log entry.

Lots of FP from apple imessages

Hi there,

apple devices with iMessages using:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

Will need to disable this plugin and/or modify lua/rule on my own to take care of these requests! (lots of)

Thanks for your work!