crowdsecurity / hub Goto Github PK

View Code? Open in Web Editor NEW

143.0 11.0 138.0 154.2 MB

Main repository for crowdsec scenarios/parsers

Home Page: https://hub.crowdsec.net

Go 38.55% Shell 3.50% Python 57.25% Dockerfile 0.71%

hub's Introduction

📚 Documentation 💠 Configuration Hub 💬 Discourse (Forum) 💬 Discord (Live Chat)

CrowdSec Hub for parsers, enrichers and scenarios.

Foreword

This repository stores most of the official parsers/scenarios/collections for crowdsec.

The repository is not intended for use as-is, but rather as source of truth for the CrowdSec Hub and cscli.

Feel free to use the parsers/scenarios here as a source of inspiration.

Testing & Continuous integration

cscli provides a hubtest sub-command to help contributors to create tests for parsers and scenarios.

View & use existing tests

⚠️ most of cscli hubtest commands are expected to be run from the root directory of the hub. A git clone of this repository is the easier way to work ⚠️

list existing tests

cscli hubtest list

run a specific test

cscli hubtest run [test-name]

show current tests coverage

cscli hubtest coverage

Create your own (parser) test

We're going to create the CI tests for the dovecot-parser. Before you start :

you will need some actual logs
you'd better know if the service logs on its own or via syslog (we're in the later case here)

Create a new test

▶ cscli hubtest create dovecot-logs --type syslog

  Test name                   :  dovecot-logs
  Test path                   :  .../github/hub/.tests/dovecot-logs
  Log file                    :  .../github/hub/.tests/dovecot-logs/dovecot-logs.log (please fill it with logs)
  Parser assertion file       :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)
  Scenario assertion file     :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)
  Configuration File          :  .../github/hub/.tests/dovecot-logs/config.yaml (please fill it with parsers, scenarios...)

What is relevant here is that every test is composed of :

A log file and it's associated type (same type as seen in acquis labels:type)
A configuration specifying which parsers and/or scenarios must be enabled for the test
A ultimately list of assertions that must be run against the parsers and/or scenarios output

Note: You can provide the parsers and scenarios you want in your test with --parsers and --scenarios (you can provide multiple parsers and scenarios)

If you want to test only a scenario, you can specify (--ignore-parsers) or set the ignore_parsers to true in the config.yaml

Configure your test

We need to edit the test configuration to use the relevant parsers :

▶ cat .../github/hub/.tests/dovecot-logs/config.yaml
parsers:
- crowdsecurity/syslog-logs
- crowdsecurity/dovecot-logs
scenarios:
postoverflows:
- ""
log_file: dovecot-logs.log
log_type: syslog

note: the order doesn't matter. If the parser name is in the form author/parser it's from the hub, but relative paths are allowed for non-versioned parsers

Now we need to dump some actual logs into the test's log file :

▶ cat > .tests/dovecot-logs/dovecot-logs.log
Jan 28 10:16:13 dovecot-box dovecot[7508]: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>
Sep 8 07:16:29 canyon dovecot: auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?)
Sep 8 07:46:51 canyon dovecot: auth-worker(24544): pam(toto,1.1.1.1): unknown user

Run the test for the first time

Now that we have config & logs, let's run it for the first time :

▶ cscli hubtest run dovecot-logs

INFO[27-09-2021 06:13:59 PM] Running test 'dovecot-logs'
INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/dovecot-logs' installed successfully in runtime environment
INFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/syslog-logs' installed successfully in runtime environment
WARN[27-09-2021 06:14:02 PM] Assert file '/home/bui/github/hub/.tests/dovecot-logs/parser.assert' is empty, generating assertion:

results["s00-raw"]["crowdsecurity/syslog-logs"][0].Success == true
...
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "[email protected]"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4"
...

As our parser.assert is empty, the tool is generating some "suggested" asserts for us. Your careful eye will keep only the ones relevant to the parser you're testing :

▶ cat > .tests/dovecot-logs/parser.assert
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["pid"] == "7508"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["timestamp"] == "Jan 28 10:16:13"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_login_result"] == "Disconnected (auth failed, 1 attempts in 6 secs)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["message"] == "imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=<[email protected]>, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=<3650VvK5bdIaW-iK>"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["protocol"] == "imap"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_local_ip"] == "7.7.7.7"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_remote_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Parsed["dovecot_user"] == "[email protected]"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][0].Evt.Meta["source_ip"] == "4.4.4.4"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_login_result"] == "Authentication failure (password mismatch?)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["message"] == "auth-worker(24058): pam(toto,1.1.1.1,<youpi>): pam_authenticate() failed: Authentication failure (password mismatch?)"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["timestamp"] == "Sep 8 07:16:29"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Parsed["dovecot_user"] == "toto"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["datasource_type"] == "file"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][1].Evt.Meta["source_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Success == true
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_login_result"] == "unknown user"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["program"] == "dovecot"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_remote_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["dovecot_user"] == "toto"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["logsource"] == "syslog"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["message"] == "auth-worker(24544): pam(toto,1.1.1.1): unknown user"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed["timestamp"] == "Sep 8 07:46:51"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["log_type"] == "dovecot_logs"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["source_ip"] == "1.1.1.1"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_path"] == "dovecot-logs.log"
results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Meta["datasource_type"] == "file"

Test your newly crafted test

▶ cscli hubtest run dovecot-logs
INFO[27-09-2021 06:19:33 PM] Running test 'dovecot-logs'
INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/syslog-logs' installed successfully in runtime environment
INFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/dovecot-logs' installed successfully in runtime environment
Test 'dovecot-logs' passed successfully (39 assertions) 🟩

And be amazed.

Debug your own (parser) test

Things went wrong ? Don't panic

When working on a test, you can as well pass expressions directly to hubtest command and see the results :

▶ cscli hubtest  eval dovecot-logs -e 'results["s01-parse"]["crowdsecurity/dovecot-logs"][2].Evt.Parsed'
dovecot_login_result: unknown user
dovecot_remote_ip: 1.1.1.1
dovecot_user: toto
facility: ""
logsource: syslog
message: 'auth-worker(24544): pam(toto,1.1.1.1): unknown user'
pid: ""
priority: ""
program: dovecot
timestamp: Sep 8 07:46:51
timestamp8601: ""

Contributing

We gratefully accept PR to enrich the content of Hub. Follow the steps on our documentation website: https://docs.crowdsec.net/docs/next/contributing/contributing_hub/

hub's People

Contributors

Stargazers

Watchers

Forkers

mobula9 cyrilbrulebois dani godsnightmare synsecurity rembo-fw belphemur zecloud jusabatier pelletier6666 davidosipov gmelodie fstap sbs2001 lightiam be-secure skewel sobinge hmidani-abdelilah bennet-esyoil umglurf rhoekman mschiessl lepresidente bamx23 abeautifulpumpkin davislakhan ltvthang pasientskyhosting woopstar pdf jostrus madchristian robomagus hitech95 corvese sdbdevelopment crt-fork thespad tomtom5152 apo-mak lelanta ekmixon aderumier ornicar raphmad fredericsimard lourys y0ngg4n libreshare testklaus ggiammat slimshizn kumy byter09 4maself kofl pojuvubu webdienste lockbrock dstr-wpd leeshan87 naimo84 guard43ru bmunro-peralex mstilkerich andyoulovexy barnoux gauth-fr saygoweb juliencouturier bradpeczka bdcarr a1ad jwildeboer csabyka aritas1 laurencejjones ketan-vijayvargiya alixyz mwinters-stuff gnu-plus-windows-user nadavtsa chkp-omris baolunkk710 va2xjm chrisblech zaulao kociubin ppkarwasz m3e-g jkunczik koffeinkaio lperdereau inherent-io vamega kornelius777 jebabin mr-bolle sabag

hub's Issues

Dovecot logs

Actually there is no module about dovecot.
The fail2ban filter work just fine in the aggressive mode for me.
Contact me if you need specific logs files.

I think would be good to add DNS example "Pi Hole" scenario.
For help to protect.
On some DNS product is hard too block all bad ip.
I think especially to be able to make sure to check certain type of attack or malicious use on port 53 for example UDPMIX

Best Regards

Bouncer inclusion: cs-wordpress-bouncer

Hello,

I would like to suggest the addition of the cs-wordpress-bouncer to the hub :

Source repository: https://github.com/crowdsecurity/cs-wordpress-bouncer
Licence : MIT
Current status : unstable
README/doc : https://github.com/crowdsecurity/cs-wordpress-bouncer/README.md
Existing tests :
- functional tests : https://github.com/crowdsecurity/cs-wordpress-bouncer/blob/main/run-tests-with-all-wp-versions.sh
Short/Long description : OK
Howto : in README
At least one release : yes

PHPMyAdmin brute force

add scenarios for PHPMyAdmin brute force.
I have a fail2ban filter on that :
^.GET.(?i)phpmyadmin/index.php?pma_username=root.*
^.GET.(?i)phpmyadmin/index.php?lang=en&pma_username=admin.*
^.GET.(?i)phpmyadmin/index.php?lang=en&pma_username=root.*
^.POST.(?i)phpmyadmin/index.php.*

crowdsecurity/apache2 distinguish access logs from error logs

Hello,

crowdsecurity/nginx makes the distinction between access logs and error logs.
Could we have the same for apache2 logs, please ?
This is quiet straightforward:

Example of error logs:

[Sat Nov 14 18:38:54.891620 2020] [authz_core:error] [pid 30873:tid 140345882953472] [client 42.231.130.238:36326] AH01630: client denied by server configuration: /var/www/default/boaform
[Sat Nov 14 18:39:39.518161 2020] [authz_core:error] [pid 30873:tid 140347174807296] [client 95.38.202.202:59281] AH01630: client denied by server configuration: /var/www/default/
[Sat Nov 14 19:04:36.773086 2020] [auth_basic:error] [pid 30873:tid 140347272279808] [client 192.168.1.1:46510] AH01617: user toto: authentication failure for "/backup/": Password Mismatch

What do you think ?
Cheers.

Bad user agent case sensitive ?

As asked on gitter :

Hi, don't know if someone is here, but for the bad user agent, are the name key sensitive ? Like the masscan is not detected correctly, in the file it's Masscan, but the user agent is : masscan/1.0 (https://github.com/robertdavidgraham/masscan)
I have write my own scenario with masscan and it's correctly detected

Add Nuclei as bad user agent

Nuclei is a tool used to scan website and find vulnerability.
Here are some exemple :
193.32.164.27 - - [09/Jan/2021:21:02:43 +0100] "GET /public/adminer.php HTTP/1.1" 301 5524 "-" "Nuclei - Open-source project (github.com/projectdiscovery/nuclei)"

Home is here : https://github.com/projectdiscovery/nuclei

I try to deploy this scenario :
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.agent contains 'Nuclei'"

I check to see if I will detect other scan.

Improvement/services: target caddy and haproxy logs

Describe the solution you'd like

It would be nice to be able to have crowdsec recognize haproxy or caddy reverse proxies as services that have been installed.
I think it would complement both tools if crowdsec could parse their log files and be able to apply e.g. the netfilter/iptables bouncer when malicious activity is detected.

scenarios/http-bf-wordpress_bf doesn't work

This scenarios atm doesn't do anything for me.
I have to remove && evt.Parsed.status == '200' or change it to evt.Meta.http_status == '200'.

I think the status is not needed and can ignore some threat.
Some of my server need apache auth to access wp-login, this mean the status is 401 and not 200.
And if some bots try to reach wp-login on non wordpress site this will give à 301 or 404 and then ignore the bot.

modsec collection is inconsistent

https://hub.crowdsec.net/author/crowdsecurity/collections/modsecurity is inconsistent, as it refers twice the parser and doesn't refer the scenario
the APACHEERRORPREFIX and MODSECAPACHEERROR seems to have some mistakes as well according to crowdsecurity/crowdsec#349

Improvement on wordpress bruteforce scenario

I have made some change on the WP BrufteForce scenario to avoid blocking on reset pass or lost password.
The leakspeed is increased, the default scenario don't detect a lot of attack.
And this scenario detect access on the xmlrpc file too. Take care if you use xmlrpc to whitelist the IP or to not use this one.

type: leaky
name: si/http-bf-wordpress_bf-custom
description: "detect wordpress bruteforce"
debug: false
# failed auth on wp-login.php with POST and no try to reset passs
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.file_name in ['wp-login.php','xmlrpc.php'] && evt.Parsed.verb == 'POST' && evt.Parsed.http_args not in ['action=lostpassword','action=resetpass']"
groupby: evt.Meta.source_ip
capacity: 3
leakspeed: "300s"
blackhole: 5m
labels:
 service: http
 type: bruteforce
 remediation: true

crowdsecurity/http-probing : Exclude static ressources

the scenario crowdsecurity/http-probing should exclude static ressources to avoid FP.

Add Pinterest to Whitelist / Good Actors

crawl-xxx-xxx-xxx-xxx.pinterest.com (xxx being the actual ip) should be allowed as a good actor. If not crowdsecurity/http-probing will block this ip.

create collection to parse and detect auth fail on promox api

following this thread, we need to create a parser and scenario to detect auth failure on promox api parser example.

nginx parser : support basic auth error messages

As seen on gitter, nginx parser doesn't support basic auth, ie. 2021/03/16 08:29:32 [error] 5669#5669: *2829 user "123" was not found in "/etc/nginx/.htpasswd", ....

It should be supported in a similar way to https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/apache2-logs.yaml

static file in apache2 need to NOT be case sensitive

Recently I had 1 false positive on the crawl non static scenarios because the pictures had as extension .JPG instead of .jpg...

some data from the database :
Ip xxx performed 'crowdsecurity/http-crawl-non_statics' (48 events over 18.897202709s)
"key":"http_path","value":"/bleunautique/img/tmp/cms/winch1.JPG"
"key":"http_path","value":"/bleunautique/img/tmp/cms/fixation-nylon2.JPG"
"key":"http_path","value":"/bleunautique/img/tmp/cms/sea-drogue.JPG"

w00tw00t

Harmless most of the time, but no one should try to access to any file with w00tw00t...
This is a trigger scenario and will act on the first access.

type: trigger
name: si/w00tw00t
description: "detect w00tw00t"
debug: false
# request on xmlrpc
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.file_name contains 'w00tw00t.at.ISC.SANS.DFind'"
groupby: evt.Meta.source_ip
#capacity: 1
#leakspeed: "60s"
blackhole: 5m
labels:
 service: http
 type: scan
 remediation: true

Typo to fix

In this repo, global search for ressource.

ressource => resource.

crowdsecurity/seo-bots-whitelist: google bot comes from two different domain names

Currently only googlebot.com is implemented but we are also seeing requests from rate-limited-proxy-xxx.google.com. This is confirmed by gooogle documentation: https://support.google.com/webmasters/answer/80553?hl=en

False negative with sshd

As reported by users, we're missing some sshd-logs, ie.

Mar  5 20:24:14 testhost sshd[455340]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=178.200.xxx.xxx  user=root

cf. https://discourse.crowdsec.net/t/help-having-only-unparsed-logs/134/4

crowdsecurity/vsftpd-logs: support "permission denied" logs

When using userlist_file=/etc/vsftpd.allowed_users in vsftpd, it produce this log line for non allowed users:

Mon Apr 12 15:19:22 2021 [pid 15685] [www-data] FTP response: Client "1.2.3.4", "530 Permission denied."

Migrate data list from mitchellkrogza/nginx-ultimate-bad-bot-blocker

Several scenarios are using lists maintained and provided by mitchellkrogza/nginx-ultimate-bad-bot-blocker.
However, the maintainers don't seem to answer to issues or at least don't acknowledge them.

As they're under MIT licence we should do it so we can actively maintain it :

bad user agent

Whitelist semrush

Hi, semrush is actually a "legit" service : https://www.semrush.com/
Those services are actually blocked by the bad user agent list.

To avoid those block I have create a whitelist with those IPs (according to the semrush support) :

semrush_ips.txt

mariadb logs

Hi,
please add support for MariaDB logs instead of mysql.log only.

add .webp file extension to static content

The files with the extension .webp are static content.
They should be added in parsers/s02-enrich/http-logs.yaml

False positive with ssh-logs when disconnecting

the sshd-logs parser catches such logs as a failure

Mar 08 22:34:28 pve2 sshd[36002]: Received disconnect from 10.11.12.13 port 17534:11: disconnected by user
Mar 08 22:34:28 pve2 sshd[36002]: Disconnected from user dani 10.11.12.13 port 17534
Mar 08 22:34:28 pve2 sshd[35968]: pam_unix(sshd:session): session closed for user dani

(the relevant line is the first one, the others are just for some context)
This is done by the SSHD_RECE_PREAUTH pattern, included in SSHD_PREAUTH, used in crowdsecurity/sshd-logs. Only problem is that these are from a legitimate session, when I disconnect. This triggers false positives, including each time a backup ends.

tests : finalresults order is not maintained

finalresults section of the parser_results.yaml is not necessary maintained, making tests maintenance hard. You can have "random" differences because of that order, making review of tests annoying.

Google Bot new Domain?

We see traffic from google-proxy-xxx-xxx-xxx-xxx.google.com (xxx being the actual ip) that is blocked by crowdsecurity/http-probing. So far i can not not find any documentation if this is a legitimate google bot domain or not.

Here it only states "crawl-" and "rate-limited-proxy" ... but who knows ;-) https://support.google.com/webmasters/answer/80553?hl=en

Whitelist ahref

ahrefs is a legit service, actually blocked by the bad user agent scenarios.
To avoid those ban I have created a whitelist for the public IP.
According to the website you need to whitelist those IP : https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

ahrefs_ips.txt

apache2 vs nginx : unify output fields of http user agent

The user agent is not stored in the same field in apache2 and nginx :

Parsed.http_user_agent in nginx
Parsed.agent in apache2

Add scenario(s) for commonly probed directories

I have noticed that my VPS constantly gets these directories (and/or subdirectories of these) scanned and it would be awesome if there was a scenario in the hub that bans the IP on the first try.

/vendor/
/console/
/wp-content/
/wp-login.php
/wp-includes
/wp
/wordpress
/jenkins/
/Autodiscover
/solr/
/cgi-bin/
/admin
/boaform
/plus
/laravel
/shell
/cms
/owa
/manager
/phpmyadmin
/pma
/phpunit
/actuator
/mysql
/boaform
/0bef
/config
/?XDEBUG_SESSION_START=phpstorm
/TP
/thinkphp
/GponForm

These aren't sorted in any order.

I've also noticed a lot of CONNECT + {domain}:443 and GET + {url} where the domain and URL lead to a Chinese website.
Examples:

CONNECT www.baidu.com:443 HTTP/1.1
GET http://www.soso.com/ HTTP/1.1

Old scenario to update and move to postoverflow

This scenario is an old one written at the beginning and forgotten afterwards. It must we updated and moved to postoverflows.

http-crawl-non_statics false positives

I see a lot of false positives in http-crawl-non_statics when static files include a cache-buster get-param, like this:

x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET / HTTP/1.1" 301 237 "https://example.com/service-worker.js" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET /de/ HTTP/1.1" 200 37213 "https://example.com/service-worker.js" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET /media/js/35b822b6f2b73f695042666be2020dfc.js?1603273200 HTTP/1.1" 200 32596 "https://example.com/de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET /media/js/9e2e90e4591a5b9d495de7cd8a8c90a0.js?1603273200 HTTP/1.1" 200 24325 "https://example.com/de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET /media/css/de1ed404f1a75e02a098c8311b12c5d9.css?1603273200 HTTP/1.1" 200 79594 "https://example.com/de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:41 +0200] "GET /media/js/8360d8cd9b9e78aa8fca9f3e4f98be94.js?1603273200 HTTP/1.1" 200 471 "https://example.com/de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:42 +0200] "GET /media/js/cc9fc4c354fd10f3aacaba6f3cdd694b.js?1603273200 HTTP/1.1" 200 5183 "https://example.com/de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"
x.x.x.x - - [22/Oct/2020:18:13:42 +0200] "GET /service-worker.js HTTP/1.1" 304 0 "https://example.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1"

Not sure if this happens because of the param or if there is another reason. I'm happy tp provide detailed logs as private message.

traefik parser

I am running a couple of services using docker using traefik as a reverse proxy. I would like to enable an access log for traefik and have crowdsec check those whic hwould help secure all services behind the reverse proxy.

afaik traefik can write the access logs in json or in CLF format

(x-post from crowdsecurity/crowdsec#343)

Whitelist Ezoic

Ezoic provide ads service but in a special way.
To work you need to use their proxy server, then the IP that you server will see are the ezoic IP address.
If you block those IPs your site will not work anymore.

To avoid that this is a whitelist parser for ezoic.

name: ezoic/whitelist
description: "Whitelist IP Ezoic"
whitelist:
  reason: "Servers Ezoic"
  expression:
    - "any(File('ezoic_ips.txt'), { IpInRange(evt.Meta.source_ip ,#)})"
data:
  - source_url: https://client.siegler-informatique.fr/ezoic_ips.txt
    dest_file: ezoic_ips.txt
    type: string

The orignal file with the IP list can be find here : https://support.ezoic.com/kb/article/how-to-fix-origin-errors
Or the direct link here : https://support.ezoic.com/file.php?key=u6doqtm0x6kkkeyk4_diykzdfufopsz0&expires=1611878400&signature=2731f7ca6ca03ff2c5fc8258b928620de3b905c9&id=67323
Be carefull, adding so much IPs on crowdsec will slow down the start / stop process.
But outside that everything work fine.

apache2 : parser is too restrictive

Currently, the parser for apache2 requires referrer and user-agent to work. However, as reported on gitter, some users with other setups (ie. bitnami), will not have referer and user-agents in the logs.

Proposition would be to make those two fields optional :

pattern: '(%{IPORHOST:target_fqdn} )?%{COMMONAPACHELOG}( %{QS:referrer} %{QS:http_user_agent})?'

Instead of :

pattern: '(%{IPORHOST:target_fqdn} )?%{COMMONAPACHELOG} %{QS:referrer} %{QS:http_user_agent}'

Typos in scenario names

'probbing' should be 'probing

Bouncer inclusion: cs-express-bouncer

Hello,

I would like to suggest the addition of the cs-express-bouncer to the hub :

Source repository: https://github.com/crowdsecurity/cs-express-bouncer
Licence : MIT

Current status : stable

README/doc : https://github.com/crowdsecurity/cs-express-bouncer/README.md

Existing tests :

unit tests: npm run test
unit tests: npm run test:func
Short/Long description : OK

Howto : in README

At least one release : yes

http-logs enrich and scenario/http-probe fails

Hello,

I am still comparing fail2ban behavior and crowdsec and I encounter one issue that crowdsec fails to detect.
Attached is the scenario that fails with access_1.log and related crowdsec debugging logs cs_1.log.
I have attached a scenario that does not fails for comparison: access_2.log and related cs_2.log.

Note: I am using a custom pattern for my apache2 log format: pattern: '(%{URIHOST:target_fqdn} )?%{COMMONAPACHELOG} %{POSINT} %{POSINT} %{QS:referrer} %{QS:agent}'

I believe (I may be wrong) that the issue comes from the fact that simple root urls like / are not considered as non static file by the http-logs enricher and thus the evt.Parsed.static_ressource is not set and thus the http-probe scenario cannot filter it because the condition evt.Parsed.static_ressource == 'false' is undefined.

I am expected this to be considered as probing because return status code is 403.

cs_1.log
cs_2.log
access_1.log
access_2.log

Thanks in advance for your help and comments here.
Cheers.

proftpd collection

proftpd parser :

Nov 3 14:51:38 proftpd: pam_unix(proftpd:auth): authentication failure; logname= uid=0 euid=0 tty=/dev/ftpd27769 ruser=www-data rhost=1.2.3.4 user=www-data

proftpd scenario : bruteforce (heavily inspired from ssh-bf and such)

crowdsecurity/geoip-enrich : improve documentation about populated fields

On the hub, geoip-enrich documentation's is very lightweight and doesn't specify populated fields.

https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich only speaks about AS info for example, and doesn't mention latitude/longitude fields or source cidr range.

add qwant bot to goods actors collection

Apparently the reverse dns always finish with .search.qwant.com

host 91.242.162.40
40.162.242.91.in-addr.arpa domain name pointer qwantbot-91-242-162-40.search.qwant.com.

host 91.242.162.75
75.162.242.91.in-addr.arpa domain name pointer qwantbot-91-242-162-75.search.qwant.com.

host 91.242.162.65
65.162.242.91.in-addr.arpa domain name pointer qwantbot-91-242-162-65.search.qwant.com.

generic http brute force detection

Hello,

Similar to ssh brute force detection (ssh-bf plugin) and wordpress brute force detection (http-bf-wordpress_bf plugin) I believe it would be really great to have a generic http brute force detection (similar to fail2ban apache-auth).

My user story : I have a web server hosting several websites protected by different login/password forms. I wan to protect all of them from password brute force bots. A bot which fails to authenticate after several attempts during a given time window should be banned for a small period of time. several, small, time window should be up to users to define, but Crowsec should provide reasonable default.

Here are the offending access logs coming from a given bad guy IP (I hope I won't be sued posting his private bad activities ...):

127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:04:41:20 +0100] "POST /api/jsonws/invoke HTTP/1.1" 403 4741 628 412 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3
904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:04:41:22 +0100] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 4741 648 376 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (
KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:04:41:22 +0100] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 4741 698 349 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:04:41:23 +0100] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 403 4741 589 400 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) C
hrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:04:41:23 +0100] "GET /index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP21 HTTP/1.1" 403 4741 671 395 "-"
 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:28 +0100] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 531 326 354 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KH
TML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:28 +0100] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 531 376 323 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (K
HTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:29 +0100] "GET /index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP21 HTTP/1.1" 403 531 349 337 "-" "
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:29 +0100] "POST /api/jsonws/invoke HTTP/1.1" 403 531 306 330 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.390
4.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:29 +0100] "GET /solr/admin/info/system?wt=json HTTP/1.1" 403 531 267 448 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chr
ome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:10:00:29 +0100] "GET /?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php> HTTP/1.1" 403 531 321 308 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (
KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:16:16:24 +0100] "POST /api/jsonws/invoke HTTP/1.1" 403 4741 628 432 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3
904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:16:16:25 +0100] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 4741 698 324 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:16:16:27 +0100] "GET /wp-content/plugins/wp-file-manager/readme.txt HTTP/1.1" 403 4741 604 401 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML
, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:16:16:27 +0100] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 403 4741 589 171 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) C
hrome/78.0.3904.108 Safari/537.36"
127.0.0.1:443 91.241.19.84 - - [12/Nov/2020:16:16:32 +0100] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 4741 648 434 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (
KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:25 +0100] "POST /api/jsonws/invoke HTTP/1.1" 403 531 306 299 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.390
4.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:25 +0100] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 531 326 384 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KH
TML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:26 +0100] "GET /index.php?s=/Index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP21 HTTP/1.1" 403 531 349 350 "-" "
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:26 +0100] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 403 531 376 380 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (K
HTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:26 +0100] "GET /wp-content/plugins/wp-file-manager/readme.txt HTTP/1.1" 403 531 282 377 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:26 +0100] "GET /?a=fetch&content=<php>die(@md5(HelloThinkCMF))</php> HTTP/1.1" 403 531 321 361 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (
KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1:80 91.241.19.84 - - [12/Nov/2020:21:58:26 +0100] "GET /solr/admin/info/system?wt=json HTTP/1.1" 403 531 267 355 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chr
ome/78.0.3904.108 Safari/537.36"

Here are the related error logs:

[Thu Nov 12 04:41:20.235933 2020] [authz_core:error] [pid 31390:tid 140346445002496] [client 91.241.19.84:37984] AH01630: client denied by server configuration: /var/www/default/api
[Thu Nov 12 04:41:22.380018 2020] [authz_core:error] [pid 31391:tid 140347272451840] [client 91.241.19.84:58894] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 04:41:22.418966 2020] [authz_core:error] [pid 31390:tid 140346411431680] [client 91.241.19.84:37976] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 04:41:23.691671 2020] [authz_core:error] [pid 31390:tid 140346285606656] [client 91.241.19.84:58572] AH01630: client denied by server configuration: /var/www/default/
[Thu Nov 12 04:41:23.736691 2020] [authz_core:error] [pid 31390:tid 140346277213952] [client 91.241.19.84:57116] AH01630: client denied by server configuration: /var/www/default/index.php
[Thu Nov 12 10:00:28.979981 2020] [authz_core:error] [pid 31391:tid 140346906371840] [client 91.241.19.84:58862] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 10:00:28.984748 2020] [authz_core:error] [pid 31391:tid 140346872801024] [client 91.241.19.84:58852] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 10:00:29.524109 2020] [authz_core:error] [pid 31391:tid 140346881193728] [client 91.241.19.84:52162] AH01630: client denied by server configuration: /var/www/default/index.php
[Thu Nov 12 10:00:29.532327 2020] [authz_core:error] [pid 31391:tid 140345824237312] [client 91.241.19.84:50856] AH01630: client denied by server configuration: /var/www/default/api
[Thu Nov 12 10:00:29.892464 2020] [authz_core:error] [pid 31391:tid 140345799059200] [client 91.241.19.84:58486] AH01630: client denied by server configuration: /var/www/default/solr
[Thu Nov 12 10:00:29.919051 2020] [authz_core:error] [pid 31391:tid 140345706804992] [client 91.241.19.84:60096] AH01630: client denied by server configuration: /var/www/default/
[Thu Nov 12 16:16:24.932891 2020] [authz_core:error] [pid 31391:tid 140347007018752] [client 91.241.19.84:43182] AH01630: client denied by server configuration: /var/www/default/api
[Thu Nov 12 16:16:25.046709 2020] [authz_core:error] [pid 31391:tid 140346914764544] [client 91.241.19.84:43026] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 16:16:27.253420 2020] [authz_core:error] [pid 31390:tid 140346403038976] [client 91.241.19.84:46032] AH01630: client denied by server configuration: /var/www/default/wp-content
[Thu Nov 12 16:16:27.382592 2020] [authz_core:error] [pid 31390:tid 140347289253632] [client 91.241.19.84:44040] AH01630: client denied by server configuration: /var/www/default/
[Thu Nov 12 16:16:32.152530 2020] [authz_core:error] [pid 31391:tid 140346881193728] [client 91.241.19.84:50990] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 21:58:25.648238 2020] [authz_core:error] [pid 31390:tid 140346302392064] [client 91.241.19.84:59430] AH01630: client denied by server configuration: /var/www/default/api
[Thu Nov 12 21:58:25.973277 2020] [authz_core:error] [pid 31390:tid 140346436609792] [client 91.241.19.84:59394] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 21:58:26.215790 2020] [authz_core:error] [pid 31390:tid 140346428217088] [client 91.241.19.84:57748] AH01630: client denied by server configuration: /var/www/default/index.php
[Thu Nov 12 21:58:26.363833 2020] [authz_core:error] [pid 31390:tid 140346310784768] [client 91.241.19.84:59448] AH01630: client denied by server configuration: /var/www/default/vendor
[Thu Nov 12 21:58:26.443328 2020] [authz_core:error] [pid 31390:tid 140346445002496] [client 91.241.19.84:53068] AH01630: client denied by server configuration: /var/www/default/wp-content
[Thu Nov 12 21:58:26.720264 2020] [authz_core:error] [pid 31390:tid 140346260428544] [client 91.241.19.84:39296] AH01630: client denied by server configuration: /var/www/default/
[Thu Nov 12 21:58:26.735341 2020] [authz_core:error] [pid 31390:tid 140347264050944] [client 91.241.19.84:39314] AH01630: client denied by server configuration: /var/www/default/solr

What we see here is a bad guy scanning for vulnerable software installed on my server, but the URLs are protected URL and he gets a 403 Forbidden error code. This might not be a good example because this guy should have been caugh by crowdsecurity/http-backdoors-attempts or similar scenarios. What is interesting here is that the traffic continues despite the 403 Forbidden status code. A threashold (bucket leak) could be the scenario here.

Another better example is a true login failed attempt.
access logs:

foo.bar.com:443 192.168.1.1 - toto [13/Nov/2020:12:51:38 +0100] "GET /backup/ HTTP/2.0" 401 614 73 1488 "-" "curl/7.64.0"
foo.bar.com:443 192.168.1.1 - toto [13/Nov/2020:12:51:45 +0100] "GET /backup/ HTTP/2.0" 401 614 73 960 "-" "curl/7.64.0"

error logs:

[Fri Nov 13 12:51:38.825807 2020] [auth_basic:error] [pid 14509:tid 140347289204480] [client 192.168.1.1:43454] AH01617: user toto: authentication failure for "/backup/": Password Mismatch
[Fri Nov 13 12:51:45.361104 2020] [auth_basic:error] [pid 14508:tid 140347289204480] [client 192.168.1.1:43456] AH01617: user toto: authentication failure for "/backup/": Password Mismatch

So here we have 2 schenarios:

regular access to protected URLs (403 status code): Here a simple bucket leak algorithm should do.
Password Mismatch with 401 status code: Here again a simple bucket leak algorithm feets, but we could be more clever, grouping by user (not sure this is very usefull).

What do you think ?
Cheers.

Dovecot parser and scenario

Here is my dovecot log parser

onsuccess: next_stage
debug: false
filter: "evt.Parsed.program == 'dovecot'"
name: si/dovecot-logs
description: "Parse dovecot logs"
grok:
  pattern: "%{WORD:protocol}-login: %{DATA:dovecot_login_result}: user=<%{DATA:dovecot_user}>.*, rip=%{IP:dovecot_remote_ip}, lip=%{IP:dovecot_local_ip}"
  apply_on: message
statics:
    - meta: log_type
      value: dovecot_logs
    - meta: source_ip
      expression: "evt.Parsed.dovecot_remote_ip"

scenario/http-probing on same site but different port 80 and 443

Hello,

scenario/http-probing is grouping by "evt.Meta.source_ip + '/' + evt.Parsed.target_fqdn" .
In my logs I noticed the following:

127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:05 +0100] "HEAD / HTTP/1.1" 403 232 160 180 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a3pre) Gecko/20070330"
127.0.0.1:443 172.105.13.165 - - [25/Nov/2020:07:26:05 +0100] "GET /nmaplowercheck1606285565 HTTP/1.1" 403 4218 739 360 "-" "\"Mozilla/5.0"
127.0.0.1:443 172.105.13.165 - - [25/Nov/2020:07:26:06 +0100] "GET /evox/about HTTP/1.1" 403 4218 725 425 "-" "\"Mozilla/5.0"
127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:09 +0100] "HEAD / HTTP/1.1" 403 232 121 447 "-" "\"Mozilla/5.0"
127.0.0.1:443 172.105.13.165 - - [25/Nov/2020:07:26:42 +0100] "HEAD / HTTP/1.1" 403 3906 755 449 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a3pre) Gecko/20070330"
127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:48 +0100] "GET / HTTP/1.1" 403 543 120 350 "-" "\"Mozilla/5.0"
127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:50 +0100] "GET / HTTP/1.0" 403 527 36 233 "-" "-"
127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:50 +0100] "GET / HTTP/1.1" 403 524 75 333 "-" "-"
127.0.0.1:80 172.105.13.165 - - [25/Nov/2020:07:26:51 +0100] "POST /sdk HTTP/1.1" 403 543 586 406 "-" "\"Mozilla/5.0"
127.0.0.1:443 172.105.13.165 - - [25/Nov/2020:07:26:52 +0100] "POST /sdk HTTP/1.1" 403 4218 1181 413 "-" "\"Mozilla/5.0"

The guy is clearly probing my web server but alternates between port 80 and port 443.
What about grouping by host instead of fqdn ? It looks better for security and I am not sure it will increase the false positive cases. What do you think ?

Cheers.

Bouncer inclusion: cs-cloud-firewall-bouncer

Hello,

I would like to suggest the addition of the cs-cloud-firewall-bouncer to the hub :

Source repository: https://github.com/fallard84/cs-cloud-firewall-bouncer
Licence : MIT
Current Status : ?
README/doc : https://github.com/fallard84/cs-cloud-firewall-bouncer/blob/main/README.md
Existing tests :
- functional tests : https://github.com/fallard84/cs-cloud-firewall-bouncer
Short/Long description : NOK
Howto : in README
At least one release : Only pre-release
Continuous integration: https://github.com/fallard84/cs-cloud-firewall-bouncer/tree/main/.github/workflows

php-url-fopen

Based on the fail2ban filter I have make this one for crowdsec.
Work fine. This will detect request with http_args containing =http:// who is generally used to inject code on website.

type: leaky
name: si/php-url-fopen
description: "detect php url fopen"
debug: false
# request with http://
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.http_args contains '=http://'"
groupby: evt.Meta.source_ip
capacity: 3
leakspeed: "600s"
blackhole: 5m
labels:
 service: http
 type: scan
 remediation: true

false positive with bad user agent

Today I've got this false positive on bad user agent, this is the user agent :
Mozilla/5.0 (Linux; Android 4.4.2; Archos 59 Titanium Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.141 Mobile Safari/537.36

The link that I have is with "titan" in the bad user agent list that will match with Titanium here.

bucket crowdsecurity/ssh-bf with parsing errors

Hello maintainers,

today I installed crowdsec on my ubuntu 20.04. It kinda works, but it has two problems with the bucket crowdsecurity/ssh-bf:

cscli decisions list

+-----+----------+-------------------+----------------------+--------+---------+------------------------+--------+--------------------+----------+
| ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID |
+-----+----------+-------------------+----------------------+--------+---------+------------------------+--------+--------------------+----------+
| 402 | crowdsec | Ip:69.194.8.23 | crowdsecurity/ssh-bf | ban | US | IT7 Networks Inc | 6 | 3h55m56.161233828s | 6 |
| 301 | crowdsec | Ip:140.249.204.23 | crowdsecurity/ssh-bf | ban | CN | No.31,Jin-rong Street | 6 | 3h54m42.066105083s | 4 |
+-----+----------+-------------------+----------------------+--------+---------+------------------------+--------+--------------------+----------+

The IPs listed are missing the last number of the last octet:

Apr 21 00:01:19 cs sshd[16536]: Invalid user citrus from 140.249.204.233 port 50223
Apr 21 00:01:19 cs sshd[16536]: Received disconnect from 140.249.204.233 port 50223:11: Bye Bye [preauth]
Apr 21 00:01:19 cs sshd[16536]: Disconnected from invalid user citrus 140.249.204.233 port 50223 [preauth]

Apr 21 00:07:14 cs sshd[16882]: Invalid user acacia from 69.194.8.237 port 58898
Apr 21 00:07:15 cs sshd[16882]: Received disconnect from 69.194.8.237 port 58898:11: Bye Bye [preauth]
Apr 21 00:07:15 cs sshd[16882]: Disconnected from invalid user acacia 69.194.8.237 port 58898 [preauth]

I get some messages in the crowdsec.log:

time="21-04-2021 00:07:16" level=error msg="unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/ssh-bf: scope is Ip but Meta[source_ip] doesn't exist"
time="21-04-2021 00:07:18" level=error msg="unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/ssh-bf: scope is Ip but Meta[source_ip] doesn't exist"

I think the parser has problems with following lines.

Apr 21 00:07:17 cs sshd[16879]: error: PAM: Authentication failure for root from 221.131.165.56
Apr 21 00:07:17 cs sshd[16885]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.131.165.56 user=root

I feel very sorry, if these issues have been addressed yet, but I did not find them using the search function.

Best regards,
Ben

Bouncer inclusion: caddy-crowdsec-bouncer

Hello,

I would like to suggest the addition of the caddy-crowdsec-bouncer to the hub :

Source repository: https://github.com/hslatman/caddy-crowdsec-bouncer
Licence : Apache License, Version 2.0
Current status : ?
README/doc : https://github.com/hslatman/caddy-crowdsec-bouncer/README.md
Existing tests :
Short/Long description : OK
Howto : in README
Status: dev
At least one release : no

crowdsecurity / hub Goto Github PK

hub's Introduction

Foreword

Testing & Continuous integration

View & use existing tests

Create your own (parser) test

Debug your own (parser) test

Contributing

hub's People

Contributors

Stargazers

Watchers

Forkers

hub's Issues

cscli decisions list

Recommend Projects

Recommend Topics

Recommend Org