CertStreamMonitor
Monitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.
CertStreamMonitor architecture relies on 3 scripts :
certstreammonitor.py
- this script runs as a daemon.
- reading the certstream feed, it selects hostnames covered by certificates that match your criteria (SearchKeyWords parameter in conf).
- it writes these hostnames along with its certificate relevant informations to the database.
scanhost.py
- this script can be executed as often as you like.
- it checks if site corresponding to the hostanme stored in DB is UP ot not.
- it collects informations about the sites that are up to DB and to a JSON file.
gethost.py
- Due to @nbeguier contribution, the project has also a
gethost.py
script that provides a way for security operators to request the last hostnames which have been detected bycertstreammointor.py
since X seconds.
- Due to @nbeguier contribution, the project has also a
check_rules.py
- Due to @nbeguier contribution, the project has also a
check_rules.py
script that provides a way for security operators to check their own rules, especially SearchKeywords, DetectionThreshold and BlacklistKeywords.
- Due to @nbeguier contribution, the project has also a
Features
- Monitoring:
- monitor
wss://certstream.calidog.io
CT logs aggregator server with certstream-python (see certstream-python), but you can choose, and operate, your own server (see certstream-server). - choose strings you want to monitor in
Subject Alt Names
field of certificates
- monitor
- Storing:
- store hostnames found along with its certificate relevant data into a sqlite3 database
- Alerting:
- for each hostname not already flagged as up : check if corresponding site is up or not
- if it's up :
- collects informations (IP address, AS informations, HTTP code, web page title, abuse email, (optional) google safe browsing status)
- write them to a JSON file in the
/alerts
directory (default value) to push forward investigation. - (optional) push them to a destination (through apprise package) such as an email address, a Slack channel or even Twitter account
- flags the hostname in the DB as up
- if it's not :
- the hostname will be checked until the Alert_Monitor_timelapse configuration variable (in days) will be reached
- when the limit is reached a string like
Stop checking on 2019-09-27
is included into the StillInvestig column
- if it's up :
- for each hostname not already flagged as up : check if corresponding site is up or not
Requirements
- Python 3
- certstream
- sqlite3
- ipwhois
- PySocks
- hues
- websocket-client
- apprise
Install
Install the requirements
$ pip3 install -r requirements.txt
Configuration file
You can find a configuration file example in 'conf' directory. Configurable parameters are:
SearchKeywords
: Keywords to look for (with '|' (or) as separator)DetectionThreshold
: set the minimum number of detected SearchKeywords in a hostname before writing it to DB. Under this value but above zero, detected hostnames are only written to logfile. Default value: 2.DBFile
: SQLite3 database file (the path and file will be created if don't exist)TABLEname
: The name of the database tableLogFile
: The logging file (the path and file will be created if don't exist)UAfile
: you can provide a User-Agent file to masquerade this value of requests (random change for each request)Alerts_dir
: you can specify where JSON alert files are written You can use the following strings to add time/date hashed based subdirectories: %%m -> month, %%d -> day, %%Y -> year, %%H -> hour, %%M -> minute. Example: Alerts_dir = ./alerts/%%Y/%%m/%%d
Optional:
Proxy
: allows to give a SOCKS or HTTP proxy to process your scanhost.py's requests (as Tor)Proxy_*
parameters : allow you to specify HTTP proxy informations (server, port[, user, password]) for CertStreamMonitor.py script to connect to the CT logs aggregator server.ACTServer
: you can specify the CT logs aggregator server of your choice. By default, it is the server run by Calidog Security.Safe_Browsing_API_Key
: indicate (if you want) your Google Safe Browsing API key in order to check hostnames that are UP against Google Safe Browsing Lookup API (How-To get an API key for the Safe Browsing Lookup API).Notification_Destination
: specify a notification destination as attended by apprise package. Documentation about the format of this parameter is available on the apprise Github page.BlacklistKeywords
: Keywords to ignore matched hosts (with '|' (or) as separator)
Usage
CertStreamMonitor.py
$ python3 ./CertStreamMonitor.py -c conf/example.conf
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
Connection established to CertStream! Listening for events...
[2018-03-12T11:40:15] cpanel.my-appleid-apple.net (SAN: mail.my-appleid-apple.net,my-appleid-apple.net,webdisk.my-appleid-apple.net,webmail.my-appleid-apple.net,www.my-appleid-apple.net) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 45:11:51:2D:24:D3:04:6E:DF:49:46:6D:64:56:67:4B:0A:48:8D:93) (StartTime: 2018-03-12T10:39:40)
[2018-03-12T11:41:19] cpanel.verification-account-apple-now.com (SAN: mail.verification-account-apple-now.com,verification-account-apple-now.com,webdisk.verification-account-apple-now.com,webmail.verification-account-apple-now.com,www.verification-account-apple-now.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 2D:90:F9:F7:83:F6:48:26:EF:C9:72:50:4B:06:FA:36:53:94:3C:8B) (StartTime: 2018-03-12T10:40:49)
[2018-03-12T11:41:36] login-apple.sytes.net (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: C7:78:2F:08:1E:CC:83:6C:06:EF:77:14:D2:1A:4E:06:A8:B3:F9:77) (StartTime: 2018-03-12T10:41:08)
[2018-03-12T11:42:26] cpanel.restore-account-apple.com (SAN: mail.restore-account-apple.com,restore-account-apple.com,webdisk.restore-account-apple.com,webmail.restore-account-apple.com,www.restore-account-apple.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: F3:CA:B1:C6:DE:4F:05:16:FD:06:F3:FF:29:8A:D3:1F:10:9D:50:1A) (StartTime: 2018-03-12T10:41:59)
[2018-03-12T11:49:37] securelogin.here.att.thysseankrupp.com (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 8F:9B:98:8D:5D:9B:03:0B:4F:62:56:40:C1:DE:9A:A4:FB:2D:A3:3E) (StartTime: 2018-03-12T09:22:41)
...
scanhost.py
$ python3 scanhost.py --help
-h --help Print this help
-c --config Configuration file to use
-f --fqdn-dirs Store JSON files in sub-directories based on the hostname
$ python3 ./scanhost.py -c conf/example.conf
Test all domains in DB for Internet Presence:
*********************************************
14:30:12 - ERROR - https://socialparadiseweb.cf.socialparadise.cf - Connection error
14:32:18 - ERROR - https://rapportannuel-assurancemaladie.paris - Connection error
14:32:23 - SUCCESS - HTTP 200 - socialmediaforsocialaction.com
Creating ./alerts/socialmediaforsocialaction.com.json : {'hostname': 'socialmediaforsocialaction.com', 'http_code': 200, 'cert_serial_number': '89:6C:03:F6:82:57:03:2A:A8:D0:E1:2F:E8:56:0E:32:83:E5:EC:29', 'webpage_title': 'Social Media for Social Action', 'ip_addr': '198.49.23.145', 'asn': '53831', 'asn_cidr': '198.49.23.0/24', 'asn_country_code': 'US', 'asn_description': 'SQUARESPACE - Squarespace, Inc., US', 'asn_abuse_email': '[email protected]'}
14:32:25 - ERROR - https://social.socialbride.co.za - Connection error
14:32:34 - SUCCESS - HTTP 503 - assurances-sociales.com
Creating ./alerts/assurances-sociales.com.json : {'hostname': 'assurances-sociales.com', 'http_code': 503, 'cert_serial_number': '1A:0D:45:D9:05:15:DC:17:6C:9F:9E:47:A5:62:03:D9:25:02:F9:3C', 'webpage_title': 'Accueil', 'ip_addr': '164.132.235.17', 'asn': '16276', 'asn_cidr': '164.132.0.0/16', 'asn_country_code': 'FR', 'asn_description': 'OVH, FR', 'asn_abuse_email': '[email protected]'}
gethost.py
$ python3 gethost.py --help
-h --help Print this help
-c --config Configuration file to use
--since Since when it displays findings (seconds)
$ python3 ./gethost.py -c conf/example.conf --since 36000 # 10 hours
Display all domains in DB for Internet Presence:
************************************************
socialparadiseweb.cf.socialparadise.cf None
rapportannuel-assurancemaladie.paris None
socialmediaforsocialaction.com 2019-06-12T15:54:31
social.socialbride.co.za None
check_rules.py
$ python3 check_rules.py --help
-h --help Print this help
-c --config Configuration file to use
-d --domain Domain name to check
# No match - Keywords not found.
$ ./check_rules.py -c conf/example.conf -d www.google.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Keywords not found.
# No match - Detection threashold not reached.
$ ./check_rules.py -c conf/example.conf -d paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Detection threashold not reached.
# No match - Blacklisted keywords.
$ ./check_rules.py -c conf/example.conf -d login-paypal.gouv
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Blacklisted keywords.
# This is a match, detection threashold reached.
$ ./check_rules.py -c conf/example.conf -d login-paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
This is a match, detection threashold reached.
Authors
- Thomas Damonneville ([email protected])
- Christophe Brocas ([email protected])
Contributors
Presentations
- SSTIC 2018 | June 2018 - C.Brocas, T. Damonneville: "Certificate Transparency ou comment un nouveau standard peut aider votre veille sur certaines menaces". Slides (fr), full article (fr), video (fr).
- Hack-it-n 2018bis | December 2018 - C. Brocas, T. Damonneville (given by C. Brocas): "CertStreamMonitor, use Certificate Transparency to improve your threats detection". Slides (en).
- Toulouse hacking Convention 2019 | March 2019 - C. Brocas, T. Damonneville (given by C. Brocas): "Certificate Transparency & threats detection, 24 months later". Slides (en), video (fr).
License
GNU GENERAL PUBLIC LICENSE (GPL) Version 3