x0rz / phishing_catcher Goto Github PK

Started seeing this last week. Cert is running really slow, less that 50, down from 150+. Anyone else seeing this? I have tried through multiple connections on different environments, samething.

Have not been able to connect since 11/29.

certificate_update: 0cert [00:00, ?cert/s]Error connecting to CertStream - Handshake status 521 None -+-+- {'date': 'Tue, 12 Dec 2023 00:16:04 GMT', 'content-type': 'text/plain; charset=UTF-8', 'content-length': '15', 'connection': 'keep-alive', 'report-to': '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=47bybxwiYVWIOCgXfDQJsZ386aYaL1CUIcGsKym5Sel6GuLwI2XV7iZdJxa4pTyq6DKJLSBlxJzpc8KyY3mWXv7q0a3GQku2Ahdp7qISSF9qK92bQhIknydjz5KhZ3Lw%2BKQI0VRmJdM%3D"}],"group":"cf-nel","max_age":604800}', 'nel': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'x-frame-options': 'SAMEORIGIN', 'referrer-policy': 'same-origin', 'cache-control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'server': 'cloudflare', 'cf-ray': '8341c6c99c4736fc-YYZ', 'alt-svc': 'h3=":443"; ma=86400'} -+-+- b'error code: 521' - Sleeping for a few seconds and trying a

[ERROR:websocket] 2021-09-03 06:59:20,009 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7f083221f8e0>>: name 'suspicious' is not defined

Great project!
I've installed it via pip and are getting the following error;

$ ./catch_phishing.py
certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2018-08-14 17:03:04,419 - error from callback <bound method CertStreamClient._on_open of <certstream.core.CertStreamClient object at 0x7fa525fc73d0>>: _on_open() takes exactly 2 arguments (1 given)
[ERROR:websocket] 2018-08-14 17:03:06,919 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7fa525fc73d0>>: _on_message() takes exactly 3 arguments (2 given)

Tried updating all required components to the latest but still the same error,
tried python3 and python2.7.12, no difference
Guess its a certstremclient error..?

Anyone have any clues on what I should change?

I tried to search for a phishing site by turning off the override_suspicious.yaml: true instead of false, but it generated this line instead [ERROR:websocket] 2023-07-11 11:39:51,676 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7f86db4c8040>>: 'NoneType' object is not iterable

Did I do something wrong?

Just out of nowhere, phishing_catcher stopped showing certificates and upon executing throws the following error:

No handlers could be found for logger "certstream"

I've tried to pip install logging, but that didn't fix the issue. I've also downloaded a fresh version from Github and it doesn't work either.

Any ideas what could have happened? I'm running Python 2.7.18 or 3.9.2 on Debian 11.7.

Your help is much appreciated. Thank you.

Everything is in the title.

What do you think about adding some travis CI as much as checks for PEP-8 compliancy, sorted imports and so on?

I cloned the project installed , pip install -r requirements.txt
then while running python catch_phishing I am getting error:
maximum recursion depth exceeded while calling a Python object

~/phishing_catcher$ ./catch_phishing.py
certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2017-11-08 17:56:53,509 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0x7fb9b7901bd0>>: maximum recursion depth exceeded while calling a Python object
[ERROR:websocket] 2017-11-08 17:56:59,220 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0x7fb9b7901e10>>: maximum recursion depth exceeded while calling a Python object
^C[INFO:root] 2017-11-08 17:57:01,067 - Kill command received, exiting!!

When I installing phishing_catcher, I got the following error:

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/entropy/

I tried finding on Google if there's any similar packages, but there are too many (results: https://pypi.org/search/?q=entropy) and I'm not too sure which package to use.

May I know what should this package be replaced with?

1. unconfuse not working

from confusables import unconfuse
....
domain = unconfuse(domain)

It seems that now this is not a great idea. It does not work for me

2. it’s completely unclear how national domains are handled
   xn--d1acufc.xn--p1ai

Below error prompts when running the script

certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2020-12-20 15:23:01,269 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0xb65c70ac>>: _on_error() takes exactly 3 arguments (2 given)

Could you explain why I get the following error when trying to run the script?

[ERROR:root] 2017-12-03 09:15:13,384 - Error connecting to CertStream - [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:579) - Sleeping for a few seconds and trying again...

Seems to be an error with certstream and the encoding and decoding within the confusables.py file.

I'm getting an error when trying to install. When I go to the link to get Microsoft Visual
C++ Build Tools, I got a 404 Not Found!
Complete output from command c:\users\aalborzfard\appdata\local\programs\pyt
hon\python36\python.exe -u -c "import setuptools, tokenize;file='C:\Users\
AALBOR1\AppData\Local\Temp\pip-install-rhefjs4r\entropy\setup.py';f=getat
tr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close
();exec(compile(code, file, 'exec'))" install --record C:\Users\AALBOR1\App
Data\Local\Temp\pip-record-2otn21ya\install-record.txt --single-version-external
ly-managed --compile:
running install
running build
running build_ext
building 'entropy' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual
C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

Hello,
Your repo can help a lot of people, but you should check for domains that have special characters like the ỵ , ṙ .
Let me know about updates!

Netflix domain is generating too many false positive alerts

Running Docker Version 4.10.1 (82475)
System : macOS Monterey(12.4); MacBook Pro Intel Based

I have built docker container by running docker build . -t phishing_catcher from the phishing_catcher file. There were no errors during this time

When ever I try to run docker container by clicking the play button in 2-3 seconds the docker containers quits and Exit(2) is shown in status

How to resolve this issue.

Hey!
Could someone help me with that the phishing_catcher is disconnecting and reconnecting in every minutes?

certificate_update: 386526cert [42:33, 119.09cert/s][ERROR:root] 2024-01-28 09:55:53,383 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:55:58,669 - Connection established to CertStream! Listening for events...
certificate_update: 395220cert [43:19, 238.83cert/s][ERROR:root] 2024-01-28 09:56:38,713 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:56:44,068 - Connection established to CertStream! Listening for events...
certificate_update: 404788cert [44:33, 205.77cert/s][ERROR:root] 2024-01-28 09:57:53,020 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:57:58,294 - Connection established to CertStream! Listening for events...
certificate_update: 418396cert [45:52, 186.19cert/s][ERROR:root] 2024-01-28 09:59:12,524 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:59:17,842 - Connection established to CertStream! Listening for events...

I have updated the certstream module with pip, but I have the same issue, but less output of it:

certificate_update: 16887cert [01:23, 143.12cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 26518cert [02:27, 107.49cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 38765cert [03:35, 203.87cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 51049cert [04:44, 145.87cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 58365cert [05:34, 198.15cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...

Also, downgrading the websocket-client is not a solution anymore as i get this error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
certstream 1.12 requires websocket-client>=0.58.0, but you have websocket-client 0.52.0 which is incompatible.

Is this intended behaviour?

Thank you in advance!

For performance improvement, quick change from lists to sets.

Hi, Trying to get this phishing catcher set up but keep running into this error. I have followed the set up and have installed certstream, entropy and tqdm.

Every time I try running './catch_phishing.py' it comes up with this error message:

Traceback (most recent call last):
File "./catch_phishing.py", line 17, in
import certstream
ImportError: No module named 'certstream'

Know what the fix is?

@x0rz

I get the below error while installing . Please assist

phishing_catcher # pip install -r requirements.txt 
Collecting termcolor==1.1.0 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    
    ----------------------------------------

I Propose to score domains higher if they got more than two short parts towards the end so that

some-domain.co.uk

is ok but something like

some-domain.com.info.co.uk.com

is punished.

This pattern happens surprisingly often - i played around with this project just for two days and this is something that got my attention...

Create support to read custom include configs for suspicious_keywords, highly_suspicious and suspicious_tld arrays so that every commit avoids stomping on local changes.

Hello
I was wondering...
Maybe a option to search into certstream log for search specific tags and parameters, will be useful in Osint investigations.

Getting the Following Error while trying to run ./catch_phishing.py

Traceback (most recent call last):
  File "./catch_phishing.py", line 15, in <module>
    import certstream
ImportError: No module named certstream

I have installed all the requirements using pip3 install -r requirements.txt

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
Requirement already satisfied: termcolor==1.1.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: certstream==1.10 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (1.10)
Requirement already satisfied: tqdm==4.19.4 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (4.19.4)
Requirement already satisfied: tld==0.7.9 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (0.7.9)
Requirement already satisfied: python_Levenshtein==0.12.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (0.12.0)
Requirement already satisfied: websocket-client==0.48.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (0.48.0)
Requirement already satisfied: PyYAML==5.1 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 7)) (5.1)
Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.9/site-packages (from tld==0.7.9->-r requirements.txt (line 4)) (1.16.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from python_Levenshtein==0.12.0->-r requirements.txt (line 5)) (62.3.2)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: There was an error checking the latest version of pip.

I am Using a 2019 Intel Based MacBook Pro; macOS Monterey(12.4)

How to Resolve this?

I have also tried running the docker file but it is also showing an error( Don't know if its because of improper docker installation ). Will Create a new issue for that.

Hey,

in order to be able to tweak an external.yaml and assess its efficiency, I feel there is a good need of a simulation mode where the code will read domains from a file instead of a certstream server.

PR #58 is an attempt to implement that idea.

Cheers,

--
Mathieu

when I run the script, some domains are printed more that one time. some domains are printed again after a few minutes. what is the reason and how to prevent this?

Awesome script u wrote, although I might have a small improvement idea on it.

Maybe it's possible to whitelist all known google domains. There are various public list that state all legit google domains , for instance: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/List_of_Google_domains.html

My output gets a lot of legit google domains, so it's time for a clean up :)

Not working with python 2 whereas working fine with python 3

While playing around with the script and the stream i often saw something like this:

[!] Likely    : catfinder-beta.corp.amazon.com (score=84)
[!] Likely    : catfinder-test.corp.amazon.com (score=84)
[!] Likely    : catfinder.corp.amazon.com (score=83) 
[!] Likely    : cctracker.corp.amazon.com (score=81) 
[!] Likely    : cefeedback.corp.amazon.com (score=83)
[!] Likely    : cepromotions.corp.amazon.com (score=82)
[!] Likely    : contractcentral-gamma.corp.amazon.com (score=82)
[!] Likely    : contractcentral.amazon.com (score=80)
[!] Likely    : cornerstone.amazon.com (score=81)    
[!] Likely    : cosmos-dashboard.corp.amazon.com (score=82)
[!] Likely    : cube-dub.corp.amazon.com (score=83)  
[!] Likely    : cube-metrics.corp.amazon.com (score=84)
[!] Likely    : cube-pdx.corp.amazon.com (score=84)  
[!] Likely    : cube-preview.corp.amazon.com (score=84)
[!] Likely    : cube-showcase.corp.amazon.com (score=84)
[!] Likely    : cube.amazon.com (score=80)           
[!] Likely    : daenerys-beta.corp.amazon.com (score=84)
[!] Likely    : dvatools.corp.amazon.com (score=82)  
[!] Likely    : dxa-dashboard.corp.amazon.com (score=83)
[!] Likely    : fleet-widget.corp.amazon.com (score=85)
[!] Likely    : fm-console.corp.amazon.com (score=83)
[!] Likely    : fua.corp.amazon.com (score=81)       
[!] Likely    : gcxgiftfindertools-eu.corp.amazon.com (score=86)
[!] Likely    : gcxgiftfindertools-fe.corp.amazon.com (score=86)

Therefore i thought that one could find the last part acting like the tld (.com or .co.uk - compare #38 ) and ignore that. The afterwards rightmost part - in this example corp.amazon - is them checked for how often it appeared in the stream in the last say hour (or day,...) and based on that, its score is computed: the highrr this number, the higher the score...