calidog / certstream-python Goto Github PK

View Code? Open in Web Editor NEW

416.0 416.0 72.0 39 KB

Python library for connecting to CertStream

License: MIT License

Makefile 0.54% Python 99.46%

certstream-python's People

Contributors

Stargazers

Watchers

certstream-python's Issues

Filtering server-side?

[I'm not sure if I should file this issue here or with CaliDog/certstream-server. But I experience the problem with a Python program, so...]

If you are interested only by a subset of the certificates (for instance based on the domain name), filtering them client-side is a lot of work since the firehose is very active. My Python client takes a lof of CPU for that. And it also requires to download everything.

Is there a way to request a filtering server-side? I cannot find it in the code or the server nor in the documentation.

SIGPIPE not for Windows

From running the certclient.exe program:

Traceback (most recent call last):
  File "f:\programfiler\python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "f:\programfiler\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "F:\ProgramFiler\Python27\Scripts\certstream.EXE\__main__.py", line 5, in <module>
  File "f:\programfiler\python27\lib\site-packages\certstream\cli.py", line 8, in <module>
    from signal import signal, SIGPIPE, SIG_DFL
ImportError: cannot import name SIGPIPE

I'm using Python 2.7.13 on Win-10.
The use of termcolor tells me this package hasn't been tested on Windows at all. Why not coloramainstead?

as_der field available?

Just wonder: according to https://github.com/CaliDog/certstream-python/blob/master/README.md there should be a field named as_der in the returned datastructure. But with our certstream-python code we do not have this field in the results. Also when checking results here (https://certstream.calidog.io/) I cannot see that field. So I assume ;-) that docs specify a field that does not exist?

Error Connecting to Certstream

Since last week, I can't connect to Certstream calidog to get certification info. Does anyone know when this problem is going to be fixed?

Error connecting to CertStream - [Errno -3]

Hello,

I use a Python script to get stream. It works properly for 2 hours and get the error following:

Error connecting to CertStream - [Errno -3] Temporary failure in name resolution - Sleeping for a few seconds and trying again...

I kill and rerun the script but same error

Do you have any idea?

Thanks in advance.

SSL Certificate_Verify_Failed: On Mac Machine

[ERROR:certstream] 2020-06-04 01:17:14,112 - Error connecting to CertStream - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108) - Sleeping for a few seconds and trying again...

Source code:

import logging
import sys
import datetime
import certstream

def print_callback(message, context):
    logging.debug("Message -> {}".format(message))

    if message['message_type'] == "heartbeat":
        return

    if message['message_type'] == "certificate_update":
        all_domains = message['data']['leaf_cert']['all_domains']

        if len(all_domains) == 0:
            domain = "NULL"
        else:
            domain = all_domains[0]

        sys.stdout.write(u"[{}] {} (SAN: {})\n".format(datetime.datetime.now().strftime('%m/%d/%y %H:%M:%S'), domain, ", ".join(message['data']['leaf_cert']['all_domains'][1:])))
        sys.stdout.flush()




if __name__ == '__main__':
    logging.basicConfig(format='[%(levelname)s:%(name)s] %(asctime)s - %(message)s', level=logging.INFO)
    certstream.listen_for_events(print_callback, url='wss://certstream.calidog.io/')

Recursion error in certstream-python

See https://twitter.com/PaulWebSec/status/927800636703461376, and https://pastebin.com/6VXsfM7G (line 42).

It seems as though we're tripping Python's built-in stack overflow protection for some reason. The exact cause is unclear right now, but this seems to be a transient issue (as it happened a few hours into the run).

Relevant line (for google's indexes):

[ERROR:websocket] 2017-11-06 20:40:08,625 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0x7fda0c028780>>: maximum recursion depth exceeded while calling a Python object

Error connecting to CertStream - Handshake status 521 None

Hello there,

Since today, Tuesday December 5 04:27:42 UTC 2023, certstream-python can not connect to certstream:

# pip install certstream
Collecting certstream
  Downloading certstream-1.12.tar.gz (9.6 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: termcolor in /usr/lib/python3/dist-packages (from certstream) (1.1.0)
Requirement already satisfied: websocket-client>=0.58.0 in /usr/local/lib/python3.11/dist-packages (from certstream) (0.59.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from websocket-client>=0.58.0->certstream) (1.16.0)
Building wheels for collected packages: certstream
  Building wheel for certstream (setup.py) ... done
  Created wheel for certstream: filename=certstream-1.12-py3-none-any.whl size=4728 sha256=9d6e4c0ec98f7beb282199f9511f3bbf58d9d79fb2f2ac49d0fc71130483301d
  Stored in directory: /sec/root/.cache/pip/wheels/88/02/89/d6a96184779630f955590660a4d2ad038ff39c9c4e7a77d9c1
Successfully built certstream

# certstream
[ERROR:certstream] 2023-12-05 11:04:32,394 - Error connecting to CertStream - Handshake status 521 None - Sleeping for a few seconds and trying again...

And a visit to https://certstream.calidog.io/ returns a CloudFlare 521 error:

Any insight about the recovery of this awesome service ?

Cheers.

as_der field no longer included in certificate_update

Firstly, thanks for your invaluable service. It's been of great help to me, and I'm sure many others.

I just wanted to check: I have observed that recently content of certificate_update events no longer contain the as_der field in the leaf_cert section. Is this intentional or is it an issue?

Python not producing any domains after CloudFlare outage

If I click OPEN THE FIRE HOSE from https://certstream.calidog.io/ I'll see domains streaming in, but if I fresh run pip install certstream and run $ certstream --full I get no domains.

The issue started happening with the CloudFlare outage. Initially I saw some timeouts/resets around 2:10-2:15 PDT:

Certificate stream exception -> [Errno 104] Connection reset by peer
Certificate stream exception -> [Errno 110] Connection timed out

And then DNS resolution failures between 2:16-2:25:

Certificate stream exception -> [Errno -3] Temporary failure in name resolution

But since then I can establish the connection without error, but no messages are being sent over the socket:

$ certstream --full
[INFO:certstream] 2020-07-17 16:46:35,371 - Connection established to CertStream! Listening for events...
^C[INFO:certstream] 2020-07-17 16:47:55,299 - Kill command received, exiting!!

Newbie question about the cert stream

I am new to cert transparency, and I don't understand something I am seeing:

I get multiple X509LogEntries for a single host (I filtered out PreCerts).
(The layout is:
hostname
source log
update type
message type
authority
fingerprint
serial number, and
I added a timestamp of when I it was read from the certstream)

The host, source, and CA is the same, and the fingerprint and serial number is different.

I would really like to understand what I am looking. I was hoping you could explain it :)

Thx,
Adam

zz5b0zbooks.ml, Google 'Pilot' log, X509LogEntry, certificate_update, OCSP - URI:http://ocsp.comodoca4.comCA Issuers - URI:http://crt.comodoca4.com/COMODOECCDomainValidationSecureServerCA2.crt, 03:9D:02:34:E8:E7:DF:DC:10:19:24:8A:2C:A9:93:E9:20:71:95:93, 4514A395208FD36ADB8582D9394EE1CE, 2018-06-08 18-27-20

zz5b0zbooks.ml, Google 'Pilot' log, X509LogEntry, certificate_update, OCSP - URI:http://ocsp.comodoca4.comCA Issuers - URI:http://crt.comodoca4.com/COMODOECCDomainValidationSecureServerCA2.crt, 06:13:3E:FB:F5:01:5D:BD:02:E9:DC:E2:05:C3:D4:38:64:03:DD:68, D2C3635EFF70175FC53A20F09724CA65, 2018-06-08 18-27-26

zz5b0zbooks.ml, Google 'Pilot' log, X509LogEntry, certificate_update, OCSP - URI:http://ocsp.comodoca4.comCA Issuers - URI:http://crt.comodoca4.com/COMODOECCDomainValidationSecureServerCA2.crt, 87:67:E5:37:67:5E:57:1A:5F:B7:C7:C5:4F:92:6F:13:1D:9E:2B:98, DE706E2A682BBB580B63725941E49195, 2018-06-08 18-27-20

Missing tags and releases on pypi

The latest 1.9 release is not on PyPi and no release is tagged in git.

TypeError: listen_for_events() got an unexpected keyword argument 'url'

pip install certstream

$ python examples/echo.py 
Traceback (most recent call last):
  File "echo.py", line 25, in <module>
    certstream.listen_for_events(print_callback, url='wss://certstream.calidog.io/')
TypeError: listen_for_events() got an unexpected keyword argument 'url'

Is there something else missing to make the echo example work?

Thanks!

I found that the example snippet in Readme needs a URL

certstream.listen_for_events(print_callback, "wss://certstream.calidog.io")

Is possible to monitor phishing/fake certificate?

Hi,
I am sorry, I think I didn't understand purpose of this script, so I would like to ask about one thing.
I want to monitor all issued certificate for domain which contains my domain. Let's say, my domain is example.com so I would like to know about new certificates for e.g. xxxexample.com, example.asdfkjgfle.com etc. Is possible for this purpose use this script? Are there any other examples of using this script? Thank you very much for your answers.

Element "chain" no longer contained in the datastructure though still in the README?

SSL connection issue

I am getting the following error when trying to use certstream - (it used to work without any issues before though) : Error connecting to CertStream - Handshake status 502

Could you please help me resolve it in my env?

I have Python 2.7.14 running on CentOS 7.4.1708.

Thanks.

Server is down

I can't get anything from CertStream since April 11. I've tried both Python, Go, CLI and JavaScript

Error in connecting to CertStream...

Experiencing frequent websocket stream reconnects

Using version 1.10 and the sample code from the readme, roughly every minute a reconnect occurs, preceded by the following message:

[ERROR:root] 2019-10-11 16:38:18,582 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...

Websocket errors

I had some initial issues that I thought was entirely down to me and my code so I tried the example in the Readme just to have a working starting point. However, that throws errors as well. I've tried it with both Python 2.7 and 3.7 and both gives the same errors.

[ERROR:websocket] 2018-08-17 11:58:04,009 - error from callback <bound method CertStreamClient._on_open of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_open() takes exactly 2 arguments (1 given)
[ERROR:websocket] 2018-08-17 11:58:04,068 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_message() takes exactly 3 arguments (2 given)
[ERROR:websocket] 2018-08-17 11:58:05,445 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_message() takes exactly 3 arguments (2 given)
[ERROR:websocket] 2018-08-17 11:58:05,482 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_message() takes exactly 3 arguments (2 given)
[ERROR:websocket] 2018-08-17 11:58:05,528 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_message() takes exactly 3 arguments (2 given)
[ERROR:websocket] 2018-08-17 11:58:05,537 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7facdeefb3d0>>: _on_message() takes exactly 3 arguments (2 given)

I've tried both version 1.8 and 1.9 - just to check - and both gives the same errors under both versions of Python.

I'm still very open to it being me - but the sample code should in theory work - could there have been any changes to websocket that might need looking at from this projects code?

Setting CA bundle as parameter

I am using version 1.8 and get the error mentioned above:

> certstream 
[ERROR:root] 2017-12-18 15:04:01,994 - Error connecting to CertStream - cafile, capath and cadata cannot be all omitted - Sleeping for a few seconds and trying again...

module 'certstream' has no attribute 'listen_for_events'

Hello,

I'm trying to use CertStream with a proxy, however I get this error message.

Traceback (most recent call last):
File "certstream.py", line 1, in
import certstream
File "/home/.../certstream.py", line 6, in
AttributeError: module 'certstream' has no attribute 'listen_for_events'

Does anyone know a solution to this problem?

Instance parameter of on_error & on_open callbacks

Hi,

There is a small bug in the on_open and on_error callbacks. The documentation for both indicates that the first parameter to the callback should be instance, which is intended to the instance of the originating certstream listener (which could be handy in the case where a consumer application has more than one listener). From https://github.com/CaliDog/certstream-python/blob/9e56f7e6586f8a94b4538edb632445ced7a4c3ec/README.md:

def on_open(instance):
    # Instance is the CertStreamClient instance that was opened
    print("Connection successfully established!")

def on_error(instance, exception):
    # Instance is the CertStreamClient instance that barfed
    print("Exception in CertStreamClient! -> {}".format(exception))

However, in that case the invocations of those handlers need to pass an explicit self parameter (else Python complains TypeError about the missing parameter and the handlers are not invoked).
See:

certstream-python/certstream/core.py

Line 33 in 9e56f7e

self.on_open_handler()
certstream-python/certstream/core.py

Line 47 in 9e56f7e

self.on_error_handler(ex)

Fix:

diff --git a/certstream/core.py b/certstream/core.py
index d590413..7c70990 100644
--- a/certstream/core.py
+++ b/certstream/core.py
@@ -30,7 +30,7 @@ class CertStreamClient(WebSocketApp):
     def _on_open(self):
         certstream_logger.info("Connection established to CertStream! Listening for events...")
         if self.on_open_handler:
-            self.on_open_handler()
+            self.on_open_handler(self)
 
     def _on_message(self, message):
         frame = json.loads(message)
@@ -44,7 +44,7 @@ class CertStreamClient(WebSocketApp):
         if type(ex) == KeyboardInterrupt:
             raise
         if self.on_error_handler:
-            self.on_error_handler(ex)
+            self.on_error_handler(self, ex)
         certstream_logger.error("Error connecting to CertStream - {} - Sleeping for a few seconds and trying again...".format(ex))
 
 def listen_for_events(message_callback, url, skip_heartbeats=True, setup_logger=True, on_open=None, on_error=None, **kwargs):

Example: this starts the listener, but the open callback fails with an error (and the error is eaten, erk):

import certstream
from certstream.cli import parser as certstream_cli

def callback(_instance):
    print('Startup callback')

URL = certstream_cli.get_default('url')
certstream.listen_for_events(callback, URL, on_open=callback)

If the _instance parameter is removed from callback() (or patch applied as above) then the open callback message is seen on stdout.

SSL Connection error when connecting to local certstream server

Hi,

I set certstream server with exilir on cent OS 7 and seems to work fine.
(Web page works)

At another server, I set certstream-python and chaned wss server from calidog to my local server like below

certstream.listen_for_events(callback, url='wss://10.10.10.10:8080/') --> I changed port number

when I run the python code, error occur like this.

certificate_update: 0cert [00:00, ?cert/s]Error connecting to CertStream - [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:1123) - Sleeping for a few seconds and trying again...

Do you have any idea to solve this?

Returns the meaning of the field.

Hello, I have a suggestion. I wonder if you can provide a copy of the field meaning of demo (https://certstream.calidog.io/example.json) in the readme? Although you can guess the meaning of most fields by looking at words, it always feels that the description is not very accurate after translation. If there is a description of each field, it will feel better to read. Thank you.

Example script prints duplicate domains

when I run the example script, some domains are printed more that one time. some domains are printed again after a few minutes. what is the reason and how to prevent this?

import logging
import sys
import datetime
import certstream

def print_callback(message, context):
    logging.debug("Message -> {}".format(message))

    if message['message_type'] == "heartbeat":
        return

    if message['message_type'] == "certificate_update":
        all_domains = message['data']['leaf_cert']['all_domains']

        if len(all_domains) == 0:
            domain = "NULL"
        else:
            domain = all_domains[0]

        sys.stdout.write(u"[{}] {} (SAN: {})\n".format(datetime.datetime.now().strftime('%m/%d/%y %H:%M:%S'), domain, ", ".join(message['data']['leaf_cert']['all_domains'][1:])))
        sys.stdout.flush()

logging.basicConfig(format='[%(levelname)s:%(name)s] %(asctime)s - %(message)s', level=logging.INFO)

certstream.listen_for_events(print_callback, url='wss://certstream.calidog.io/')

Loosen the version constraint of websocket-client to avoid dependency conflicts

Hi, certstream locked the version constraint of websocket-client as websocket-client==0.48.0, which leads to a troubling scenario that its direct downstream project ['certstreammonitor','analyst-arsenal', 'bucket-stream','bounty-monitor'] has to lock websocket-client.

Could you please loosen the version constraint of websocket-client?
Benefit of this is that users using both of ['certstreammonitor','analyst-arsenal', 'bucket-stream','bounty-monitor'] and websocket-client can upgrade their third party libraries in a timely manner to reduce technical debts.

Solution

The dependency trees of your project and affected downstream projects are shown as follows.
Taking the version constraints of upstream and downstream projects into comprehensive consideration, you can

Loosen websocket-client==0.48.0 to be websocket-client>=0.48.0.
Loosen websocket-client==0.48.0 to be websocket-client==*.
Try to add an upper bound for websocket-client’ version constraint, according to your compatibility.

@Fitblip Please let me know your choice. I can submit a PR to fix this issue.

Thanks for your attention.
Best,
Neolith

ERROR:websocket

[ERROR:websocket] 2021-03-30 06:50:03,531 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7f400c29b3d0>>: 'chain'

i am getting this error using
websocket-client>=0.58.0 and
certstream==1.10

how i can solve this ?

Certs rate

Hi there!..

Does anyone know what certificate rate I should expect? Is there a public dashboard?
The following image shows the number of certs I received in the last month.

Thanks!

Missing tags for versions 1.10 - 1.12

The latest tag is 1.9, you could please add the newer releases as tags as well? That makes comparing the versions easier. Thanks!

WebSockets package updated which breaks certstream

Hi!

It seems that https://github.com/websocket-client/websocket-client has changed something, which causes certstream to fail. Could somebody fix this issue?

error from callback <bound method CertStreamClient._on_open of <certstream.core.CertStreamClient object at 0x1185f6e20>>: _on_open() takes 1 positional argument but 2 were given

Similar to #39 ?

Quick fix
Pin requirements to websocket-client==0.57.0

Func listen_for_events param setup_logger not used

Hello team,
I want to supress logging of the script on stdout and redirect it to the log file. I discovered a parameter in listen_for_events function called setup_logger that is by default set to True. However, this variable seems not to be used anywhere within the code itself, so it is kind of useless asit will not allow me to use it.

Any ideas how to workaround this?

Thanks, S.

Tests

Tests are needed for buildbots to verify this package is working correctly.

Error on Windows

I did install certstream with $ pip install certstream

but get the following error when executing it:

PS Desktop [09/17/2020 14:31:52]> certstream
Traceback (most recent call last):
  File "C:\Python37\Scripts\certstream-script.py", line 11, in <module>
    load_entry_point('certstream==1.11', 'console_scripts', 'certstream')()
  File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2793, in load_entry_point
    return ep.load()
  File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2411, in load
    return self.resolve()
  File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "c:\python37\lib\site-packages\certstream\cli.py", line 8, in <module>
    from signal import signal, SIGPIPE, SIG_DFL
ImportError: cannot import name 'SIGPIPE' from 'signal' (c:\python37\lib\signal.py)
PS Desktop [09/17/2020 14:32:05]>

Do you miss a reference?

Sever down.

Hi ,
It seems the server is down since Feb 3rd :

https://p.datadoghq.com/sb/e72980047-c37b10c02bf48fa80b013416ac899d15?refresh_mode=sliding&theme=dark&from_ts=1706816326600&to_ts=1707421126600&live=true

Proxy issues when using certstream

As obvious as it might have been, I did not check if the proxy was actually blocking this url.

Error when installing package on MacOS

Not sure if this is a known issue but wanted to raise it just in case:

⚡️•100% ➜ python3 -m pip install certstream
Defaulting to user installation because normal site-packages is not writeable
Collecting certstream
  Using cached certstream-1.11.tar.gz (9.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/pip-install-d8u3jrh3/certstream/setup.py'"'"'; __file__='"'"'/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/pip-install-d8u3jrh3/certstream/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/pip-pip-egg-info-pabbyqkp
         cwd: /private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/pip-install-d8u3jrh3/certstream/
    Complete output (157 lines):
    Traceback (most recent call last):
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 172, in run
        cmd = self.call_command('install_lib', warn_dir=0)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 158, in call_command
        self.run_command(cmdname)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 15, in run
        self.byte_compile(outfiles)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/command/install_lib.py", line 132, in byte_compile
        byte_compile(files, optimize=0,
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/util.py", line 466, in byte_compile
        compile(file, cfile, dfile)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/py_compile.py", line 157, in compile
        os.makedirs(dirname)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs
        makedirs(head, exist_ok=exist_ok)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs
        makedirs(head, exist_ok=exist_ok)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 213, in makedirs
        makedirs(head, exist_ok=exist_ok)
      [Previous line repeated 1 more time]
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 223, in makedirs
        mkdir(name, mode)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 310, in wrap
        path = self._remap_input(name, path, *args, **kw)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 452, in _remap_input
        self._violation(operation, os.path.realpath(path), *args, **kw)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 407, in _violation
        raise SandboxViolation(operation, args, kw)
    setuptools.sandbox.SandboxViolation: SandboxViolation: mkdir('/Users/nexxai/Library/Caches/com.apple.python/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu', 511) {}

    The package setup script has attempted to modify files on your system
    that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not
    support alternate installation locations even if you run its setup
    script by hand.  Please inform the package's author and the EasyInstall
    maintainers to find out if a fix or workaround is available.


    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu/termcolor-1.1.0/setup.py", line 37, in <module>
        "Topic :: Security :: Cryptography",
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: SandboxViolation: mkdir('/Users/nexxai/Library/Caches/com.apple.python/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu', 511) {}

    The package setup script has attempted to modify files on your system
    that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not
    support alternate installation locations even if you run its setup
    script by hand.  Please inform the package's author and the EasyInstall
    maintainers to find out if a fix or workaround is available.


    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
        self.gen.throw(type, value, traceback)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
        self.gen.throw(type, value, traceback)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/_vendor/six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu/termcolor-1.1.0/setup.py", line 37, in <module>
        "Topic :: Security :: Cryptography",
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: SandboxViolation: mkdir('/Users/nexxai/Library/Caches/com.apple.python/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu', 511) {}

    The package setup script has attempted to modify files on your system
    that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not
    support alternate installation locations even if you run its setup
    script by hand.  Please inform the package's author and the EasyInstall
    maintainers to find out if a fix or workaround is available.


    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/pip-install-d8u3jrh3/certstream/setup.py", line 15, in <module>
        setup(
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py", line 716, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 780, in resolve
        dist = best[req.key] = env.best_match(
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/dist.py", line 786, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1146, in run_setup
        raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: SandboxViolation: mkdir('/Users/nexxai/Library/Caches/com.apple.python/private/var/folders/dv/pj9mc1x10sdbffw03jh28bsr0000gn/T/easy_install-4ec8qpyu', 511) {}

    The package setup script has attempted to modify files on your system
    that are not within the EasyInstall build area, and has been aborted.

    This package cannot be safely installed by EasyInstall, and may not
    support alternate installation locations even if you run its setup
    script by hand.  Please inform the package's author and the EasyInstall
    maintainers to find out if a fix or workaround is available.

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

~/Library/Caches took 2s
⚡️•100% ➜

Error connecting to CertStream - [SSL: CERTIFICATE_VERIFY_FAILED]

I saw the previous issues with the same error that were closed, is there a new reason why this isn't working? (Error connecting to CertStream - [SSL: CERTIFICATE_VERIFY_FAILED]) unable to get local issuer certificate (_ssl.c:1051)

FIX for: _on_message() takes 2 positional arguments but 3 were given

Hello,

pip install certstream generates the following error:

To fix it I did the following steps (You may have to use pip and pip3 just in case):

pip uninstall certstream
pip uninstall websocket-client
git clone https://github.com/CaliDog/certstream-python.git && cd certstream-python

Now edit the requirements.txt file and change websocket-client>=0.56.0 to websocket-client==0.56.0

Now run: pip install -r requirements.txt

Everything should work now. If not try the following as well:
pip3 install -r requirements.txt
python3 setup.py install

Now you can just run the Python script with:

python3 certstream-python/certstream/cli.py

Note
I'm using Python 3.8.5

is certstream down?

I am using the sample script provide in this github and i am not getting any result from certstream anymore.

Error connecting to CertStream: SSLV3_ALERT_HANDSHAKE_FAILURE

I have the following error:

[ERROR:root] 2018-07-19 18:56:49,459 - Error connecting to CertStream - [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:579) - Sleeping for a few seconds and trying again...

Any ideas?

How to write certstream output to json file?

Hello,

I see this statement certstream --json | jq -r '.data | [.source.url, (.cert_index|tostring), .leaf_cert.subject.aggregated] | join(",")' on the certstream.io website, my question is how do I write the certstream results to a json file I can save.

I would greatly appreciate your help!

Error connecting to CertStream - Connection is already closed

I implemented this tool, following the given examples. I found out that sometimes ( usally every 2 minutes, more or less) the connection drops and it takes almost a minute to reconnect to the stream.

I don't know why this happens and this could probably lead to missing some hits i'm looking for.

Here's an example of log ( integrated with python logging module )


[ERROR:certstream] 2020-09-02 16:20:31,002 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:20:36,097 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:23:10,796 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:23:15,908 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:25:11,015 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:25:16,182 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:27:17,051 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:27:22,194 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:29:01,343 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:29:06,522 - Connection established to CertStream! Listening for events...

Any suggestion about how to fix this?

calidog / certstream-python Goto Github PK

certstream-python's People

Contributors

Stargazers

Watchers

Forkers

certstream-python's Issues

Solution

Recommend Projects

Recommend Topics

Recommend Org