Giter Club home page Giter Club logo

certstream-java's Introduction

CertStream-Java

See SSL certs as they're issued live.

This is a library for interacting with the certstream network to monitor an aggregated feed from a collection of Certificate Transparency Lists.

For this project I used only the very bare necessities to avoid causing dependency issues for folks downstream. The three libraries are:

        <dependency>
            <groupId>org.java-websocket</groupId>
            <artifactId>Java-WebSocket</artifactId>
            <version>1.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.30</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>

You need a java version later than Java 8u101 - earlier ones don't support the Let's Encrypt certificates used on our server.

You will note that there is no SLF4J appender in that list. The logging is up to you. If you're using SLF4J already it will just spit out the messages. If you aren't it is super cool and I very deeply hope you consider taking a look, it will make your logging better and your life richer. When I started working on this project I stuggled for a while with how to write the interface. I've mostly written abandonware or corporate stuff so I wasn't really sure where to start. Then someone (who wishes to remain nameless) suggested that I write an interface I would want to use and then write the implementation around that. So I did. I hope you like how it turned out.

Installing

Step 1: Add Repository to pom.xml:

<repository>
    <id>jitpack.io</id>
    <url>https://jitpack.io</url>
</repository>

Step 2: Add dependency to pom.xml:

<dependency>
    <groupId>com.github.CaliDog</groupId>
    <artifactId>certstream-java</artifactId>
    <version>0.3</version>
</dependency>

Usage

The main interface for this project is two functions, both of which take a single Consumer (read Lambda) as an argument. All you have to do is say what you want to do with the data and this library does the rest. Either pass a Consumer<String> into CertStream.onMessageString() or a Consumer<CertStreamMessage> into CertStream.onMessage(). All you have to to is use the -> or the ::. No really, take a look:

package example;

import com.google.gson.Gson;
import io.calidog.certstream.CertStream;

/**
 * Prints out Messages
 */
public class ExampleClient {

    /**
     * Main method
     * @param args unused
     */
    public static void main(String[] args)
    {
        //string version of the message
        CertStream.onMessageString(System.out::println);


        CertStream.onMessage(msg -> System.out.println(new Gson().toJson(msg)));
    }
}

Example data structure

The data structure coming from CertStream looks like this:

{
    "message_type": "certificate_update",
    "data": {
        "update_type": "X509LogEntry",
        "leaf_cert": {
            "subject": {
                "aggregated": "/CN=e-zigarette-liquid-shop.de",
                "C": null,
                "ST": null,
                "L": null,
                "O": null,
                "OU": null,
                "CN": "e-zigarette-liquid-shop.de"
            },
            "extensions": {
                "keyUsage": "Digital Signature, Key Encipherment",
                "extendedKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication",
                "basicConstraints": "CA:FALSE",
                "subjectKeyIdentifier": "AC:4C:7B:3C:E9:C8:7F:CB:E2:7D:5D:64:F2:25:0C:89:C2:AE:F0:5E",
                "authorityKeyIdentifier": "keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\n",
                "authorityInfoAccess": "OCSP - URI:http://ocsp.int-x3.letsencrypt.org\nCA Issuers - URI:http://cert.int-x3.letsencrypt.org/\n",
                "subjectAltName": "DNS:e-zigarette-liquid-shop.de, DNS:www.e-zigarette-liquid-shop.de",
                "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.letsencrypt.org\n  User Notice:\n    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\n"
            },
            "not_before": 1508123861.0,
            "not_after": 1515899861.0,
            "as_der": "::BASE64_DER_CERT::",
            "all_domains": [
                "e-zigarette-liquid-shop.de",
                "www.e-zigarette-liquid-shop.de"
            ]
        },
        "chain": [
            {
                "subject": {
                    "aggregated": "/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3",
                    "C": "US",
                    "ST": null,
                    "L": null,
                    "O": "Let's Encrypt",
                    "OU": null,
                    "CN": "Let's Encrypt Authority X3"
                },
                "extensions": {
                    "basicConstraints": "CA:TRUE, pathlen:0",
                    "keyUsage": "Digital Signature, Certificate Sign, CRL Sign",
                    "authorityInfoAccess": "OCSP - URI:http://isrg.trustid.ocsp.identrust.com\nCA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c\n",
                    "authorityKeyIdentifier": "keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\n",
                    "certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.root-x1.letsencrypt.org\n",
                    "crlDistributionPoints": "\nFull Name:\n  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl\n",
                    "subjectKeyIdentifier": "A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1"
                },
                "not_before": 1458232846.0,
                "not_after": 1615999246.0,
                "as_der": "::BASE64_DER_CERT::"
            },
            {
                "subject": {
                    "aggregated": "/O=Digital Signature Trust Co./CN=DST Root CA X3",
                    "C": null,
                    "ST": null,
                    "L": null,
                    "O": "Digital Signature Trust Co.",
                    "OU": null,
                    "CN": "DST Root CA X3"
                },
                "extensions": {
                    "basicConstraints": "CA:TRUE",
                    "keyUsage": "Certificate Sign, CRL Sign",
                    "subjectKeyIdentifier": "C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10"
                },
                "not_before": 970348339.0,
                "not_after": 1633010475.0,
                "as_der": "::BASE64_DER_CERT::"
            }
        ],
        "cert_index": 19587936,
        "seen": 1508483726.8601687,
        "source": {
            "url": "mammoth.ct.comodo.com",
            "name": "Comodo 'Mammoth' CT log"
        }
    }
}

certstream-java's People

Contributors

dependabot[bot] avatar fitblip avatar joshbooks avatar nitram22 avatar steely-glint avatar tim269 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

certstream-java's Issues

Empty jar from mvn install

Looks like the pom.xml does not find any sources.
I tried moving src/* to src/java/main and got a better build.

java.lang.InterruptedException: sleep interrupted

time to time Thread is stop by that exception

2021-12-25 13:45:26.663 INFO 427595 --- [onLostChecker-1] io.calidog.certstream.BoringParts : Thread sleep interrupted, weird, exiting

java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method) ~[na:na]
at io.calidog.certstream.BoringParts.onClose(BoringParts.java:67) ~[certstream-java-0.3.3.jar:na]
at io.calidog.certstream.CertStreamClientImplFactory$1.onClose(CertStreamClientImplFactory.java:80) ~[certstream-java-0.3.3.jar:na]
at io.calidog.certstream.CertStreamClientImpl.onClose(CertStreamClientImpl.java:49) ~[certstream-java-0.3.3.jar:na]
at org.java_websocket.client.WebSocketClient.onWebsocketClose(WebSocketClient.java:617) ~[Java-WebSocket-1.5.0.jar:na]
at org.java_websocket.WebSocketImpl.closeConnection(WebSocketImpl.java:522) ~[Java-WebSocket-1.5.0.jar:na]
at org.java_websocket.WebSocketImpl.closeConnection(WebSocketImpl.java:545) ~[Java-WebSocket-1.5.0.jar:na]
at org.java_websocket.AbstractWebSocket.executeConnectionLostDetection(AbstractWebSocket.java:212) ~[Java-WebSocket-1.5.0.jar:na]
at org.java_websocket.AbstractWebSocket.access$100(AbstractWebSocket.java:44) ~[Java-WebSocket-1.5.0.jar:na]
at org.java_websocket.AbstractWebSocket$1.run(AbstractWebSocket.java:188) ~[Java-WebSocket-1.5.0.jar:na]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]

Stream ends after 1 min

Hey there! I really love this project and the possibilities it does offer. Sadly recently it did somehow break. When I use the following as a test code:

public static void main(String[] args){
        CertStream.onMessageString(msg -> {
        	JSONArray jsonArray =  new JSONObject(msg).getJSONObject("data").getJSONObject("leaf_cert").getJSONArray("all_domains");
        	for(Object domain : jsonArray) {
        		System.out.println(domain.toString());        			
        	}
        });    	
}

The stream / program does just stop after about 1 minute or 7k - 8k domains.

Is there something I do wrong? Or did something else change? Thank you!

Expose getter and setter

Currently CertStreamMessage app has protected members that cannot be accessed from outside.
I suggest to decorate class with Lombok to expose getter and setter to be able to manipulate/filter incoming messages.
Thanks

NullPointerException on every incoming message

Hello,

First of all, thanks for your effort in maintaining this repository.

I've recently started to get an error on every incoming message from the server (last week I would say, although I am not sure because I've been on vacations). After some debugging, the issue seems to be caused by incoming messages not having the field "as_der" in "data" -> "leaf_cert" -> "as_der", which causes a null pointer in CertStreamCertificate.java, line 37, if statement.

I imagine this is due to a recent change in the message format. My current workaround is to surround the above-mentioned if statement with a pojo.asDer != null check, but I am not sure if that would be the optimal solution.

I would appreciate if you could take a look at the error and provide me some feedback.

Thanks in advance!

java.lang.NullPointerException: Cannot invoke "String.isEmpty()" because "pojo.asDer" is null

I am using the Java sources in Kotlin. I get the following errors on a lot of the messages:


java.lang.NullPointerException: Cannot invoke "String.isEmpty()" because "pojo.asDer" is null
	at io.calidog.certstream.CertStreamCertificate.fromPOJO(CertStreamCertificate.java:37)
	at io.calidog.certstream.CertStreamMessageData.fromPOJO(CertStreamMessageData.java:37)
	at io.calidog.certstream.CertStreamMessage.fromPOJO(CertStreamMessage.java:22)
	at io.calidog.certstream.CertStream.lambda$onMessage$1(CertStream.java:74)
	at io.calidog.certstream.CertStreamClientImplFactory$1.onMessage(CertStreamClientImplFactory.java:46)
	at io.calidog.certstream.CertStreamClientImpl.onMessage(CertStreamClientImpl.java:35)
	at org.java_websocket.client.WebSocketClient.onWebsocketMessage(WebSocketClient.java:342)
	at org.java_websocket.drafts.Draft_6455.processFrame(Draft_6455.java:599)
	at org.java_websocket.WebSocketImpl.decodeFrames(WebSocketImpl.java:370)
	at org.java_websocket.WebSocketImpl.decode(WebSocketImpl.java:202)
	at org.java_websocket.client.WebSocketClient.run(WebSocketClient.java:278)
	at java.base/java.lang.Thread.run(Thread.java:833)

This is my Main.kt:

import com.google.gson.Gson
import io.calidog.certstream.CertStream
import io.calidog.certstream.CertStreamMessage


fun main(args: Array<String>) {

    try {
        CertStream.onMessageString { x: String? ->
            if(!(x.isNullOrEmpty())){
                println(x)
            }
        }
        CertStream.onMessage { msg: CertStreamMessage? ->
            println(Gson().toJson(msg))
        }
    } catch (ex: Exception){
        print("Exeption" + ex.message)
    }
}

build.gradle.kts

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
    kotlin("jvm") version "1.6.0"
    application
}

group = "me.admin"
version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
    maven("https://jitpack.io")
}

dependencies {
    implementation("com.google.code.gson:gson:2.8.9")
    testImplementation(kotlin("test"))
    implementation("com.github.CaliDog:certstream-java:0.3")
}

tasks.test {
    useJUnit()
}

tasks.withType<KotlinCompile>() {
    kotlinOptions.jvmTarget = "1.8"
}

application {
    mainClass.set("MainKt")
}

IllegalStateException if entry contains "extra" field

Hi guys,

while running the onMessage method I noticed that from time to time incoming messages throw an error like "java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 7984 path $.data.leaf_cert.extensions." .

I did some debugging and found out that the error occurs when a CT-Log entry has the field under "leaf_cert" --> "extensions" --> "extra" that begins with an array. I've added a picture of an entry that produced an error:

issue

CT-Log entries that don't contain this "extra" field get parsed normally. Here's the Stack-Trace:

stacktrace.txt

Server down from 3 feb? Boringpart reconection loop

Hello!

Since 3 Febrary 2024 i cant get the domain using this project. Is the server down? The socket is performed but there is a loop always in the "boringpart" code and it tries to reconect all time in onclose-onopen functions.

Thank you!

Regards

Log spam: certstream.BoringParts

Thanks for this repo! Using it now for a few hours. However my log gets spammed with this message:

WARN certstream.BoringParts: OnClose was called wih i = 1006, s = , b = b

Is there a way I can suppress this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.