florimondmanca / kafka-fraud-detector Goto Github PK

View Code? Open in Web Editor NEW

80.0 5.0 48.0 19 KB

🚨 Simple, self-contained fraud detection system built with Apache Kafka and Python

License: MIT License

Dockerfile 9.39% Python 90.61%

kafka python stream-processing showcase-app docker tutorial blog-post

kafka-fraud-detector's Introduction

Kafka Fraud Detector

This is the supporting repository for my blog post: Building A Streaming Fraud Detection System With Kafka And Python.

Install

This fraud detection system is fully containerised. You will need Docker and Docker Compose to run it.

You simply need to create a Docker network called kafka-network to enable communication between the Kafka cluster and the apps:

$ docker network create kafka-network

All set!

Quickstart

Spin up the local single-node Kafka cluster (will run in the background):

$ docker-compose -f docker-compose.kafka.yml up -d

Check the cluster is up and running (wait for "started" to show up):

$ docker-compose -f docker-compose.kafka.yml logs -f broker | grep "started"

Start the transaction generator and the fraud detector (will run in the background):

$ docker-compose up -d

Usage

Show a stream of transactions in the topic T (optionally add --from-beginning):

$ docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic T

Topics:

queuing.transactions: raw generated transactions
streaming.transactions.legit: legit transactions
streaming.transactions.fraud: suspicious transactions

Example transaction message:

{"source": "yGfZ1Xa6k1r0", "target": "N5RvY7RO5sQF", "amount": 217.46, "currency": "EUR"}

Teardown

To stop the transaction generator and fraud detector:

$ docker-compose down

To stop the Kafka cluster (use down instead to also remove contents of the topics):

$ docker-compose -f docker-compose.kafka.yml stop

To remove the Docker network:

$ docker network rm kafka-network

kafka-fraud-detector's People

Stargazers

Watchers

kafka-fraud-detector's Issues

Dockerfile question

Hi,

First off, thank you for the great tutorial!

I have a question regarding the Dockerfile for the generator and the detector

FROM python:3.6

WORKDIR /usr/app

ADD ./requirements.txt ./
RUN pip install -r requirements.txt
ADD ./ ./

CMD ["python", "app.py"]

Couldn't you delete ADD ./requirements.txt ./ and move ADD ./ ./ up before the Run Pip line since it adds all the necessary files anyway? If I'm correct, I can submit a PR if you'd like.

Also, the blog post source is not consistent with the repo source. See below

# Dockerfile
FROM python:3.7

WORKDIR /usr/app

ADD ./requirements ./
RUN pip install -r requirements.txt
ADD ./app.py ./

CMD ["python", "app.py"]

Actually the command line to watch the stream should be this one

docker exec kafkafrauddetector_broker_1 kafka-console-consumer --bootstrap-server localhost:9092 --topic streaming.transactions.legit

The kafkafrauddetector_broker_1 is the name of the containers start with image kafkafrauddetector_detector .

I am not very sure why the docker-compose not works.

Then I got the stream printed as

{"source": "uZxSDHD4wcE5", "target": "g0wgLDJftHBW", "amount": 150.45, "currency": "EUR"}
{"source": "3wiBuTZ9Sknw", "target": "r2UMsWnddfB4", "amount": 533.24, "currency": "EUR"}
{"source": "JgxFHAgwN8pj", "target": "YIEKjarfhMME", "amount": 273.84, "currency": "EUR"}
{"source": "UwpUOkML6rE0", "target": "lpth8JQQJYn3", "amount": 580.66, "currency": "EUR"}
{"source": "dqjbxgwsdJLV", "target": "JPmpzh2hiHzp", "amount": 152.41, "currency": "EUR"}
{"source": "tUb6sHBbbOnJ", "target": "wqyVub0ZnsuY", "amount": 91.31, "currency": "EUR"}
{"source": "1ljt52vUC2Aw", "target": "jHGOvGtbgW5E", "amount": 884.41, "currency": "EUR"}
{"source": "tpjFGUpeWE9y", "target": "rJ28GOX1OuRT", "amount": 874.21, "currency": "EUR"}
{"source": "vIhuz9ncvkev", "target": "1IiGlLtclgp8", "amount": 261.06, "currency": "EUR"}
{"source": "F7QziI9EksWS", "target": "jdz7KaaNbro1", "amount": 740.54, "currency": "EUR"}
{"source": "8PgQvfjGx7Ux", "target": "l6U6uHA9lp07", "amount": 502.15, "currency": "EUR"}
{"source": "EVqWKd9wRtjG", "target": "i3HHMtFfVmHK", "amount": 8.62, "currency": "EUR"}
{"source": "qreSXgRlSiDr", "target": "nVPkhU7gtcc9", "amount": 840.88, "currency": "EUR"}
{"source": "m6mmbKIBqEpI", "target": "010nHlF8WYj4", "amount": 282.72, "currency": "EUR"}

syntax issue with `transactions.py`

I'm unable to run the sample; I keep getting an invalid syntax error at the line for def _random_amount() when i run docker-compose

florimondmanca / kafka-fraud-detector Goto Github PK

kafka-fraud-detector's Introduction

Kafka Fraud Detector

Install

Quickstart

Usage

Teardown

kafka-fraud-detector's People

Stargazers

Watchers

Forkers

kafka-fraud-detector's Issues

Dockerfile question

Actually the command line to watch the stream should be this one

syntax issue with `transactions.py`

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent