Demonstrating Data Offloading for Digital Twins with Kafka - Project for 521290S Distributed Systems, Spring 2023, @unioulu
Clone the repository with its submodules
git clone --recurse-submodules https://github.com/husmen/Kafka4Edge.git
cd Kafka4Edge
# If cloned already, update submodules
git submodule update --init
Run the main and edge clusters: NOTE: ALL COMMANDS ARE FOR LINUX DISTRIBUTION
docker-compose -f docker-compose-main.yml up -d
docker-compose -f docker-compose-edge.yml up -d --build --force-recreate
Make sure kafka-connect has started by checking logs (it'll take 30/40 seconds)
docker logs kafka-connect
After kafka-connect has started, run the following command to configure mongoDB as the sink for our kafka cluster
curl -d @config/kafka_connect/connect-mongodb-sink.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors
Then check:
- Kafka Producer Web API on http://localhost:8000/docs.
- MongoDB Client on http://localhost:3000.
- Grafana on http://localhost:4000.
When opening grafana:
- Enter
admin
as username and password - Go to Configurations -> Data-Source -> Add Data Source, then select prometheus
- Enter
http://kafka4edge_prometheus_1:9090
in the URL, Click Save & Test
To bring everything down:
docker-compose -f docker-compose-edge.yml down
docker-compose -f docker-compose-main.yml down
- Kafka cluster configuration
- Kafka edge node configuration, to join main cluster
- Simple Producer
- Simple Consumer
- MQTT Source
- MongoDB Sink
- Prometheus Monitoring
- Grafana Visualization
- Multithreaded Producer(s), to simulate multi sensor nodes and larger data loads
- Benchmarking, more info
- Attach to ditto
- Distributed database