- Scala 2.13.10
- Spark 3.2.0
- Intellij IDEA Community Edition
- Apache Kafka 3.4.0
- Apache Zookeeper 3.8.1
- Elasticsearch 8.7.0
- Kibana 8.7.0
- JDK 11 while running the project files
- Configure the keys and emails in SendEmail.scala under consumer
The aim of this project is to develop a real-time data processing pipeline for server health logs to detect anomalies. By analyzing server health logs in real-time, this project aims to predict potential failures beforehand and enable system administrators to take proactive measures to prevent downtime. The analytics are displayed on a dashboard, providing a view of system performance and behavior. The project might also utilize cloud computing services for scalability, reliability, and cost-effectiveness. Overall, this project aims to provide a powerful tool for managing and optimizing system performance in real-time, enabling organizations to maximize uptime and minimize the risk of system failures.
- System sends alerts to the user if Server load is going to reach a threshold as a preventive measure
- System analyzes the server logs to detect any anomalies
- User monitors the server traffic activity via the visualizations on the Dashboard
- System parses data over long term for user to determine business specific decisions such as scalability
- Start the Zookeeper server - bin/zookeeper-server-start.sh config/zookeeper.properties
- Start the Kafka server - bin/kafka-server-start.sh config/server.properties
- Start ElasticSearch server - bin/elasticsearch
- Build SBT dependencies
- Run main/scala/Consumer/StreamProcessing.scala
- Run main/scala/Producer/Main.scala