This template is a set of scripts to create a standard monitoring pipeline. Some key points in this repo:

• GitHub Actions for automating CI/CD pipeline
• Terraform and Ansible for creating and building infrastructure (keypair, security group, ec2, nginx, Prometheus and Grafana, ...)
• Monitor nginx status & statistics with Prometheus and Grafana

In short, Fluentd will continuously collect NGINX access logs and send them to Prometheus. Prometheus will scrape NGINX metrics endpoints to collect metrics data. Grafana will visualize the collected metrics and provide real-time dashboards for monitoring. You can access these dashboards or retrieve metrics data through the APIs for further analysis or integration with other systems.

To run this, you first need to create an AWS account and then get the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. In order to activate GitHub Actions (if you want), you also need to add these above keys to the repo:

By the way, make sure you have installed Terraform, and Ansible, then just follow the below simple format:

cd infras/
terraform init 
terraform apply # also need to enter `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`

Now you can see the public IP address of EC2 from the terraform output. Use it to access Nginx(port 8080), Prometheus(port 9090), Grafana(port 3000). Below is just some simple metrics from nginx, you can customize some more useful metrics by importing specific JSON file to Grafana dashboard.

In this repo, I did not focus on building APIs, so after all, you should access the EC2 and create ones then use the nginx as the reverse proxy. Monitoring nginx in this situation is more interesting. Okay, see you in the next template of mine 😄