- Spin up a multi-node Kubernetes cluster using vagrant
- Install and run the NGINX ingress controller.
- Install and run Prometheus, and configure it to monitor the Ingress Controller pods and Ingress resources created by the controller.
- Deploy an Ingress resource and two instances of a backend service using the “hashicorp/http-echo”. The Ingress should send requests with path “/foo” to one service; and path “/bar” to another. The services should respond with “foo” and “bar” respectively.
- Run a benchmarking tool
wrk
against the Ingress. - Generate a CSV file of time-series data using PromQL to fetch the following metrics from Prometheus:
- Average requests per second
- Average memory usage per second
- Average CPU usage per second
- Install Brew using https://brew.sh/
- Install virutalbox using
brew install --cask virtualbox
- follow steps https://www.virtualbox.org/wiki/Downloads
- Install vagrant using
brew install vagrant
- Install Helm and kubectl
brew install helm kubectl
- Install Go using https://go.dev/doc/install
- Install wrk http benchmarking tools using
brew install wrk
Bun below...
sudo mkdir -p /etc/vbox/
echo '* 0.0.0.0/0 ::/0' | sudo tee -a /etc/vbox/networks.conf
Clone the repo locally by running below command
git clone https://github.com/ramanagali/k8s-cluster-podman.git
cd k8s-cluster-prometheus
Run below command to provision new kubeadm cluster
./bootstrap.sh
- Run below command to Install latest Ingress Controller, Prometheus using helm.
- It will deploy foo, bar (http-echo) services (as NodePort) along with Ingress Resource
./install-addons.sh
NOTE: Wait 2-5mins to see ingress controller target healty
- Run below commands to get the Ingress URL
- Open the URL in browser (Cmd/Ctrl + Click)
export NODE_IP=192.168.56.10
export ING_PORT=$(kubectl get svc nginx-ingress-nginx-controller -o jsonpath="{.spec.ports[0].nodePort}")
echo "http://$NODE_IP:$ING_PORT/foo"
echo "http://$NODE_IP:$ING_PORT/bar"
Run below commands to get the Prometheus Server URL & open the URL in browser (Cmd/Ctrl + Click)
export NODE_IP=192.168.56.10
export GRAFANA_PORT=$(kubectl get svc -n prometheus prometheus-grafana -o jsonpath="{.spec.ports[0].nodePort}")
echo "http://$NODE_IP:$GRAFANA_PORT"
Run below commands to get the Grafana Server URL & open the URL in browser (Cmd/Ctrl + Click)
export NODE_IP=192.168.56.10
export PROM_PORT=$(kubectl get svc -n prometheus prometheus-kube-prometheus-prometheus -o jsonpath="{.spec.ports[0].nodePort}")
echo "http://$NODE_IP:$PROM_PORT"
Grafana Credentials
Username: admin
Password: prom-operator
Total HTTP Load test - duration is 30 seconds for each service
- 1st strike - 100 connections, 30 seconds for each service (foo & bar)
- 2nd strike - 200 connections, 30 seconds for each service
- 3rd strike - 400 connections, 30 seconds for each service
./loadtest.sh
Run Below commands in Prom server
PROM_URL=http://$NODE_IP:$PROM_PORT:9090
echo $PROM_URL
- Average requests per second
avg(rate(nginx_ingress_controller_nginx_process_requests_total[4h]))
- Average memory usage per second
avg(rate(process_resident_memory_bytes{service="nginx-ingress-nginx-controller-metrics"}[4h]))
- Average CPU usage per second
avg(rate(process_cpu_seconds_total{service="nginx-ingress-nginx-controller-metrics"}[4h]))
PROM_URL=http://$NODE_IP:$PROM_PORT:9090
echo $PROM_URL
- Average requests per second
curl -fs --data-urlencode 'query=avg(rate(nginx_ingress_controller_nginx_process_requests_total[4h]))' $PROM_URL/api/v1/query | jq -r '.data.result[] | .value[1]' > avg_req_ps.csv
- Average memory usage per second
curl -fs --data-urlencode 'query=avg(rate(process_resident_memory_bytes{service="nginx-ingress-nginx-controller-metrics"}[4h]))' $PROM_URL/api/v1/query | jq -r '.data.result[].value[1]' > avg_mem_ps.csv
- Average CPU usage per second
curl -fs --data-urlencode 'query=avg(rate(process_cpu_seconds_total{service="nginx-ingress-nginx-controller-metrics"}[4h]))' $PROM_URL/api/v1/query | jq -r '.data.result[].value[1]' > > avg_cpu_ps.csv
vagrant halt
NOTE: for strating the cluster refer 2. Bootstrapping
vagrant destroy -f