contains Handy notes and Hands-on exercises

Grafana is an open-source data visualization and monitoring tool used to analyze and visualize data from various sources.

• It supports a wide range of data sources, including databases, cloud services, and time series databases like Prometheus and InfluxDB.

Data Sources:

• Grafana supports a variety of data sources, such as Prometheus, InfluxDB, Elasticsearch, MySQL, and more.
• Configure your data source(s) in Grafana by providing connection details and credentials.
• Each data source has specific query options and settings, so consult the official documentation for your chosen data source.

Dashboards:

• Dashboards in Grafana are the visual representation of your data.
• Create a new dashboard and add panels to display different metrics and visualizations.
• Choose the appropriate panel type based on the type of data you want to display (e.g., Graph, Table, Gauge).
• Customize the panel settings, including the query, time range, and visualization options.

Queries and PromQL:

• Grafana uses a query language called PromQL (Prometheus Query Language) for time series data.
• Learn PromQL to write queries that retrieve and manipulate data from the connected data source.
• PromQL supports various functions, aggregations, and operators. Refer to the official Prometheus documentation for detailed information.

Alerting:

• Grafana allows you to set up alerts based on specific conditions or thresholds.
• Define alert rules and configure notification channels (e.g., email, Slack) to receive alerts when the conditions are met.
• Ensure you have a supported alerting mechanism configured, such as Prometheus Alertmanager or an external service like PagerDuty.

Templating:

• Templating enables dynamic dashboards by allowing users to select variables or parameters.
• Use template variables to create flexible and reusable dashboards that can be customized on the fly.
• Templating variables can be based on data queries, custom values, or even obtained from other dashboards.

Plugins and Extensions:

• Grafana has a rich ecosystem of plugins and extensions that provide additional functionalities and integrations.
• Explore the Grafana plugin repository for extensions like custom panels, data source connectors, and authentication providers.

What is Observability?

• Grafana provides Observability which may be divided into Monitor, Logs, and Alerts

• For Logging, Loki should be used, and Prometheus is not the best option as it is widely used for time-series database. Diff between Grafana(Loki) and Prometheus :

• Loki and Prometheus are both open-source tools. While Loki is a log aggregation tool, Prometheus is a metrics monitoring tool.

• Loki's design is inspired by Prometheus but for logs.

• Prometheus collects metrics whereas Grafana is used as a Datastore

Grafana workflow :

• Promtail captures and delivers to Loki and Loki routes those to Grafana

  Promtail --> Loki ---> Grafana

Create an AWS EC2 instance(Ubuntu,t2 micro,security:allow all)

Installation: run the below commands

https://docs.google.com/document/d/1kHhk2LYss_c2pbBCCSXo4sqaqwNF3bj_MIiqm8L5b1U/edit

Note: Install the stable release version in the above doc.

Before installing Loki and Promtail,

run the grafana-server:

sudo systemctl start grafana-server

sudo systemctl enable grafana-server

• Grafana usually runs on port 3000, we can enable that in SG/inbound:

  

• To open the Grafana-server, open: public_ip_address:3000

• Default username and password for grafana are : admin, admin

  

  Add the data source :

  

Now, we can integrate with Loki, Prometheus, or anything based on use case.

For Grafana with Loki, we need to install Loki and promtail using docker.

# Install Docker

sudo apt-get install docker.io -y

give permissions to the current user : sudo usermod -aG docker $USER

Note: if it does not work, restart the server and it will work.

# Download Loki Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

# Run Loki Docker container

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

Loki docker container is running on 3100

• add port 3100 to SG/inbound

• Go to the URL: -> public_ip_address:3100/ready

Download Promtail Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Promtail Docker container

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

# carefully observe the above command, as there is `--link` that is used to connect from Promtail to Loki so that it can share the info and then
# Loki can send those logs to Grafana 

• Make sure both docker containers are running and linked up correctly from promtail to loki

• once it is done, open the Grafana dashboard : public_ip_address:3000 and add the datasource : Loki

• Add the localhost address of Loki : http://localhost:3100

• Once we hit explore, we will get:

  

• select the jobs and run the query :

  

• How and from where it is working ?

  • The path and jobs are given in promtail.yml that we pulled intially

    

  • we can go to that job path: /var/log/grafana

    

    we can see all the data but not formatted.

• update the promtail.yml with a new label,job and location so that it can pick data from this location

  - targets:
      - localhost
    labels:
      job: balaji_grafana_logs
      __path__: /var/log/grafana/*log

• we need to restart the promtail docker container: docker restart container_id

  

• once the data/logs are flowing to grafana UI, simply click Add to Dashboard

  

• we can choose to visualize and other:

  

  

Objective: Generate a graph that shows the number of words Nginx is repeated.

  sudo apt-get install nginx

• Go back to the Grafana dashboard, add a filter with operation>range functions> rate as below:

Adding more visualisations :

• Add error in the field and use operations > sum

---- EOF ---