contains Handy notes and Hands-on exercises
Grafana is an open-source data visualization
and monitoring tool
used to analyze and visualize data from various sources.
- It supports a wide range of data sources, including databases, cloud services, and time series databases like Prometheus and InfluxDB.
Data Sources:
- Grafana supports a variety of data sources, such as Prometheus, InfluxDB, Elasticsearch, MySQL, and more.
- Configure your data source(s) in Grafana by providing connection details and credentials.
- Each data source has specific query options and settings, so consult the official documentation for your chosen data source.
Dashboards:
- Dashboards in Grafana are the visual representation of your data.
- Create a new dashboard and add panels to display different metrics and visualizations.
- Choose the appropriate panel type based on the type of data you want to display (e.g., Graph, Table, Gauge).
- Customize the panel settings, including the query, time range, and visualization options.
Queries and PromQL:
- Grafana uses a query language called PromQL (Prometheus Query Language) for time series data.
- Learn PromQL to write queries that retrieve and manipulate data from the connected data source.
- PromQL supports various functions, aggregations, and operators. Refer to the official Prometheus documentation for detailed information.
Alerting:
- Grafana allows you to set up alerts based on specific conditions or thresholds.
- Define alert rules and configure notification channels (e.g., email, Slack) to receive alerts when the conditions are met.
- Ensure you have a supported alerting mechanism configured, such as Prometheus Alertmanager or an external service like PagerDuty.
Templating:
- Templating enables dynamic dashboards by allowing users to select variables or parameters.
- Use template variables to create flexible and reusable dashboards that can be customized on the fly.
- Templating variables can be based on data queries, custom values, or even obtained from other dashboards.
Plugins and Extensions:
- Grafana has a rich ecosystem of plugins and extensions that provide additional functionalities and integrations.
- Explore the Grafana plugin repository for extensions like custom panels, data source connectors, and authentication providers.
What is Observability?
-
Grafana provides
Observability
which may be divided intoMonitor
,Logs
, andAlerts
-
For
Logging
,Loki
should be used, andPrometheus
is not the best option as it is widely used fortime-series database
. Diff between Grafana(Loki) and Prometheus : -
Loki and Prometheus are both open-source tools. While
Loki
is alog aggregation tool
,Prometheus
is ametrics monitoring tool
. -
Loki's design is inspired by Prometheus but for
logs
. -
Prometheus collects
metrics
whereasGrafana
is used as aDatastore
Grafana workflow :
-
Promtail
captures and delivers toLoki
andLoki
routes those toGrafana
Promtail
-->Loki
--->Grafana
Create an AWS EC2 instance(Ubuntu,t2 micro,security:allow all)
Installation: run the below commands
https://docs.google.com/document/d/1kHhk2LYss_c2pbBCCSXo4sqaqwNF3bj_MIiqm8L5b1U/edit
Note: Install the stable release version
in the above doc.
Before installing Loki
and Promtail
,
run the grafana-server
:
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
-
Grafana usually runs on port
3000
, we can enable that in SG/inbound: -
To open the Grafana-server, open: public_ip_address:3000
-
Default username and password for grafana are :
admin
,admin
Add the data source :
Now, we can integrate with Loki, Prometheus, or anything based on use case.
For Grafana with Loki, we need to install Loki and promtail
using docker.
# Install Docker
sudo apt-get install docker.io -y
give permissions to the current user : sudo usermod -aG docker $USER
Note: if it does not work, restart the server and it will work.
# Download Loki Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
# Run Loki Docker container
docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml
Loki docker container
is running on 3100
Download Promtail Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
Run Promtail Docker container
docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml
# carefully observe the above command, as there is `--link` that is used to connect from Promtail to Loki so that it can share the info and then
# Loki can send those logs to Grafana
-
Make sure both docker containers are running and linked up correctly from promtail to loki
-
once it is done, open the Grafana dashboard :
public_ip_address:3000
and add the datasource : Loki -
Once we hit
explore
, we will get: -
select the jobs and run the query :
-
How and from where it is working ?
- update the
promtail.yml
with a new label,job and location so that it can pick data from this location
- targets:
- localhost
labels:
job: balaji_grafana_logs
__path__: /var/log/grafana/*log
-
once the data/logs are flowing to grafana UI, simply click
Add to Dashboard
-
we can choose to visualize and other:
Objective: Generate a graph that shows the number of words Nginx
is repeated.
sudo apt-get install nginx
Adding more visualisations :
- Add
error
in the field and useoperations > sum
---- EOF ---