Prometheus adapter for Azure Data Explorer

Glossary

• Overview
• How to run
• How to develop
• Prometheus protocol buffer generator
• KQL examples

Overview

This is a production-ready implementation of Prometheus remote storage adapter for Azure Data Explorer (a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming).

The idea and first start for this project (previously it was a fork) became possible thanks to PrometheusToAdx repository.

The major functionality features:

1. Uses azure funtions;
2. Fully workable remote read/write;
3. Possible to write a large amounts of metrics because of ingesting data via sets of blobstorage -> eventgrid -> eventhub chain (see integration principe bellow);
4. Does downsampling for old metrics for save db size.

Information links:

• Prometheus remote storage integrations
• Azure Data Explorer data ingestion

Integration principe

miro

How to run

1. Required resources table

2. Step-by-step guide

1. Create Service Principal account;
2. Create Storage Account;
3. Create Azure Data Explorer Cluster;
4. Create Function App;
5. Add Contributor for Service Principal account;
6. Add Admin for Azure Data Explorer Cluster database (only for database);
7. Create Event Hub Namespace with Event Hub Instance
8. Create Event Grid Subscription with following options:
   • Topic type - Storage Accounts
   • Topic resource - Storage Account (2)
   • Filter event type - Blob created
   • Endpoint details - Event Hub (7)
9. Create database <cluster>-prometheus in Azure Data Explorer Cluster:
   • Permissions: Add View role for Service Principal (1)
   • Create tables RawData, Metrics (KQL bellow)
   • Data injestion:
     • Connection type - Blob storage
     • Storage Account / Event Grid - Storage Account (2) and Event Grid Subscription (8)
     • Event type - Blob created
     • Table - Metrics
     • Data format - CSV
     • Mapping name - CsvMapping

Init tables KQLs

.drop table Metrics ifexists
.drop table RawData ifexists

.create table RawData (Datetime: datetime, Timestamp: long, Name: string, Instance: string, Job: string, Labels: dynamic, LabelsHash: long, Value: real)

.create-or-alter table RawData ingestion csv mapping 'CsvMapping'
'['
'   { "column" : "Datetime", "DataType":"datetime", "Properties":{"Ordinal":"0"}},'
'   { "column" : "Timestamp", "DataType":"long", "Properties":{"Ordinal":"1"}},'
'   { "column" : "Name", "DataType":"string", "Properties":{"Ordinal":"2"}},'
'   { "column" : "Instance", "DataType":"string", "Properties":{"Ordinal":"3"}},'
'   { "column" : "Job", "DataType":"string", "Properties":{"Ordinal":"4"}},'
'   { "column" : "Labels", "DataType":"dynamic", "Properties":{"Ordinal":"5"}},'
'   { "column" : "LabelsHash", "DataType":"long", "Properties":{"Ordinal":"6"}},'
'   { "column" : "Value", "DataType":"real", "Properties":{"Ordinal":"7"}},'
']'

.alter-merge table RawData policy retention softdelete = 4d recoverability = disabled

.create table Metrics (LabelsHash: long, StartDatetime: datetime, EndDatetime: datetime, Name: string, Instance: string, Job: string, Labels: dynamic, Samples: dynamic)

How to develop

This repo contains .devcontainer for VSCode, which runs a set of containers with:

• Prometheus server as is
• Prometheus server with remote_read/remote_write for local debug
• Grafana
• Node-exporter node for some example data
• Dev-container with .net core for azure functions installation with all requirements and vscode extensions for C# development

Before run within .devcontainer:

You will need to setup some credentials to fully debug the functions. There are 2 ways for doing this:

• Check the .devcontainer/devcontainer.json file > remote Env section for environment vars.
• Or you can change the ./src/PromADX/local.settings.json config file. Don't forget to exclude this file from commit.

Prometheus protocol buffer generator

• This repo uses prometheus .proto and convert them via protoc util to C# classes
• Follow this documentation and scripts to get, compile and keep up the latest versions

Additional information

Prometheus read/write format

Samples:[
	{
		timestamp:
		value:
	},
	{
		timestamp:
		value:
	}...
],
Labels:[
    {
		name:
		value:
	},
	{
		name:
		value:
	}...
]

Get metrics KQL example

Metrics
| where (EndDatetime >= unixtime_milliseconds_todatetime(1591084670098)) and (StartDatetime <= unixtime_milliseconds_todatetime(1591092170098)) and ( ( Name == 'mysql_global_status_queries' ) )
| summarize Labels=tostring(any(Labels)), Samples=make_list( Samples ) by LabelsHash
| mv-apply Samples = Samples on
(
    order by tolong(Samples['Timestamp']) asc
    | summarize Samples=make_list(pack('Timestamp', Samples['Timestamp'], 'Value', Samples['Value']))
)