Comments (4)
samber
commented on July 23, 2024
Do you have multiple Prometheus instances in federation mode or a remote-write setup ?
In that case, add a label to differentiate both jobs/prometheus.
from awesome-prometheus-alerts.
gskornowicz
commented on July 23, 2024
Hi @samber
No multiple Prometheus instances nor the remote-write setup.
It seems that the problem is due to multiple jobs per one instance?
from awesome-prometheus-alerts.
samber
commented on July 23, 2024
Yes, you may have multiple series with identical labels.
Do you use service discovery? Did you check if an exporter endpoint is declared twice in prometheus.yml ?
from awesome-prometheus-alerts.
gskornowicz
commented on July 23, 2024
I don't use service discovery.
I don't see any duplicated exporter endpoints
It's not clear to me what caused it, because labels are not identical:
{__name__="up", instance="server.company.com", job="bind", type="dns"} <- it's blackbox-dns exporter
{__name__="up", instance="server.company.com", job="node", os="Linux", type="vm"} <- it's node exporter
The job and the type are different, it is because instance label is the same?
attaching prometheus.yml
global:
scrape_interval: 20s
scrape_timeout: 20s
evaluation_interval: 15s
external_labels:
environment: prometheus.company.com
rule_files:
- /etc/prometheus/rules/*.rules
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- prometheus.company.com:9093
scrape_configs:
- job_name: prometheus
metrics_path: /metrics
static_configs:
- targets:
- prometheus.company.com:9090
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- job_name: grafana
static_configs:
- targets:
- prometheus.company.com:3000
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- job_name: alertmanager
static_configs:
- targets:
- prometheus.company.com:9093
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- job_name: metrics-snmp
metrics_path: /metrics
static_configs:
- targets:
- prometheus.company.com
relabel_configs:
- target_label: instance
replacement: exporter
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: metrics-vmware
metrics_path: /metrics
static_configs:
- targets:
- prometheus.company.com
relabel_configs:
- target_label: instance
replacement: exporter
- target_label: __address__
replacement: 127.0.0.1:9272
- job_name: metrics-blackbox
metrics_path: /metrics
static_configs:
- targets:
- prometheus.company.com
relabel_configs:
- target_label: instance
replacement: exporter
- target_label: __address__
replacement: 127.0.0.1:9115
- job_name: node
file_sd_configs:
- files:
- /etc/prometheus/file_sd/node.yml
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- job_name: bind
file_sd_configs:
- files:
- /etc/prometheus/file_sd/bind.yml
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- job_name: snmp-idrac
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-idrac.yml
params:
module:
- idrac
auth:
- idrac
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-synology
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-synology.yml
params:
module:
- synology
auth:
- synology
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-wlan
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-wlan.yml
params:
module:
- cisco
auth:
- cisco
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- source_labels:
- __address__
regex: .*:(.*)$
replacement: $1
target_label: __param_target
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-firewall
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-firewall.yml
params:
module:
- barracuda
auth:
- barracuda
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- source_labels:
- __address__
regex: .*:(.*)$
replacement: $1
target_label: __param_target
- source_labels:
- instance
regex: (.*).*1$
replacement: primary
target_label: boxrole
- source_labels:
- instance
regex: (.*).*2$
replacement: secondary
target_label: boxrole
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-powerwalker
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-powerwalker.yml
params:
module:
- powerwalker
auth:
- powerwalker
relabel_configs:
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- source_labels:
- __address__
regex: .*:(.*)$
replacement: $1
target_label: __param_target
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-switch
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-switch.yml
params:
module:
- switch
auth:
- switch
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: snmp-brocade
metrics_path: /snmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/snmp-brocade.yml
scrape_interval: 30s
scrape_timeout: 30s
params:
module:
- brocade
auth:
- brocade
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9116
- job_name: blackbox-http
metrics_path: /probe
file_sd_configs:
- files:
- /etc/prometheus/file_sd/blackbox-http.yml
relabel_configs:
- source_labels:
- module
target_label: __param_module
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115
- job_name: blackbox-icmp
metrics_path: /probe
params:
module:
- icmp
file_sd_configs:
- files:
- /etc/prometheus/file_sd/blackbox-icmp.yml
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __address__
target_label: instance
- source_labels:
- __address__
regex: (.*):.*$
replacement: $1
target_label: instance
- source_labels:
- __address__
regex: .*:(.*)$
replacement: $1
target_label: __param_target
- target_label: __address__
replacement: 127.0.0.1:9115
- job_name: blackbox-dns
metrics_path: /probe
params:
module:
- dns
static_configs:
- targets:
- one.one.one.one
- ns1.company.com
- ns2.company.com
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115
from awesome-prometheus-alerts.
Related Issues (20)
- 7.2.1. Loki process too many restarts label HOT 1
- Make alert description suffix customizable
- Invalid PostgresqlTooManyConnections HOT 1
- KubernetesNodeOutOfPodCapacity fails when multiple replicas of kube-state-metrics
- Broken on iOS?
- Adjust "Kubernetes Volume full in four days" query? HOT 2
- Node-exporter option has been renamed HOT 1
- Rule "Host RAID array got inactive" has misleading description HOT 1
- flux alerts HOT 3
- changed Kernel info breaks querie(s) HOT 1
- Rule "Host out of inodes" triggers false positive with FAT16 on FreeBSD HOT 6
- Host Memory underutilized uses a `rate` on the `node_memory_MemAvailable_bytes` gauge
- Need to fix use of deprecated apiserver_request_latencies_bucket metric HOT 1
- HostFilesystemDeviceError should use `for: 2m` HOT 1
- HostPhysicalComponentTooHot throws "many-to-many matching not allowed: matching labels must be unique on one side" HOT 1
- RedisTooManyMasters in a multi-cluster setup HOT 2
- "Haproxy HTTP Slowing Down" uses max total time
- How to do downtime monitoring for linux HOT 1
- Update Minio metric HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from awesome-prometheus-alerts.