Comments (17)
bzon
commented on May 27, 2024
Is this an error from prometheus? If yes, localhost should be the Kubernetes service name of the prometheus-msteams.
from prometheus-msteams.
vrathore18
commented on May 27, 2024
@bzon thanks for helping. I didn't get you. I am able to get the notification on slack.
When I ran kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,6783/TCP 9d
alertmanager-sns-forwarder-svc ClusterIP xxx.20.33.166 <none> 9087/TCP 1d
kube-prometheus ClusterIP xxx.20.216.216 <none> 9090/TCP 9d
kube-prometheus-alertmanager ClusterIP xxx.20.75.13 <none> 9093/TCP 9d
kube-prometheus-exporter-kube-state ClusterIP xxx.20.245.75 <none> 80/TCP 9d
kube-prometheus-exporter-node ClusterIP xxx.20.200.142 <none> 9100/TCP 9d
kube-prometheus-grafana LoadBalancer xxx.20.167.239 ae9b681f60d16... 80:31358/TCP 9d
prometheus-msteams ClusterIP xxx.20.137.248 <none> 2000/TCP 50m
prometheus-operated ClusterIP None <none> 9090/TCP 9d
I tried http://prometheus-msteams:2000/alertmanager
getting this error: Could not resolve host: prometheus-msteams
from prometheus-msteams.
bzon
commented on May 27, 2024
Can you try if you can resolve prometheus-msteams:2000 using curl or nc from the prometheus instance/pod where you got the following error:
level=error ts=2019-01-10T12:15:31.712451123Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post http://localhost:2000/alertmanager: dial tcp 127.0.0.1:2000: connect: connection refused"
from prometheus-msteams.
vrathore18
commented on May 27, 2024
Nope. I am not able to resolve the http://prometheus-msteams:2000 using curl.
from prometheus-msteams.
bzon
commented on May 27, 2024
Is your prometheus instance running in the Kubernetes cluster?
from prometheus-msteams.
vrathore18
commented on May 27, 2024
Yes. prometheus is up and running. I am getting notifications on slack, aws sns. I want to implemenet for teams. please suggests what to do
from prometheus-msteams.
bzon
commented on May 27, 2024
You may have a firewall issue with port 2000 in your cluster. Basically, you should be able to curl prometheus-msteams.monitoring.svc.cluster.local:2000 from any pod in the monitoring namespace.
from prometheus-msteams.
vrathore18
commented on May 27, 2024
So there any way I can change the port number?
from prometheus-msteams.
bzon
commented on May 27, 2024
You can either change it using kubectl edit svc or use helm upgrade/install with --set service.port=<Your Preference Port>
from prometheus-msteams.
vrathore18
commented on May 27, 2024
I changed the port now:
prometheus-msteams ClusterIP 172.20.137.248 9094/TCP 1h
Still getting the same error. :(
$ curl http://prometheus-msteams:9094
curl: (6) Could not resolve host: prometheus-msteams
from prometheus-msteams.
bzon
commented on May 27, 2024
@vrathore18 are you running curl inside the pod? It may be your kubernetes dns..
Can you try using prometheus-msteams.monitoring.svc.cluster.local instead of just prometheus-msteams.
And instead of using curl, try the ff commands:
nslookup prometheus-msteams
nslookup prometheus-msteams.monitoring.svc.cluster.local
dig prometheus-msteams
dig prometheus-msteams.monitoring.svc.cluster.localfrom prometheus-msteams.
vrathore18
commented on May 27, 2024
No. I am running on the cluster.
[ec2-user@ip-10-10-xxxx tempappd]$ nslookup prometheus-msteams.monitoring.svc.cluster.local
Server: 10.10.8.2
Address: 10.10.8.2#53
** server can't find prometheus-msteams.monitoring.svc.cluster.local: NXDOMAIN
[ec2-user@ip-10-10-1xxxx tempappd]$ dig prometheus-msteams.monitoring.svc.cluster.local
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.58.amzn1 <<>> prometheus-msteams.monitoring.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 59818
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;prometheus-msteams.monitoring.svc.cluster.local. IN A
;; AUTHORITY SECTION:
. 41 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2019011000 1800 900 604800 86400
;; Query time: 0 msec
;; SERVER: 10.10.8.2#53(10.10.8.2)
;; WHEN: Thu Jan 10 12:52:35 2019
;; MSG SIZE rcvd: 140
[ec2-user@ip-10-10-xxxx tempappd]$ curl prometheus-msteams.monitoring.svc.cluster.local
curl: (6) Could not resolve host: prometheus-msteams.monitoring.svc.cluster.local
from prometheus-msteams.
bzon
commented on May 27, 2024
Hold on. This is confusing. Are you running Prometheus in an EC2 instance or in a Kubernetes Pod? I think you are running Prometheus in an EC2 instance hence you can't resolve the Kubernetes service names.
from prometheus-msteams.
vrathore18
commented on May 27, 2024
I am running kubernates in aws. So all my kubernates setup is in ec2 instance.
This is what I am getting. you want to run the curl command inside any pod?
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-0 2/2 Running 0 9d
alertmanager-sns-forwarder-55db5448d6-b2ksj 1/1 Running 0 1d
kube-prometheus-exporter-kube-state-85975c8577-wv589 2/2 Running 0 9d
kube-prometheus-exporter-node-66xkq 1/1 Running 1 9d
kube-prometheus-exporter-node-9jpmd 1/1 Running 0 9d
kube-prometheus-grafana-57d5b4d79f-mvspq 2/2 Running 0 9d
prometheus-kube-prometheus-0 3/3 Running 1 9d
prometheus-msteams-54c5f57f9b-ltj68 1/1 Running 0 1h
prometheus-operator-d75587d6-kdffr 1/1 Running 0
from prometheus-msteams.
bzon
commented on May 27, 2024
I understand that. What I don't understand is why you are running the troubleshooting from the VM and not from the Pod.
Referring to your comment a while ago, you are running it from the EC2 VM..
ec2-user@ip-10-10-xxxx
from prometheus-msteams.
vrathore18
commented on May 27, 2024
@bzon: Ok the issue is fixed now. I am getting alerts on Team. What I did was used
url: 'http://prometheus-msteams.monitoring:9094/alertmanager'
You can close the ticket. Just one question. AWS SNS is also using webhook_configs in alertmanager.yaml and team is also using webhook_configs. I guess we can only use 1 webhook_configs in the alretmanager.yaml then how can i use both? any suggestion?
from prometheus-msteams.
bzon
commented on May 27, 2024
@vrathore18 Congrats!
Maybe try doing something like:
route:
group_by: ['alertname']
group_interval: 30s
repeat_interval: 30s
group_wait: 30s
receiver: 'low_priority_receiver' # default/fallback request handler
routes:
- receiver: aws_sns_receiver
match:
labelkey: labelvalue # same key value
- receiver: msteams_receiver
match:
labelkey: labelvalue # same key valueSo the same labeled alert goes to different receiver at the same time.
from prometheus-msteams.
Related Issues (20)
- [BUG] Helm repo no longer working HOT 4
- [FEATURE] Upgrade github.com/go-kit/kit to 0.12.0
- Release dependeny updates HOT 1
- Direct integration into Alertmanager? HOT 5
- [BUG] Give up the way to run "prometheus-msteams" in docker
- Image ARM Architecture HOT 2
- Grafana dashboard
- [BUG] Simulating a Prometheus Alerts to Teams Channel - results in not found
- Security Vulnerability [BUG]
- [BUG] cannot recieve alerts in MSTeams HOT 1
- [BUG] services dies when using systemctl reload prometheus_msteams.service HOT 1
- [BUG] Invalid memory address on Kubernetes v1.24 HOT 3
- [Message: Not Found] using uir static or dynamic. No Alerts are send to Teams channel
- Sunset of this repository HOT 3
- [BUG] When alerts are grouped, the card does not differentiate between status firing and resolved
- [BUG] Image tag v1.5.2 is not available anymore
- [FEATURE] Turn off automountServiceAccountToken
- [FEATURE] Timeline of new release HOT 4
- [BUG] The DEBUG logging level cannot be changed
- [BUG] Unable to pull the image from M2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-msteams.