Comments (9)
Thank you @christophkluenter1234 for reporting this. Unfortunately I wasn't able to reproduce this issue. Can you provide some information how you've deployed prometheus-msteams? Did you use the helm chart?
This error message indicates that the message card template does not have the key "sections" which is required for O365 Connector cards, and this exists in default-message-card.tmpl.
from prometheus-msteams.
I've just deployed off the helm chart 1.1.0 helm chart and I'm getting the same behavior as well. The only thing I changed from the chart values was the connector endpoint. At first glance it does look like the sections key is there.
"Created a card for Microsoft Teams /alertmanager"
time="2019-04-04T21:39:00Z" level=debug msg="[{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"808080\",\"summary\":\"\",\"title\":\"Prometheus Alert (firing)\",\"sections\":[{\"activityTitle\":\"[](http://example.com)\",\"facts\":[{\"name\":\"message\",\"value\":\"KubeControllerManager has disappeared from Prometheus target discovery.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecontrollermanagerdown\"},{\"name\":\"alertname\",\"value\":\"KubeControllerManagerDown\"},{\"name\":\"prometheus\",\"value\":\"default/soft-panda-prometheus-oper-prometheus\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true},{\"activityTitle\":\"[](http://example.com)\",\"facts\":[{\"name\":\"message\",\"value\":\"KubeScheduler has disappeared from Prometheus target discovery.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeschedulerdown\"},{\"name\":\"alertname\",\"value\":\"KubeSchedulerDown\"},{\"name\":\"prometheus\",\"value\":\"default/soft-panda-prometheus-oper-prometheus\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true},{\"activityTitle\":\"[](http://example.com)\",\"facts\":[{\"name\":\"message\",\"value\":\"There are 2 different semantic versions of Kubernetes components running.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch\"},{\"name\":\"alertname\",\"value\":\"KubeVersionMismatch\"},{\"name\":\"prometheus\",\"value\":\"default/soft-panda-prometheus-oper-prometheus\"},{\"name\":\"severity\",\"value\":\"warning\"}],\"markdown\":true},{\"activityTitle\":\"[](http://example.com)\",\"facts\":[{\"name\":\"message\",\"value\":\"97% throttling of CPU in namespace default for container prometheus-msteams in pod prometheus-msteams-6c947b454d-c75hq.\"},{\"name\":\"runbook\\\\_url\",\"value\":\"https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh\"},{\"name\":\"alertname\",\"value\":\"CPUThrottlingHigh\"},{\"name\":\"container\\\\_name\",\"value\":\"prometheus-msteams\"},{\"name\":\"namespace\",\"value\":\"default\"},{\"name\":\"pod\\\\_name\",\"value\":\"prometheus-msteams-6c947b454d-c75hq\"},{\"name\":\"prometheus\",\"value\":\"default/soft-panda-prometheus-oper-prometheus\"},{\"name\":\"severity\",\"value\":\"warning\"}],\"markdown\":true}]}]"
time="2019-04-04T21:39:00Z" level=error msg="Failed to parse json with key 'sections': Key path not found"
from prometheus-msteams.
Interesting. I'll have a look soon.
Additionally, I will exit the for loop in such a case to avoid getting 100% CPU workload.
from prometheus-msteams.
Hi,
I've got the same issue
poc_alertmanager_msteams | time="2019-04-05T08:11:42Z" level=error msg="Failed to parse json with key 'sections': Key path not found"
template:
{{ define "teams.card" }} { "@type": "MessageCard", "@context": "http://schema.org/extensions", "themeColor": "{{- if eq .Status "resolved" -}}2DC72D {{- else if eq .Status "firing" -}} {{- if eq .CommonLabels.severity "critical" -}}8C1A1A {{- else if eq .CommonLabels.severity "warning" -}}FFA500 {{- else -}}808080{{- end -}} {{- else -}}808080{{- end -}}", "text": "{{ .CommonAnnotations.summary }}", "title": "rePrometheus Alert ({{ .Status }})", "sections": [ {{$externalUrl := .ExternalURL}} {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }} { "activityTitle": "[{{ $alert.Annotations.description }}]({{ $externalUrl }})", "facts": [ {{- range $key, $value := $alert.Annotations }} { "name": "{{ reReplaceAll "_" "\\\\_" $key }}", "value": "{{ reReplaceAll "_" "\\\\_" $value }}" }, {{- end -}} {{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }} { "name": "{{ reReplaceAll "_" "\\\\_" $key }}", "value": "{{ reReplaceAll "_" "\\\\_" $value }}" } {{- end }} ], "markdown": true } {{- end }} ] } {{ end }}
verison prometheus: 2.8.1
version alertmanager: 0.16.1
I think, it's not problem with template, maybe alertmnager sends a new parametter and your server.go doesn't expect it
from prometheus-msteams.
Thanks for all the feedback. I will take a look into it latest on sunday.
from prometheus-msteams.
@kolikons, @psiservices-dstaples or @christophkluenter1234 can you please set log-level to DEBUG and provide more lines of the log output? I'd like to see the incoming prometheus alert.
from prometheus-msteams.
@kolikons, @psiservices-dstaples or @christophkluenter1234 can you please set log-level to DEBUG and provide more lines of the log output? I'd like to see the incoming prometheus alert.
Hi @Knappek
hm, very strange, right now i don't have like that issue, maybe some lables i've changed and it's fixed it
in any case the logs is the following:
poc_alertmanager_msteams | time="2019-04-09T06:53:29Z" level=info msg="Version: latest, Commit: 0578b32, Branch: HEAD, Build Date: 2019-03-26T21:29:58+0000" poc_alertmanager_msteams | time="2019-04-09T06:53:29Z" level=info msg="Parsing the message card template file: /etc/msteams/card.tmpl" poc_alertmanager_msteams | time="2019-04-09T06:53:29Z" level=info msg="Creating the server request path \"/alertmanager\" with webhook \"https://outlook.office.com/webhook/aaaa-bbbbb\"" poc_alertmanager_msteams | time="2019-04-09T06:53:29Z" level=info msg="prometheus-msteams server started listening at 0.0.0.0:2000" poc_alertmanager_msteams | time="2019-04-09T06:55:02Z" level=info msg="/alertmanager received a request" poc_alertmanager_msteams | time="2019-04-09T06:55:02Z" level=debug msg="{\"receiver\":\"prometheus-msteams\",\"status\":\"resolved\",\"alerts\":[{\"status\":\"resolved\",\"labels\":{\"alertname\":\"read_only_fs\",\"device\":\"/dev/sda1\",\"environment\":\"test\",\"fstype\":\"xfs\",\"hostname\":\"test.domain.com\",\"instance\":\"IP:9100\",\"job\":\"node_exporter\",\"metrics_path\":\"/metrics\",\"mountpoint\":\"/boot\",\"scheme\":\"http\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"/dev/sda1 with mount point: /boot is readonly. Reported by instance IP:9100 of job node_exporter.\",\"summary\":\"Host test.domain.com has readonly file system\"},\"startsAt\":\"2019-04-08T13:43:16.431352365Z\",\"endsAt\":\"2019-04-09T06:55:01.431352365Z\",\"generatorURL\":\"http://00c1f104dd51:9090/graph?g0.expr=node_filesystem_readonly+%3D%3D+1\\u0026g0.tab=1\"}],\"groupLabels\":{\"alertname\":\"read_only_fs\",\"hostname\":\"test.domain.com\"},\"commonLabels\":{\"alertname\":\"read_only_fs\",\"device\":\"/dev/sda1\",\"environment\":\"test\",\"fstype\":\"xfs\",\"hostname\":\"test.domain.com\",\"instance\":\"IP:9100\",\"job\":\"node_exporter\",\"metrics_path\":\"/metrics\",\"mountpoint\":\"/boot\",\"scheme\":\"http\",\"severity\":\"critical\"},\"commonAnnotations\":{\"description\":\"/dev/sda1 with mount point: /boot is readonly. Reported by instance IP:9100 of job node_exporter.\",\"summary\":\"Host test.domain.com has readonly file system\"},\"externalURL\":\"http://6e4cfdbca2a7:9093\",\"version\":\"4\",\"groupKey\":\"{}:{alertname=\\\"read_only_fs\\\", hostname=\\\"test.domain.com\\\"}\"}" poc_alertmanager_msteams | time="2019-04-09T06:55:02Z" level=debug msg="Size of message is 1057 Bytes (~1 KB)" poc_alertmanager_msteams | time="2019-04-09T06:55:02Z" level=info msg="Created a card for Microsoft Teams /alertmanager" poc_alertmanager_msteams | time="2019-04-09T06:55:02Z" level=debug msg="[{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"2DC72D\",\"text\":\"Host test.domain.com has readonly file system\",\"title\":\"Prometheus Alert (resolved)\",\"sections\":[{\"activityTitle\":\"[/dev/sda1 with mount point: /boot is readonly. Reported by instance IP:9100 of job node_exporter.](http://6e4cfdbca2a7:9093)\",\"facts\":[{\"name\":\"description\",\"value\":\"/dev/sda1 with mount point: /boot is readonly. Reported by instance IP:9100 of job node\\\\_exporter.\"},{\"name\":\"summary\",\"value\":\"Host test.domain.com has readonly file system\"},{\"name\":\"alertname\",\"value\":\"read\\\\_only\\\\_fs\"},{\"name\":\"device\",\"value\":\"/dev/sda1\"},{\"name\":\"environment\",\"value\":\"test\"},{\"name\":\"fstype\",\"value\":\"xfs\"},{\"name\":\"hostname\",\"value\":\"test.domain.com\"},{\"name\":\"instance\",\"value\":\"IP:9100\"},{\"name\":\"job\",\"value\":\"node\\\\_exporter\"},{\"name\":\"metrics\\\\_path\",\"value\":\"/metrics\"},{\"name\":\"mountpoint\",\"value\":\"/boot\"},{\"name\":\"scheme\",\"value\":\"http\"},{\"name\":\"severity\",\"value\":\"critical\"}],\"markdown\":true}]}]" poc_alertmanager_msteams | time="2019-04-09T06:55:03Z" level=info msg="Microsoft Teams response text: 1" poc_alertmanager_msteams | time="2019-04-09T06:55:03Z" level=info msg="A card was successfully sent to Microsoft Teams Channel. Got http status: 200 OK"
from prometheus-msteams.
I merged the PR that avoids having thousands of lines of Failed to parse json with key 'sections': Key path not found
in order to avoid having 100% CPU used.
I'll create an additional PR solving the root cause of this issue when I have been able to reproduce it.
from prometheus-msteams.
the latest version works as expected. No abnormal CPU usage anymore
from prometheus-msteams.
Related Issues (20)
- [BUG] Helm repo no longer working HOT 4
- [FEATURE] Upgrade github.com/go-kit/kit to 0.12.0
- Release dependeny updates HOT 1
- Direct integration into Alertmanager? HOT 5
- [BUG] Give up the way to run "prometheus-msteams" in docker
- Image ARM Architecture HOT 2
- Grafana dashboard
- [BUG] Simulating a Prometheus Alerts to Teams Channel - results in not found
- Security Vulnerability [BUG]
- [BUG] cannot recieve alerts in MSTeams HOT 1
- [BUG] services dies when using systemctl reload prometheus_msteams.service HOT 1
- [BUG] Invalid memory address on Kubernetes v1.24 HOT 3
- [Message: Not Found] using uir static or dynamic. No Alerts are send to Teams channel
- Sunset of this repository HOT 3
- [BUG] When alerts are grouped, the card does not differentiate between status firing and resolved
- [BUG] Image tag v1.5.2 is not available anymore
- [FEATURE] Turn off automountServiceAccountToken
- [FEATURE] Timeline of new release HOT 4
- [BUG] The DEBUG logging level cannot be changed
- [BUG] Unable to pull the image from M2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-msteams.