jelmer / prometheus-xmpp-alerts Goto Github PK

View Code? Open in Web Editor NEW

31.0 5.0 10.0 223 KB

XMPP Web hook for Prometheus

License: Apache License 2.0

Python 96.46% Dockerfile 1.58% Makefile 1.95%

webhook xmpp prometheus prometheus-alertmanager

prometheus-xmpp-alerts's Introduction

prometheus-xmpp-alerts

A simple web hook that forwards prometheus alerts over XMPP.

Usage

To use, configure a web hook in alertmanager. E.g.:

receivers:
- name: 'jelmer-xmpp'
  webhook_configs:
  - url: 'http://192.168.2.1:9199/alert'

Edit the configuration file (defaults to /etc/prometheus/xmpp-alerts.yml):

jid: '[email protected]'
password: 'PASSWORD'
# Alternatively, set a 'password_command' that should write a password to
# stdout
# password_command: 'pass show xmpp-alertmanager'
to_jid: '[email protected]'
listen_address: '192.168.2.1'
listen_port: 9199

# Text message template as jinja2; defaults to html_template with tags stripped.
text_template: |
 {{ status.upper() }}: *{{ labels.alertname }}* at {{ labels.host or labels.instance }}:\
 {{ annotations.message }}. {{ generatorURL }}

And run the web hook::

$ python3 -m prometheus_xmpp --config=/etc/prometheus/xmpp-alerts.yaml

If you have amtool set up, then you can also allow to_jid to see existing alerts and manage silences. Set the amtool_allowed option to JIDs that are allowed to use amtool.

Docker file

You can build your own docker images using the Dockerfile in this directory, or use ghcr.io/jelmer/prometheus-xmpp-alerts. Provide your configuration in /config.yaml.

Message Format

The default message format looks something like this:

FIRING: AlertName at somehost: Alert Summary. https://prometheus.example.com/expr?...

The text_template option in the configuration can be used to customize the format, using jinja2.

Testing

The web hook can be accessed on three paths:

/alert: used by Prometheus to deliver alerts, expects POST requests with JSON body
/test: delivers a test message
/metrics: exposes statistics about number of alerts received

prometheus-xmpp-alerts's People

Contributors

Stargazers

Watchers

Forkers

mhamzahkhan azrdev jangrewe andir rudd-o purplesparkle e1mo chaos-jetzt joudinet ichthyx

prometheus-xmpp-alerts's Issues

SSLV3_ALERT_HANDSHAKE_FAILURE

I was trying to hook my Alertmanager up to my XMPP server (Prosody), but it looks like your code doesn't like it? :-(

I guess this happens because i don't allow SSLv3 anymore - any chance you could add support for TLSv1.1+?

https://xmpp.net/result.php?id=1456415

======== Running on http://127.0.0.1:9199 ========
(Press CTRL+C to quit)
ERROR    CERT: Invalid certificate trust chain.
ERROR    Fatal write error on socket transport
protocol: <__main__.XmppApp object at 0x7f757a832eb8>
transport: <_SelectorSocketTransport fd=6 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/slixmpp/xmlstream/xmlstream.py", line 500, in ssl_coro
    transp, prot = yield from ssl_connect_routine
  File "/usr/lib/python3.5/asyncio/base_events.py", line 801, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 827, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 865, in _on_handshake
    self._sock.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 761, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
INFO     connection_lost: (OSError(9, 'Bad file descriptor'),)
INFO     Connection lost, exiting.

MUC

Can I post alerts to chat room with prometheus-xmpp-alerts?

time data does not match format

After running it for a couple of days and getting plenty of alerts successfully, the alerts suddenly stopped and prometheus-xmpp-alerts keels on throwing errors:

ERROR    Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/prometheus-xmpp-alerts", line 131, in serve_alert
    text = '\n'.join(create_message(alert))
  File "/prometheus_xmpp/__init__.py", line 34, in create_message
    parse_timestring(alert['startsAt']).isoformat(timespec='seconds'),
  File "/prometheus_xmpp/__init__.py", line 26, in parse_timestring
    return datetime.strptime(ts, '%Y-%m-%dT%H:%M:%S.%fZ')
  File "/usr/local/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.7/_strptime.py", line 359, in _strptime
    (data_string, format))
ValueError: time data '2019-05-19T15:07:56.89489802Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
INFO     172.17.0.1 [20/May/2019:14:34:44 +0000] "POST /alert HTTP/1.1" 500 244 "-" "Alertmanager/0.17.0"

That seems a bit strange, because to me the timestamp does look like it's in the expected format?

support OMEMO

It would be great to support OMEMO. This depends on support in slixmpp: https://lab.louiz.org/poezio/slixmpp/merge_requests/1

support /metrics

prometheus-xmpp-alerts should support monitoring; possible variables:

uptime
number of reconnects
number of messages sent

immediate crash from aiowsgi

/var/lib/prometheus-xmpp-alerts/.local/bin/prometheus-xmpp-alerts:108: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(f)
detected unhandled Python exception in '/var/lib/prometheus-xmpp-alerts/.local/bin/prometheus-xmpp-alerts'
Traceback (most recent call last):
  File "/var/lib/prometheus-xmpp-alerts/.local/bin/prometheus-xmpp-alerts", line 152, in <module>
    port=config['listen_port'], host=config['listen_address'])
  File "/var/lib/prometheus-xmpp-alerts/.local/lib/python3.6/site-packages/aiowsgi/__init__.py", line 110, in create_server
    server_name=server.server_name,
AttributeError: 'MultiSocketServer' object has no attribute 'server_name'

this is current HEAD, i.e. daba8d9

config file is valid, but apparently irrelevant

jid: [email protected]
listen_address: localhost
listen_port: 9199
to_jid: [email protected]

slixmpp new release for inclusion

slixmpp has a new release (1.8.5) that fixes the scram-sha1 issues with logging into prosody servers, https://codeberg.org/poezio/slixmpp/releases/tag/slix-1.8.5.

Fix connections to Snikket instances: Snikket decided to forbid PLAIN
authentication, which is good but exposed a bug in slixmpp, which was trying
to do SCRAM-SHA-1-PLUS authentication on TLSv1.3 using the tls-unique channel
binding, which is forbidden by spec on this version of TLS as it has various
known attacks. TLSv1.3 has the tls-exporter binding which replaces tls-unique,
but we cannot currently use it in slixmpp because CPython does not support it.
For now, connections to Snikket instances will use SCRAM-SHA-1 without binding
(note that the stanzas may say SCRAM-SHA-1-PLUS, but it is the SCRAM payload
which is important here).

I have the same setup with prosody enforcing better auth, same as snikket.

Mark as available

It would be useful if the hook marked itself as available so that it is possible easily see that it is up and running.

Feature Request: Allow for custom templating (or some templates without timestamps)

I would like to either create my own templates or at least like to have some templates available which do not have a timestamp in them. For me the timestamp is not needed and only hinders my ability to get a quick glimpse of what is broken when looking at my notifications.

build docker image for arm64

Ideally, using github actions.

Sent message with id None

When using the /test endpoint or sending a test alert with curl, i get the response Sent message with id None, but don't receive any messages.

$ curl -XGET http://127.0.0.1:9199/test
Sent message with id None

INFO     172.17.0.1 [15/May/2019:18:09:06 +0000] "GET /test HTTP/1.1" 200 176 "-" "curl/7.52.1"

$ cat test.json
{
  "version": "4",
  "groupKey": "test",
  "status": "firing",
  "receiver": "xmpp",
  "groupLabels": {},
  "commonLabels": {"severity": "test"},
  "commonAnnotations": {},
  "externalURL": "https://alertmanager.faked.org",
  "alerts": [
    {
      "status": "firing",
      "labels": {"test": true},
      "annotations": {
        "summary": "test summary",
        "description": "test description"
      },
      "startsAt": "1985-04-12T23:20:50.52Z",
      "endsAt": "1985-04-12T23:20:50.53Z",
      "generatorURL": "curl"
    }
  ]
}

$ curl -XPOST http://127.0.0.1:9199/alert -d @test.json
Sent message with id None

INFO     172.17.0.1 [15/May/2019:17:58:08 +0000] "POST /alert HTTP/1.1" 200 176 "-" "curl/7.52.1"

breaks when it loses connection

Traceback (most recent call last):
File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/home/jelmer/prometheus_xmpp.py", line 72, in call
id_ = self.client.send(nbxmpp.protocol.Message(self.to_jid, text, typ='chat'))
File "/usr/lib/python3/dist-packages/nbxmpp/dispatcher_nb.py", line 576, in send
self.sm.request_ack()
File "/usr/lib/python3/dist-packages/nbxmpp/smacks.py", line 94, in request_ack
self._owner.Connection.send(r, False)
AttributeError: 'NoneType' object has no attribute 'send'

Support querying amtool

It would be great if it was possible to query the alertmanager via the bot; i.e. invoking 'amtool' and redirecting the output to the user.