Notification Manager manages notifications in multi-tenant K8s environment. It receives alerts, cloud event, and others (such as auditing, k8s events)
from different senders and then send notifications to various tenant receivers based on tenant label like namespace or user.
Supported senders includes:
- Prometheus Alertmanager
- Custom sender
- Prometheus (Coming soon)
- Cloud Event (Coming soon)
Supported receivers includes:
Notification Manager uses Receiver and Config CRDs to store notification configs like email, WeChat and slack. It also includes an operator to create and reconcile NotificationManager CRD which watches all receivers and configs, updates notification settings accordingly and sends notifications to users.
The incoming data (alert, cloud event and others) will cache in the cache firstly, then goes through steps such as silence, inhibit (coming soon), route, filter, aggregation, etc. Notifications will generate from data using template, then send to receivers and history webhook (if set).
Silence is a straightforward way to simply mute notifications for a given time. It uses Silence CRD to define
the silence policy. If incoming data matches an active silence, no notifications will be sent out for that data.
Route find all receivers the notifications will send to.
There are two ways to determine which receivers the notifications will send to, one is via Router CRD,
and the other is auto-matching via the namespace label in the notification.
Usually the incoming data contains a namespace label, Notification Manager uses this label to decide which receiver to use for sending notifications:
- For KubeSphere, Notification Manager will try to find tenants with the right to access the namespace from sidecar
and then find receivers with
user = xxxlabel. - For Kubernetes, Notification Manager will try to find receivers with
namespace = xxxlabel.
For data without a namespace label, for example alerts of node or kubelet, user can set up a receiver with type = global label to receive notifications without a namespace label. A global receiver sends notifications for all notifications received regardless any label. A global receiver usually set for an admin role.
How the two methods work together is determined by the routePolicy.
Filter filters the notifications sent to receivers. There are two ways to filter notifications. One is using alertSelector in the receiver,
the other is using tenant silence.
Aggregation groups notifications by groupLabels. Notifications in the same group will send together.
History is a webhook used to collect all notifications sent to receivers, it can be set via history.
We assume you already have a Kubernetes cluster (v1.16+). You can install one using KubeKey if you haven't.
# Deploy CRDs and the Notification Manager Operator:
kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/bundle.yaml
# Deploy default template:
kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/template.yaml
# Deploy built-in language packs.
kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/zh-cn.yamlhelm install notification-manager --create-namespace -n kubesphere-monitoring-system https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/notification-manager.tgzNotificationManager CRD Defines the desired notification manager deployment. The Notification Manager Operator ensures a deployment meeting the resource requirements is running.
We should create a NotificationManager CR first, skip this when using helm install.
kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/notification_manager.yamlNotification Manager uses port 19093 and API path /api/v2/alerts to receive alerts sent from Alertmanager.
To receive Alertmanager alerts, add webhook config like below to the receivers section of Alertmanager configuration file:
"receivers":
- "name": "notification-manager"
"webhook_configs":
- "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"Below is the data structure passed to the notification manager, please refer to Data for more details.
{
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "KubePodCrashLooping",
"container": "busybox-3jb7u6",
"instance": "10.233.71.230:8080",
"job": "kube-state-metrics",
"namespace": "pp1",
"pod": "dd1-0",
"prometheus": "kubesphere-monitoring-system/k8s",
"severity": "critical"
},
"annotations": {
"message": "Pod pp1/dd1-0 (busybox-3jb7u6) is restarting 1.07 times / 5 minutes.",
},
"startsAt": "2020-02-26T07:05:04.989876849Z",
"endsAt": "0001-01-01T00:00:00Z",
}
],
}A custom senders can send notifications using notification manager simply by sending data to http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts.
Now it's time to create the receiver and config to receive notifications, you can find guides to create them in receiver and config.
To customize the notification template, please refer to template.
# Build notification-manager-operator and notification-manager docker images
make build
# Push built docker images to docker registry
make push
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent