Looks like nice OSS monitoring tool, do we need native support for alerts from it?

https://github.com/louislam/uptime-kuma

Btw native integrations are stored here: https://github.com/grafana/oncall/tree/dev/engine/config_integrations

If you use uptime-kuma and need templates, please let us know! 👇

Should OnCall have native support of https://github.com/louislam/uptime-kuma ?

Hi Folks,

It would be nice to have a simple/plain install option or instructions for Oncall.
The current options all heavily imply existence of Docker/Helm/Kubernetes and this is clearly not the case in all environments.

Thanks

Alice, Bob, Sam are on call. I think it would be nice to show some "schedule quality" to help users keep schedules in a good shape.

100% = ((50% * balance) + (50% * working_hours_balance)) * ( 1 / gaps_rate)

We're trying to balance on call hours between engineers on call. Balance = 1 is good. Balance = 0 is bad.

balance = abs (min(Alice, Bob) hours / max(Alice, Bob) hours) * .... all pairs

We're trying to avoid people being on call outside of working hours. At least their on call time outside of working hours should be equal.

working_hours_balance = abs (min(Alice, Bob) hours outside of working hours / max (Alice, Bob) hours outside of working hours) * .... all pairs

We want to avoid gaps in on call schedule. Shift with no users is equal to gap:

gaps_rate = gaps hours / total observed period hours 

Hi
I have run OnCall as hobby environment via docker-compose (without Grafana) on the same host with existing Grafana (v9.0.1) running as binary.
Than I tried to configure OnCall Plugin (v1.0.3) and get error:

Tried connecting to OnCall: http://localhost:8080
Error: Request failed with status code 404, retry or check settings & re-initialize.

See the end of https://raw.githubusercontent.com/grafana/mimir/main/GOVERNANCE.md for proposed structure.

Hello everyone,

I have an existing kubernetes cluter with grafana already running, and today i installed Oncall using the helm chart provided in the repository.
The Oncall chart has been installed successfully in a different namespace than grafana.
When i try to configure the Oncall plugin on grafana i get the following error message, even though everything has been set properly

Tried connecting to OnCall: http://release-oncall-engine.oncall.svc.cluster.local:8080
Error: Request failed with status code 404, retry or check settings & re-initialize

Thank you

A few HN commenters raised a topic of a large deployment footprint: https://news.ycombinator.com/item?id=31740902

We deliver OnCall with redis + rabbit + mariadb + workers + web server because we focus on reliability a lot. For some users it's not a priority. We could think about some "tiny" deployment environment based on sqlite.

• Django UWSGI web server -> SQLite
• Celery workers -> One celery worker for all queues using Celery-SQLAlchemy -> SQLite (https://docs.celeryq.dev/en/v5.2.7/getting-started/backends-and-brokers/index.html#sqlalchemy). It's marked as not stable in docs btw. Also our celery version is 4.3.
• Redis -> We could go without cache at all or with in-memory Django engine cache. Probably impacted parts:
  • Alert Group page in web.
  • Grouping performance for heavy alert storm from Alertmanager. Typical K8S-generated alert storm.
  • Google Calendar parser relies on caching a lot.
  • Anything else?
• Rabbit -> Could be removed.
• MySQL -> Could be removed.

In total we'll need uWSGI + celery. Both could be packed in the one docker container.

Also we should make users absolutely aware of the performance/reliability limitations if they choose such a deployment.

Any thoughts?

On mobile notifications from slack show contents as "This content can't be displayed".

A similar issue:
grafana/grafana#42768
was fixed in Grafana with this PR:
grafana/grafana#45750

We likely need to pass through and add a top level text field to the slack messages we are sending where it is missing.

oncall hobby not connecting via https to existing grafana,

grafana v9.0.1

https://github.com/brotandgames/ciao

Btw native integrations are stored here: https://github.com/grafana/oncall/tree/dev/engine/config_integrations

If you use ciao and need templates, please let us know!

As Django is used in the core of oncall. It will be perfect for adding the ability to use PostgreSQL as the optional database engine.

For example, some teams prefer PostgreSQL over MySQL or already have production-ready PostgreSQL clusters.

As for now we only have an ability to create maintenance that starts right now.
As owner of service i would like to create scheduled maintenance that will start at some time in future.
It would be really great if every scheduled maintenance would have some kind of workflow attached to it for example simple approval would be greatly appreciated.

Since I've already got a locally running MySQL server, it conflicts with the one that the compose file is trying to setup. It would be great if there was out of the box support for referencing an external MySQL server.

How would I update a page showing all incidents for an integration with a status filter? If I reload the filters are gone.

We show raw content of last alert in the template editor (payload example):

Users are missing the ability to check raw content of some specific alerts in the history. May be add the button here: ?

Update and improve escalation chains and routes sections of Grafana OnCall documentation,

For now celery workers are using SIGTERM shutdown within the container itself, thus increasing Restarts value each time.

It would be better to use kubeapi for such task to ensure graceful restart for the service, e.g: client-go example

Docs:

• https://docs.microsoft.com/en-us/microsoftteams/platform/messaging-extensions/what-are-messaging-extensions
• https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook

Description

1. Maintenance page shows all the maintenances from all teams
2. If integration does not belong to the team it's name not displayed in the table
3. If integration does not belong to the team it's not possible to stop it

Steps to reproduce

1. In Grafana OnCall choose Team 1 in the top right corner
2. Create any integration on the Integrations page
3. Create a maintenance on Maintenance page
4. Switch to Team2
5. Try to stop maintenance

Screenshot

This PR: https://github.com/grafana/oncall/pull/82/files may break some links in our product or ".md" files. Would be nice to check all of them.

There is a proposal to make Prometeus Exporter for application monitoring. Monitoring such a critical service is also important if it is deployed locally.

To recreate:

• Select multiple alert groups
• Press Resolve
• To the right it will say "Results out of date"
• Press Refresh

The selected alert groups should no longer be visible but they are and their status is unchanged.

Looks like nice OSS monitoring tool, do we need native support for alerts from it?

https://github.com/fzaninotto/uptime

Btw native integrations are stored here: https://github.com/grafana/oncall/tree/dev/engine/config_integrations

We have template rendering preview in the integration settings. For now all templates are rendered one under another without titles or annotations. It would be nice to add some title for each template and distinguish them somehow.

As oncall engineer i would like to take some actions from chat ops like interface to resolve common issues or launch predefined set of actions to debug environment.

Opening this issue to collect some feedback from the community.

Does somebody need this?

Also check this one: Make integration and notifications via mattermost. #96

While sending phone notifications we are providing status_callback which is builded from settings.BASE_URL. In hobby deployment BASE_URL=localhost, so Twilio responds with:
The StatusCallback URL http://localhost:8000/twilioapp/sms_status_events/ is not a valid URL.

It's needed to check if helm has this issue too @iskhakov

When running the following command from README.md: docker-compose --env-file .env_hobby -f docker-compose.yml up --build -d, uWSGI fails to start on my M1 Mac.

uWSGI logs:

[uWSGI] getting INI configuration from uwsgi.ini
[log-encoder] registered format ${strftime:%Y-%m-%d %H:%M:%S} ${msgnl}
2022-06-15 08:06:33 *** Starting uWSGI 2.0.20 (64bit) on [Wed Jun 15 08:06:33 2022] ***
2022-06-15 08:06:33 compiled with version: 11.2.1 20220219 on 14 June 2022 18:16:43
2022-06-15 08:06:33 os: Linux-5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022
2022-06-15 08:06:33 nodename: cfcc13d1c266
2022-06-15 08:06:33 machine: x86_64
2022-06-15 08:06:33 clock source: unix
2022-06-15 08:06:33 pcre jit disabled
2022-06-15 08:06:33 detected number of CPU cores: 4
2022-06-15 08:06:33 current working directory: /etc/app
2022-06-15 08:06:33 writing pidfile to /tmp/project-master.pid
2022-06-15 08:06:33 detected binary path: /usr/local/bin/uwsgi
2022-06-15 08:06:33 uWSGI running as root, you can use --uid/--gid/--chroot options
2022-06-15 08:06:33 setgid() to 2000
2022-06-15 08:06:33 setuid() to 1000
2022-06-15 08:06:33 chdir() to /etc/app
2022-06-15 08:06:33 your memory page size is 4096 bytes
2022-06-15 08:06:33 detected max file descriptor number: 1048576
2022-06-15 08:06:33 lock engine: pthread robust mutexes
2022-06-15 08:06:33 unable to set PTHREAD_PRIO_INHERIT

When building the grafana/oncall image manually on my machine and using it in the docker-compose file, all works as expected. My proposal is to add a docker image for arm64 to DockerHub, so the instruction works for Apple Silicon/arm64 users.

Now it's slack-wide, it makes sense to make it team-specific.

We Using HA grafana with postgresql endpoint and OIDC auth.

But some times getting smth like that

[grafana-oncall-engine-7d8dddb54c-6fsks oncall] 2022-06-20 06:32:12 source=engine:app google_trace_id=none logger=root inbound latency=0.010145 status=202 method=POST path=/api/internal/v1/plugin/sync content-length=0 slow=0 integration_type=N/A integration_token=N/A
[grafana-oncall-engine-7d8dddb54c-6fsks oncall] 2022-06-20 06:32:12 source=engine:uwsgi status=202 method=POST path=/api/internal/v1/plugin/sync latency=0.010862 google_trace_id=- protocol=HTTP/1.1 resp_size=277 req_body_size=0
[grafana-1 grafana] logger=context traceID=6af451b8f211da18 t=2022-06-20T06:32:12.879886186Z level=error msg="invalid API key" error="invalid API key" traceID=6af451b8f211da18
[grafana-1 grafana] logger=context traceID=6af451b8f211da18 userId=0 orgId=0 uname= t=2022-06-20T06:32:12.879972941Z level=info msg="Request Completed" method=GET path=/api/org/users status=401 remote_addr=127.0.0.6 time_ms=0 duration=129.997µs size=59 referer= traceID=6af451b8f211da18

Seems like we need add "sticky sessions" but didnt full understand - why?

The timeline of a recent incident shows the times in GMT+2. I would like to switch it to UTC.

Hi,
I have encountered an issue when trying to install the helm chart on my local kubernetes. I get the following message:

Error: INSTALLATION FAILED: template: oncall/templates/engine/job-migrate.yaml:48:14: executing "oncall/templates/engine/job-migrate.yaml" at <include "snippet.redis.env" .>: error calling include: templa te: oncall/templates/_env.tpl:157:12: executing "snippet.redis.env" at <include "snippet.redis.host" .>: error calling include: template: oncall/templates/_env.tpl:140:46: executing "snippet.redis.host" a t <.Values.externalRedis.host>: nil pointer evaluating interface {}.host

Regards,
Patrick

On the Grafana OnCall->Settings page when you create a new API token it would be nice if it also displayed an example curl command to make use of the token for testing instead of needing to lookup in the docs. This would match the behavior that Grafana has when you create an API key in its settings page.

Hello!

Using docker-compose https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml I start Oncall with this command docker-compose --env-file .env_hobby up -d and then configure slack integration. Integration works, but when user want to connect slack account he gets this error:

Something went wrong when authorizing this app.
Error details
Invalid client_id parameter

Do I need to do any additional configuration?
To get public url for slack I use localtunnel.

It would be great to automatically connect on-premise Grafana instance during upgrade (via helm etc) to the existing OnCall instance via custom env variables, or config in grafana.ini and config in oncall helm chart.

It'll be nice have easy access function to pause calls or autoaknowledge incidents.
In time when you have big crash you have a lot alerts. Its hard to aknowledge and to make work in same time.

Dear friends,

After I did as like as the docs, I can not start up service. Could you please let me know what steps I'm missing? The error log from engine. I don't use built-in Grafana

[uWSGI] getting INI configuration from uwsgi.ini
[log-encoder] registered format ${strftime:%Y-%m-%d %H:%M:%S} ${msgnl}
2022-06-15 05:32:22 *** Starting uWSGI 2.0.20 (64bit) on [Wed Jun 15 05:32:22 2022] ***
2022-06-15 05:32:22 compiled with version: 11.2.1 20220219 on 14 June 2022 18:16:43
2022-06-15 05:32:22 os: Linux-3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022
2022-06-15 05:32:22 nodename: fe317d18eb64
2022-06-15 05:32:22 machine: x86_64
2022-06-15 05:32:22 clock source: unix
2022-06-15 05:32:22 pcre jit disabled
2022-06-15 05:32:22 detected number of CPU cores: 2
2022-06-15 05:32:22 current working directory: /etc/app
2022-06-15 05:32:22 writing pidfile to /tmp/project-master.pid
2022-06-15 05:32:22 detected binary path: /usr/local/bin/uwsgi
2022-06-15 05:32:22 uWSGI running as root, you can use --uid/--gid/--chroot options
2022-06-15 05:32:22 setgid() to 2000
2022-06-15 05:32:22 setuid() to 1000
2022-06-15 05:32:22 chdir() to /etc/app
2022-06-15 05:32:22 your memory page size is 4096 bytes
2022-06-15 05:32:22 detected max file descriptor number: 1048576
2022-06-15 05:32:22 lock engine: pthread robust mutexes
2022-06-15 05:32:22 thunder lock: disabled (you can enable it with --thunder-lock)
2022-06-15 05:32:22 Listen queue size is greater than the system max net.core.somaxconn (128).

Also I increased net.core.somaxconn to 512

My system CentOS 7 and docker version:

[root@k1 grafana-oncall]# docker --version
Docker version 20.10.17, build 100c701

[root@k1 grafana-oncall]# docker-compose --version
Docker Compose version v2.6.0

Thanks so much!

Is it possible to filter alert groups by title? I could not find that option

We have a Grafana instance (v8.5.0) running on an EC2 instance and is accessed through an internal application load balancer, and OIDC as the authentication.

We have set up OnCall in our EKS cluster and that is accessible through an internet-facing ALB. When attempting to curl from the Grafana instance, I get a response and I can see the associated message in the oncall pod logs. However, when attempting to connect the plugin (v1.0.3), it throws the following error:

Tried connecting to OnCall: https://<snip>
Error: Request failed with status code 404, retry or check settings & re-initialize.

The weird thing is, there are no associated messages in the pod logs for this alleged connection attempt. I thought it might be something to do with the ALB somehow, so I enabled load balancer logs and it doesn't even show any of these alleged connection attempts, which makes me think it isn't even hitting the load balancer for some reason. This is the only message shown in the Grafana logs:

logger=context traceID=00000000000000000000000000000000 userId=2 orgId=1 uname=<snip> t=2022-06-28T13:48:43.56+0000 lvl=info msg="Request Completed" method=POST path=/api/plugin-proxy/grafana-oncall-app/api/internal/v1/plugin/sync status=404 remote_addr=<snip> time_ms=5 duration=5.928139ms size=24 referer="https://<snip>/plugins/grafana-oncall-app?page=configuration" traceID=00000000000000000000000000000000

Not really sure what's going on here. Again, a curl from the instance works, so I know for a fact that from a network perspective it should work, so it just seems like the plugin isn't working for some reason.

Any ideas on how to resolve this? Please let me know if there is more debugging output to help resolve this.

I'm trying to connect Telegram but I receive this error:

Telegram, because the incident is not posted to Telegram (reason: Telegram is disabled for the route)

The user was successfully connect to the bot, there something that I need to configure?

Thanks

Lots of small p5 alerts can wait till next working hours.
As schedule owner i would like to setup some kind of working hours or time to debug and close low priority alerts. something like
notify mon: 9-18, tue: 9-18... fri: 9-18, sat, sun do not notify
and two option notify if still firing, notify anyway.

of cause low priority alerts lead to alert fatigue and must me eliminated but still popular case

Times for shift notification are being rendered as ms instead of normal format.
Screenshot is from Slack app, Android 10 on Samsung Galaxy S10e

I have recently moved to a basement office, which has zero cell signal, so when I am on call I need to work from upstairs. No big deal. The obvious solution here is to connect Slack on my phone to the account for work, and don't complain about all of the notifications I need to look at.

However, I'm wondering if there could be a future feature that could enable a user such as myself, to receive notifications from an internet location, that is not a work-related account. Work-life balance and such. What do you think of this idea?

https://github.com/mattermost/mattermost-server

Thanks!

Thanks!

Can we use Twilio API Credentials instead of Auth Tokens? This is an issue for us as there is only one set of Auth Tokens per Twilio account.

Hi, just wanna ask if there is an installer for Grafana OnCall for windows like in .exe?
Like in grafana, we can download the .exe and just one click install to our OS.

Thank you.

Regards,
Henjoe

Hey, first off I'm really excited by this tool especially given the potential for tight integration with the rest of Grafana!

However I'm unfortunately having some issues with the simplest possible set-up, running locally, that would prevent us from adopting this. Apologies if this is the wrong place to ask for help.

Given a schedule like the following:

And the following users in Grafana:

I would expect to see these users assigned to a schedule. However there seems to be something wrong with the association between event name and Grafana user:

I am running the setup from this docker-compose file, and only set it up today so the latest version of the docker image.

Thanks in advance for your help and feel free to direct me somewhere else if GH isn't the place.

Add new doc to support the newly released MS Teams chatops integration with Grafana OnCall