Hello,

I'm trying to override the default details template sent to pager duty:

I just want my Check output to be displayed in the details section, nothing else. I tried setting PAGERDUTY_DETAILS_TEMPLATE={{.Check.Output}} as an environment variable on my handler, but I'm still getting the full check JSON.

When I configure an annotation on the check like this:
"sensu.io/plugins/sensu-pagerduty-handler/config/details-template": "{{.Check.Output}}"

I get this error message:
error preparing check: unmatched token: template: :1:945: executing "" at <.Check.Output>: map has no entry for key "Check"

Do you have a working example of how to modify the Check JSON or a way to only send the check output to pager duty?

Thanks!

As a user, I'd like to be able to have a silence in Sensu also silence in PD. For example, if I receive an alert that I need to silence, it would be useful to not have to silence in 2 different places. Likewise, if I remove a silence in Sensu, that silence should also be removed in PD.

Expected Behavior

When configuring a check, I should be able to specify the priority of that check in a "pagerduty" scope and let pagerduty handle incident priority.

Current Behavior

This is not present

Possible Solution

Allow for configuration of priority in the default pagerduty integration configuration and/or the check/contact level.

This helps bring the golang handler up to feature parity to the existing community ruby plugin.

Strawman
expose dedupkey value as a settable via this priority list:
named envvar PAGERDUTY_DEDUP_KEY
cli arg: --dedup-key
named label: pagerduty.dedup_key

envvar acts as fallback of last resort

named label is used if set

Label Scopes
label scope is also prioritied:
agent scope , client scope, handler scope

I'm not sure what label priority order makes the most sense... we should make a decision and make it a best practice for all handlers.

Within the yaml file defining the pagerduty handler, I am unable to see any effect when setting the PAGERDUTY_SUMMARY_TEMPLATE variable under the env_vars section. It seems that when notification is received via pagerduty, the default value of "{{.Entity.Name}}/{{.Check.Name}} : {{.Check.Output}}" is always used instead.

cmdline arguments can be viewed from process table ( ex: ps and top commands)

Its better practice to load sensitive config via envvar or from a file on disk than from cmdline arguments.

Repository Updates

• Review any outstanding PRs/issues (are any easily addressable as part of this work?)
• Switch default branch from master to main (ref https://github.com/github/renaming)

Module Updates

• Update SDK, core, and all other modules with 'go get -u' (note: This may require some manual cleanup)
• Cleanup with 'go mod tidy'

README Updates

• Add note on templates and link to docs, when needed
• Ensure help output is current
• Caveat on using check annotations (ref https://github.com/sensu/sensu-pagerduty-handler/blob/master/README.md)
• Make sure annotations are explained as overrides
• Add table of environment variables available
• Make sure layout matches current standard
• Remove empty sections (e.g. Files, Additional Notes)

Golang version updates

• Update go.mod to use 1.14
• Update GitHub Actions (release and test) to use 1.14

GitHub Actions Updates

• All have lint, test, and release actions
• Add PR to lint and test actions on OSS handlers

Secrets update

• For any password/token arguments, ensure 'Secret: true' is set

Code updates

• Output supporting templates where it makes sense to (make sure to document in README)
• Replace import of types with api/core/v2 and change *types.Event references to *corev2.Event
• Run 'gofmt' on all source files

Output logging

• Output logging information (e.g. any incident or job # created)

Release

• Make sure Bonsai and GoReleaser configs are in sync and functioning Remove 32 bit macOS from Bonsai
• Add any changes to CHANGELOG
• Submit PR
• After merge, cut a new release
• In bonsai make sure the handler has the right tier (supported, enterprise)

Update curation(s)

• Update (or add new) curations to use new version/assets (add in template args if available)

process a user defined json key,value hash that defines a mapping from integer valued check status, into pagerduty severity strings.

keys are pagerduty severity strings "info","critical","warning","error"

values are arrays of status values

map example:
{"info":[130,10],"error":[4]}
would map status 10 and 130 to "info" and status 4 to "error"

Additionally, if there was a way to encode numeric range that would be fantastic and would lead to very compact representation of complicated status mappings.

Strawman

Implement as a commandline switch --status-map
and as entity,check scoped label ex: pagerduty.status_map

If you're integrating with Pagerduty from multiple namespaces, it may be prudent to use a "namespaced" Pagerduty dedup_key (see: https://support.pagerduty.com/docs/event-management#section-deduplicating-incidents).

type: Handler
api_version: core/v2
metadata:
  name: pagerduty
spec:
  command: >-
    sensu-pagerduty-handler
    --dedup-key-template "{{.Entity.Namespace}}-{{.Entity.Name}}-{{.Check.Name}}"
    --summary-template "{{.Entity.Name}}/{{.Check.Name}}: {{.Check.Output}}"
    --status-map "{\"info\":[0],\"warning\": [1],\"critical\": [2],\"error\": [3,127]}"
  filters:
  - is_incident
  - not_silenced
  runtime_assets:
  - sensu/sensu-pagerduty-handler:2.0.1
  secrets:
  - name: PAGERDUTY_TOKEN
    secret: pagerduty_token
  timeout: 0
  type: pipe

This is a single issue, but contains two parts:

1. A request to add support for the remaining PD-CEF fields of:
   1. Timestamp
   2. Class
   3. Group
2. A request to support processing the details template as json so we can send structured data through to Pagerduty. At the moment the options are:
   1. Send through the full event JSON. This is a lot of unnecessary data and the deep nesting of fields (when using labels and annotations) makes it hard to read some fields on Pagerduty
   2. Use a template to select required fields. Which unfortunately sends through the data as a string with no structure on the Pagerduty side which means you can't really use custom details in event rules

I'm hoping 1 is pretty uncontroversial and can be done. I would prefer it if these work similarly to summary and use a template.

I could accept 2 not happening if 1 goes ahead, but it really would be nice to have the option of templating structured data one-way or another.

In accordance with the contributing guidelines I'm opening an issue to discuss first, but I have made initial code changes on this branch. The changes can be seen on this compare view

I have tried to separate the commits to cover both requests:

• These two commits for adding the additional PD-CEF fields.
• This commit which adds a --send-details-as-json flag to process the template as json.

There is undoubtably a better way of doing 2 than what I've done - and I definitely welcome feedback and guidance. What I've done is pretty simplistic: When I realised that using the template returned a string I just used a json.Unmarshal to parse that string as a Struct/JSON. It "works", but maybe a real solution lies deeper in the templating stuff?

Hi all,

I'm having a strange behavior about this handler to pagerduty, when sending the event is not included hostname and the check name of the alert like old version of handler pagerduty written in ruby.

for example, the ruby version the 'Related to Incident' in pagerduty console is like:

services-01289.ops/elasticsearch-cluster-status : ESClusterStatus WARNING: Cluster is yellow
hostname/check_name: output

that is fine, but the pagerduty in GO I have something like this:

CheckHttp CRITICAL: Request error: end of file reached
this is the output command without the hostname and check name, I would like to add these fields for a better understanding of the alert.

How I can fix it?

Thanks!

.Entity.labels is available to query, but not values on keys within that, like .entity.labels.malcode in this customers' case below. Sensu docs show the available event attributes here but makes no mention of nested attributes.

internal ref - here

I was reviewing .bonsai.yml and I noticed a few issues.

• The filter for 64 bit Windows says "entity.system.os == 'darwin'"
• The filter for 32 bit Linux ARM says "entity.system.arch == 'armv7'", but it should says "entity.system.arch == 'arm'" instead.
• There is no entry for 32 bit Windows, when there could be.

In internal ref https://secure.helpscout.net/conversation/2205602449/31243?folderId=5845954, it was noted that other applications sending events to pagerduty have the ability to override the default events.pagerduty.com endpoint with a different URL. This is useful in cases where someone is using 3rd party software like the Pagerduty Alt Agent to send events to pagerduty. A good example of this is the Splunk Pagerduty Integration. Specifically, it looks like the integration URL there is the kind of feature that should be added.

The PD handler is serializing everything under details template but we want this to be available deserialized in pagerduty.

Json format (when the default payload is sent without -d) can translate to table nicely. We look to be serializing the string with details template producing an undesirable view for the customer.

Config

Event details

Introducing functionality like -T, --bodyTemplateFile from the email handler could be useful here?

Internal ref - https://secure.helpscout.net/conversation/2188359742/31097?folderId=4465878

Add a new --contact-routing flag

• Looks for and merges the entity, check, and/or event contacts annotation.

  The contacts annotation(s) should be formatted as a comma-separate list of contact names.

  annotations:
    contacts: "team_a,team_b"

  NOTE: Contact names may contain only alpha-numeric characters and underscore (_).

• The --contact-routing flag should cause the handler to log the merged list of contacts

  Example:

  Pagerduty contact routing is enabled (contacts: team_a, team_b)
  

• Execute manageIncident() function once per contact

  Lookup environment variable matching fmt.Sprintf("PAGERDUTY_TOKEN_%s", strings.ToUpper(contact)) (or similar).

  Notes on environment variable lookups:

  • TBD if --contact-routing could be used with the existing --team and --team-suffix flags for looking up contact environment variables.
  • TBD if environment variables should use a configurable or hardcoded prefix (e.g. PAGERDUTY_TOKEN_) or suffix (see above RE: reusing existing --team-suffix flag).
  • Perhaps a --contact-prefix flag would be best so it doesn't conflict with existing functionality?

  If the contact environment is found, perform the manageIncident() function.

  If the contact environment variable is not found, log an warning (e.g. WARNING: skipping contact: "team-a" (no environment variable found for "PAGERDUTY_TOKEN_TEAM_A")\n).

  If at least one matching contact environment variable is found and manageIncident() returns no errors, the handler should exit with status 0.

Example pipeline configuration:

---
api_version: core/v2
type: Pipeline
metadata:
  name: pagerduty
spec:
  workflows:
    - name: pagerduty
      filters:
        - api_version: core/v2
          type: EventFilter
          name: is_incident
        - api_version: core/v2
          type: EventFilter
          name: not_silenced
      handler:
        api_version: core/v2
        type: Handler
        name: pagerduty
---
# Requires PAGERDUTY_TOKEN environment variable or secret
api_version: core/v2
type: Handler
metadata:
  name: pagerduty
spec:
  type: pipe
  command: >-
    sensu-pagerduty-handler
    --contact-routing
    --contact-prefix "PAGERDUTY_TOKEN_"
    --dedup-key-template "{{.Entity.Namespace}}-{{.Entity.Name}}-{{.Check.Name}}"
    --status-map "{\"info\":[0],\"warning\": [1],\"critical\": [2],\"error\": [3,127]}"
    --summary-template "[{{.Entity.Namespace}}] {{.Entity.Name}}/{{.Check.Name}}: {{.Check.State}}"
    --details-template ""{{ .Check.Name }} is {{ .Check.State }} on {{ .Entity.Name }} (namespace: {{ .Entity.Namespace }})\n\n{{ or (index .Annotations \"sensu.io/notifications/critical\") (index .Annotations \"sensu.io/notifications/warning\") .Check.Output }}""
  timeout: 10
  filters: []
  runtime_assets:
    - sensu/sensu-pagerduty-handler
  env_vars: []
  secrets:
    - name: PAGERDUTY_TOKEN_TEAM_A
      secret: pagerduty_team_a
    - name: PAGERDUTY_TOKEN_TEAM_B
      secret: pagerduty_team_B

Example agent config:

---
# example agent.yml
name: i-424242
labels: {}
annotations:
  contacts: "team_a,team_b"

Example check config:

---
api_version: core/v2
type: CheckConfig
metadata:
  name: example-check
  annotations:
    contacts: "team_c"
spec:
  ...: ...

According to PagerDuty API documentation, for payload.summary

The maximum permitted length of this property is 1024 characters.

Trying to send a payload with a longer summary results in the following error: Error: HTTP Status Code: 400, Message: {"status":"invalid event","message":"Event object is invalid","errors":["'summary' is too long (maximum is 1024 characters)"]}

This handler should truncate the summary at 1024 characters, rather than failing to send to PagerDuty.

I am using sensugo for monitoring. What following parameter should be.

Mu service name is devstack and team name is devcon.

• PAGERDUTY_TOKEN=76f82173acf040b8a36799bf78b4cbd1 ### Integration API key
  • PAGERDUTY_DEDUP_KEY=PAGERDUTY_DEDUP_KEY ### what is this?
  • PAGERDUTY_DEDUP_KEY_TEMPLATE={{.Entity.Name}}-{{.Check.Name}} . ### what is this?
  • PAGERDUTY_STATUS_MAP={"info":[130,10],"error":[4]} . ### what is this?

Hi,

SENSU_EVENT_LABEL is empty therefore the integration is not working, is sensu suppose to fill in or do we need to create it and if so what would be the value?

Inspired by the sensu-plugins/sensu-plugins-pagerduty contact routing implementation (see comments regarding pager_team check or client attribute).

Overview

Users need the ability to override the default Pagerduty API Token on a per-check and/or per-entity basis. This enhancement will enable users to add a single annotation to check or entity resources to configure the behavior of the Pagerduty handler.

Here's an example check with the sensu.io/plugins/sensu-pagerduty-handler/config/pager-team annotation:

---
type: CheckConfig 
api_version: core/v2 
metadata: 
  name: example-check 
  annotations: 
    sensu.io/plugins/sensu-pagerduty-handler/config/pager-team: team_1

Here's an example agent.yaml file with the sensu.io/plugins/sensu-pagerduty-handler/config/pager-team annotation set:

---
##
# agent configuration
##
# name: example-agent
# namespace: default
backend-url:
- wss://sensu-backend-0.sensu.sensu-system.svc.cluster.local:8081 
- wss://sensu-backend-1.sensu.sensu-system.svc.cluster.local:8081 
- wss://sensu-backend-2.sensu.sensu-system.svc.cluster.local:8081
subscriptions:
- linux
- docker
- kubernetes
labels:
  org: Sensu, Inc.
annotations: 
  sensu.io/plugins/sensu-pagerduty-handler/config/pager-team: team_1

Proposed changes

• Add a new --pager-team flag and corresponding annotation (i.e. sensu.io/plugins/sensu-pagerduty-handler/config/pager-team)
• Add a new --pager-team-suffix flag (default to _pagerduty_token)
• Use the string values of --pager-team and --pager-team-suffix to read the corresponding Pagerduty API Token value as an environment variable
• Users can set Pagerduty team tokens using the Handler env_vars field, or using Sensu's built-in secrets management
• Users can also manage Pagerduty team tokens outside of Sensu (e.g. by editing /etc/default/sensu-backend or via config management on the Sensu backend nodes), but we will recommend keeping the config in Sensu

The Bonsai asset definition for the Windows 386 build has amd64 for the arch.

spec:
  builds:
  - url: https://assets.bonsai.sensu.io/e6722787284bd20f68d0375ee927aa0ec5d2e57b/sensu-pagerduty-handler_1.3.2_windows_386.tar.gz
    sha512: 10eb353c624fa1853a5b6175366e6ae325315208b2289b1191629004737dbacbfc576f54b95a89560a5226e24aa0438458256b04ac1ffdc9876cf0afdf6441dd
    filters:
    - entity.system.os == 'windows'
    - entity.system.arch == 'amd64'

The details field is currently just the full event, make it a template.

Would be useful to be able to provide a custom summary string with token substitution.

What I propose is being able to set another environment variable like:
"PAGERDUTY_SUMMARY_FORMAT={ENTITY}/{CHECK} : {OUTPUT}"

We just received a notification from the PagerDuty partner team regarding upcoming changes to PagerDuty APIs that we need to review.

Re: Your integration with PagerDuty built with PagerDuty's APIs and webhooks

https://www.pagerduty.com/integrations/sensu/

Just wanted to email you to let you know the following changes will be happening in the near future.

1. CHANGES TO WEBHOOKS: We are happy to launch our v3 webhooks utilized to build a native integration. In addition, we wanted to let you know that we are deprecating our v1 webhooks and v2 webhooks at the end of November 2021 (v1 webhooks) and March, 2022 (v2 webhooks). Please review your current integration listed on the PagerDuty Directory and upgrade your integration to v3 webhooks before November, 2021.

   Resource Links for webhooks:
   https://support.pagerduty.com/docs/webhooks
   https://support.pagerduty.com/docs/webhooks#migrating-from-v1v2-generic-extensions-to-v3-webhook-subscriptions

   Integration Documentation: Please also review your integration documentation and make any changes needed.

2. INCIDENT AND ALERT ID CHANGES: We are also making changes to the Incident and Alert ID. The Incident and Alert ID are being increased in character length to 14 in place of the current 7 characters to support the increasing storage capacity of our systems starting in the month of October, 2021.
   We will first roll this out to our EU service region on October 5, 2021, followed by a rollout to the North American service region on October 12, 2021.

If you have a custom integration or your own code that makes use of PagerDuty’s API, you should review that code at this time to ensure that it will continue to function with the new identifiers. Specifically, you should confirm that your code does not rely on the existence of a P prefix for incident identifiers, nor make assumptions that they are of a certain length.

PagerDuty API has rate limits; see here and here. I would like to see this handler tackle the issue of hitting the rate limit (for example in case of a larger outage).

Maybe something like number of --retries parameter? Maybe even with configurable backoff, multiplier etc?

Hello,

According to go-pagerduty maintainer the eu datacenter endpoints are supported but i don't see any mentions of them on this handler.
PagerDuty/go-pagerduty#355

Could it be possible to include configuration also for eu endpoints?

Our partner page on pagerduty.com links to documentation for installing the old Sensu Enterprise "Classic" integration. We need to update this with instructions on how to install and configure the new Sensu Go integration.

See:

• https://www.pagerduty.com/integrations/sensu/
• https://www.pagerduty.com/docs/guides/sensu-integration-guide/ (these are the old docs)

Hello,
I'm attempting to send pagerduty events via an encrypted proxy. I am getting this error:

{"assets":null,"component":"pipelined","handler":"pagerduty","level":"info","msg":"event pipe handler executed","namespace":"default","output":"Error: Post https://events.pagerduty.com/v2/enqueue: proxyconnect tcp: x509: certificate signed by unknown authority\nUsage:\n  sensu-pagerduty-handler [flags]\n\nFlags:\n  -h, --help           help for sensu-pagerduty-handler\n  -t, --token string   The PagerDuty V2 API authentication token, use default from PAGERDUTY_TOKEN env var (default \"xxx\")\n\nerror: Post https://events.pagerduty.com/v2/enqueue: proxyconnect tcp: x509: certificate signed by unknown authority\n","status":1,"time":"2019-03-08T23:05:06Z"}

I believe the problem is that the CA which signed the cert for my proxy is not in the trust store for the sensu-pagerduty-handler. Is there a way to configure this? If not, please consider this report as a feature-request.

The customer would like the ability to define custom fields to be populated via tokenization (outside of using the -d flag)

For example:

check_output: "{{.Check.Output}}"
client_url: "https://sensugourl.com/c/~/n/{{.Entity.Namespace}}/events/{{.Entity.Name}}"
client: "{{.Entity.Name}}"

internal ref: https://secure.helpscout.net/conversation/2179913205/31023?folderId=4465878