The most recent release of the Sensu Catalog (3abe43ca6bf537910fa4a53beea49b5b8beaf5db8821634d0e9a8ebad1a62e63 ) appears to have a bug in the GET /catalog.json endpoint where namespaces (e.g. system) with multiple valid integrations are only listing one integration.

The generator is correctly identifying the integrations (see system/disk-monitoring, system/host-monitoring, and system/network-interface-monitoring):

11:10PM WRN Skipping integration version reason="unmatched git tag" tag=20220228.0.0 
11:10PM INF Found integration version name=ansible-tower-remediation namespace=ansible tag=ansible/ansible-tower-remediation/20220223.0.0 version=20220223.0.0 
11:10PM INF Found integration version name=cassandra-monitoring namespace=cassandra tag=cassandra/cassandra-monitoring/20220217.0.0 version=20220217.0.0
11:10PM INF Found integration version name=helloworld namespace=example tag=example/helloworld/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=kubernetes-monitoring namespace=kubernetes tag=kubernetes/kubernetes-monitoring/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=mattermost-alerts namespace=mattermost tag=mattermost/mattermost-alerts/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=nginx-monitoring namespace=nginx tag=nginx/nginx-monitoring/20220217.0.0 version=20220217.0.0
11:10PM INF Found integration version name=pagerduty-incidents namespace=pagerduty tag=pagerduty/pagerduty-incidents/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=slack-alerts namespace=slack tag=slack/slack-alerts/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=sumologic-analytics namespace=sumologic tag=sumologic/sumologic-analytics/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=disk-monitoring namespace=system tag=system/disk-monitoring/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=host-monitoring namespace=system tag=system/host-monitoring/20220223.0.0 version=20220223.0.0
11:10PM INF Found integration version name=network-interface-monitoring namespace=system tag=system/network-interface-monitoring/20220217.0.0 version=20220217.0.0

The resulting GET /catalog.json endpoint only shows the system/disk-monitoring integration:

$ curl -s https://catalog.sensu.io/3abe43ca6bf537910fa4a53beea49b5b8beaf5db8821634d0e9a8ebad1a62e63/v1/catalog.json | jq .namespaced_integrations.system
[
  {
    "metadata": {
      "name": "disk-monitoring",
      "namespace": "system"
    },
    "display_name": "Disk monitoring",
    "class": "supported",
    "contributors": [
      "@sensu",
      "@calebhailey",
      "@jspaleta",
      "@thoward",
      "@nikkixdev"
    ],
    "provider": "monitoring",
    "short_description": "Cross-platform disk monitoring for Linux, Windows, and macOS.",
    "supported_platforms": [
      "darwin",
      "linux",
      "windows"
    ],
    "tags": [
      "system",
      "host",
      "os",
      "windows",
      "linux",
      "macos",
      "disk",
      "volumes"
    ],
    "version": "20220223.0.0"
  }
]

But the other two integrations are available:

• https://catalog.sensu.io/3abe43ca6bf537910fa4a53beea49b5b8beaf5db8821634d0e9a8ebad1a62e63/v1/system/host-monitoring.json
• https://catalog.sensu.io/3abe43ca6bf537910fa4a53beea49b5b8beaf5db8821634d0e9a8ebad1a62e63/v1/system/network-interface-monitoring.json

See: #10

The problem

Right now catalog generate supported generating into a randomly named temp dir and then uses github action specific stdout text to communicate the randomly named directory and save it as release-dir for other github actions to use.

But this makes it hard to support private catalog use cases that use other CI/CD based workflows other than github actions.

Straw proposal

add a new argument to generate to specify the temp dir subdirectory name instead of using a random name. Default to using random if not specified.

This would allow other CI/CD workflows to explictly specify the full release directory path as an input to the api generate command instead of having to discover what it is via stdout scraping.V

The GET /<release_sha256>/v1/<integration_namespace>/<integration_name>.json should return the same contents as the GET /<release_sha256>/v1/<integration_namespace>/<integration_name>/<integration_version>.json endpoint for the "latest" version (where "latest" = the first item in a list of versions sorted in descending order).

Should we also add an explicit version field to the GET /<release_sha256>/v1/<integration_namespace>/<integration_name>.json endpoint, for the avoidance of doubt?

Current behavior:

At first I thought this might be a web UI issue but then @jamesdphillips noticed that the data is missing in the API.

Integration: https://github.com/sensu/catalog/blob/main/integrations/slack/slack-alerts/sensu-integration.yaml#L20-L29

Example build:

curl -s https://catalog.sensu.io/3ce33e1e326cd8b8bb8813286ae3b7918031f47f572850f8aba5ec458322dc74/v1/mattermost/mattermost-alerts.json | jq .prompts[1]

Output:

{
  "type": "markdown",
  "title": "",
  "name": "",
  "input": null
}

Originally posted as a web-ui issue: https://github.com/sensu/sensu-enterprise-go/issues/2115

It appears this is happening in the API.

Would it be possible to preserve the field order of the source content? Is that a thing?

Consider using file type inference instead of checking file extensions. https://github.com/h2non/filetype looks like it may be a good fit for implementing this.

Add an alias that points to the latest version of an integration, avoiding the need to first lookup, parse, and deduce the latest version before viewing its details.

GET /<release_sha256>/v1/<namespace>/<name>/latest/...

In internal ref https://secure.helpscout.net/conversation/2135922091/30754/, it was noted that the api server generates multiple versions of integrations:

$ sudo -u www-data /usr/local/bin/catalog-api catalog server --repo-dir /opt/catalog --port 8084 --log-level trace --watch true
5:18PM INF Found integration version name=check-process namespace=system source=git tag=system/check-process/20230125.0.0 version=20230125.0.0
5:18PM INF Found integration version name=check-process namespace=system source=path version=99991231.0.0
5:18PM INF Found integration version name=check-process namespace=system source=git tag=system/check-process/20230125.0.0 version=20230125.0.0
5:18PM INF Found integration version name=check-process namespace=system source=path version=99991231.0.0
5:18PM INF API generated path=/tmp/4010909176
5:18PM INF API server started address=:8084

The server should only be generating a singular version.

Add support for providing links to support/community/contribute/etc...

support_urls:
  - url: https://sensu.io/contact
    name: Support
  - url: https://discourse.sensu.io/c/integrations/
    name: Forum
  - url: mailto:[email protected]
    name: Contact
  - url: https://github.com/sensu/catalog
    name: Contribute

The current implementation of the /<shasum>/v1/catalog.json endpoint provides a list of integration names only:

{
  "namespaced_integrations": {
    "nginx": [
      "nginx-monitoring"
    ],
    "system": [
      "host-monitoring"
    ]
  }
}

Without integration metadata additional HTTP requests would be needed to collect the needed information for the in-app catalog browser list view.

In one of the design discussions we had proposed that this endpoint would provide enough metadata that we would have all the information we needed for the list view; specifically including the following fields: namespace, name, class, provider, supported_platforms, short_description, and tags. One additional logo_url metadata field may be added for convenience, but this can also be fetched in a deterministic manner.

Example:

[
  {
    "namespace": "sensu",
    "name": "nginx-healthcheck",
    "logo_url": "https://catalog.sensu.io/<shasum>/api/v1/sensu/nginx-healthcheck/logo.png",
    "class": "supported",
    "provider": "agent/check",
    "supported_platforms": [
      "darwin",
      "linux",
      "windows"
    ],
    "short_description": "NGINX monitoring",
    "tags": [
      "http",
      "nginx",
      "web"
    ]
  },
  {
    "namespace": "sensu",
    "name": "nginx-metrics",
    "logo_url": "https://catalog.sensu.io/<shasum>/api/v1/sensu/nginx-metrics/logo.png",
    "class": "supported",
    "provider": "agent/check",
    "supported_platforms": [
      "darwin",
      "linux",
      "windows"
    ],
    "short_description": "NGINX metrics",
    "tags": [
      "http",
      "nginx",
      "web"
    ]
  }
]

The GET /<release_sha256>/v1/<namespace>/<name>/<version>/sensu-resources.json endpoint should return an array of "wrapped resources".

Example:

[
  {
    "api_version": "core/v2",
    "type": "CheckConfig",
    "metadata": {},
    "spec": {}
  },
  {
    "api_version": "core/v2",
    "type": "Asset",
    "metadata": {},
    "spec": {}
  }
]

NOTE: this endpoint is currently only returning a single wrapped resource, which may be a bug.

Compare:

• https://github.com/sensu/catalog/blob/nginx/nginx-monitoring/20220125.0.0/integrations/nginx/nginx-monitoring/sensu-resources.yaml
• https://catalog.sensu.io/ed26e72d01db7c5030c5eb97a024c582a013a6bc4dbf5499d1561ac7f6edddb9/v1/nginx/nginx-monitoring/20220125.0.0/sensu-resources.json

The web interface already has the ability to display additional blocks that are displayed after the integration has been successfully applied. I think these can be very handy when you want to convey some additional steps the user can/should do; as an example when you install a package with apt or brew they'll very often provide an additional document suggesting next steps, (eg. initializing the database or enabling a service).

Example:

---
api_version: catalog/v1
type: Integration
metadata:
  namespace: kubernetes
  name: kubernetes-monitoring
spec:
  class: supported
  provider: monitoring
  display_name: Kubernetes monitoring
  short_description: Kubernetes API liveness health check.
  supported_platforms:
    - darwin
    # ...
  tags:
    - kubernetes
    # ...
  contributors:
    - "@sensu"
  prompts:
    # ...
  resource_patches:
    # ...
  post_install:
    - type: section
      title: Next Steps
    - type: markdown
      body: |
1. Add the "nginx" subscription to agents that should run this check.

   **NOTE:** set the optional `check_nginx_status_url` agent annotation to override the default URL (`http://localhost:80/nginx_status`).

1. Ensure the [NGINX `stub_status` module][nginx_stub_status] is enabled on the NGINX instances(s) to be monitored.

   **NOTE:** the following example NGINX configuration snippet shows how to enable the `stub_status` module:

   \```
   # NGINX Status Module
   location /nginx_status {
       stub_status;
       allow 0.0.0.0;
   }
   \```

Only validation failures will be outputted by default and the exit status of the command represents sucess/failure. Let's add logging around validation successes at the info log level which will be hidden by default.

Implementation Overview

1. A CI job which runs catalog-api is triggered when a tag matching integrations/<integration_namespace>/<integration_name>/<semver_version> is created. Example regex:

   ^integrations/(?P<IntegrationNamespace>[a-z0-9_-]+)/(?P<IntegrationName>[a-z0-9_-]+)/(?P<Major>0|[1-9]\d*)\.(?P<Minor>0|[1-9]\d*)\.(?P<Patch>0|[1-9]\d*)(?:-(?P<Prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P<BuildMetadata>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$

2. catalog-api finds all git tags matching the regex above and creates a mapping of integration namespace & names to their available versions. Example of the data stored for each version:

   type SemverVersion struct {
   	Major         int
   	Minor         int
   	Patch         int
   	Prerelease    string
   	BuildMetadata string
   	TagName string
   }

3. Iterate through the mapping of integrations to versions:

   1. Attempt to checkout the tag name for each version.

   2. Retrieve the required integration files (sensu-integration|sensu-resources[.yml|.yaml] & logo.png).

   3. Unmarshal & validate sensu-integration.yaml.

   4. Unmarshal & validate sensu-resources.yaml.

   5. If any errors have occurred, log them and continue iterating.

   6. If successful, create a directory to store the required files. Example:

      mkdir -p generated/api/v1/integration-namespace/integration-name/version

   7. Generate a sensu-integration.json file containing a JSON-encoded copy of the original integration YAML file and store it in the directory.

   8. Generate a sensu-resources.json file containing a JSON-encoded copy of the original resources YAML file and store it in the directory.

   9. Copy logo.png to the directory.

   10. If any errors occurred, log them.

   11. If successful, mark the version as successfully handled.

4. After iteration is complete, generate a versions.json file containing a JSON array of all successfully handled versions for the integration. Store the file at generated/api/v1/integration-namespace/integration-name/versions.json.

5. Determine the latest version for each integration and copy its resources to the root of the catalog. For example:

   generated/api/v1/integrations/integration-namespace/integration-name/sensu-integration.json
   generated/api/v1/integrations/integration-namespace/integration-name/sensu-resources.json
   generated/api/v1/integrations/integration-namespace/integration-name/logo.png
   

6. Generate a JSON file containing a hash of all integrations belong to a namespace with the minimum amount of fields (e.g. integration name, namespace, etc.) and store it at generated/api/v1/integration-namespace/integrations.json.

7. Generate a JSON file containing a hash of all integrations with the minimum amount of fields (e.g. integration name, namespace, etc.) and store it at generated/api/v1/catalog.json.

8. Generate a JSON file containing version information (e.g. shasum of the latest deployment, last updated timestamp for starters) and store it at generated/version.json.

9. If any errors occurred at any point, stop execution now to prevent the deployment.

10. Calculate the SHA of the generated directory & deploy the contents of generated/api to a bucket on S3 using a directory with the SHA as the name. Upload generated/version.json to the root of the S3 bucket (it's important that this file is uploaded last).

11. The web UI will use the shasum in version.json to determine the URL of the API that it will use. Example:

GET https://cdn.catalogs.sensu.io/version.json would return

{
  "sha256": "b6ca19bee2ea529f39a61c433701f4d9eff9a77dc03f7889ad77347f8e95c32e",
  "last_updated": 1635944238
}

The URL for the latest API deployment can now be constructed using the sha256 value: https://cdn.catalogs.sensu.io/b6ca19bee2ea529f39a61c433701f4d9eff9a77dc03f7889ad77347f8e95c32e/api/v1/catalog.json

Other Notes

• The S3 bucket should have versioning enabled
  • Enables quick reverting to a previous deployment by rolling back version.json
• Objects (files) uploaded to S3 will need to have several properties set
  • Content-Type should be set to application/json for all JSON files
  • Content-Type should be set appropriately for logo images
  • Cache-Control should be set to a long period of time for all objects except version.json
  • CORS header(s) may need to be set

We should make the Catalog API generator available as a GitHub Action to make it easier for Sensu users to maintain custom Catalogs.

Given the amount of folks asking to run their own catalog, it would be good for us to package this up. So a deb/rpm/container image would be the ideal way to go IMO.

Presently, the generate command by default stores the output on MacOS in location like /var/folders/8f/kwmnl84n75qfywy5cfk07x340000gp/T/372379783/release. Given we instruct users to take this build output and move it to a static web server, it seems unhelpful to be moving it to an atypical directory like this, particularly outside of the user's home directory. By default it might be more helpful to build in the current working directory.

Add basic smoke tests to verify the functionality of the catalog-api catalog validate command.

There are several handlers that require sensu enterprise license to function, need an enterprise class in the catalog that matches the enterprise class definition in bonsai.

SVGs are obviously quite handy due to their scalability and flexibility and as such it would be quite nice to have support for them as both the logo of an integration and within READMEs. The downside is that they're susceptible XSS and other unwanted behaviour and as such we'll want to sanitize them before we serve them.

• DOMPurify is a great tool in the Javascript space but might not be feasible for this project
• microcosm-cc/bluemonday seems like the most popular tool in the Go world, but it doesn't seem to support SVGs. The library does seem extensible enough where I think we could add support though.

Make it possible to build the API content from the tip of main local filesystem.

Inspiration: https://goreleaser.com/customization/snapshots/

Add basic smoke tests to verify the functionality of the catalog-api catalog generate command.

For context, see internal ref: https://secure.helpscout.net/conversation/2152482078/30839?folderId=5845954.

The main thrust of the issue is this:

Building integrations for our own on-prem Catalog server, I've found that an empty value prevents resource patching. Let me give you an example.

I've added questions which supply parameters to a check command. Many of these parameters are optional, and I'd like to supply empty values ("") to the check command if the question is left blank.

They've provided this example with details:

I have a certificate expiration check, and I'd like to give the user the option to skip CA validation and just look at the expiration dates.

- type: question
name: extra
required: false
input:
type: string
title: Any extra parameters?
description: >-
For example, add "-k" to ignore certificate validation and just check the expiration
default: ""

It would have been cool to make this a boolean instead of a string and automagically add -k to the check command when the user opts for true, but that's a whole other ticket. Instead, I made my question a text string where the user can type in "-k".

Under resource_patches: I have:

- path: /spec/command
op: replace
value: >-
check-cert
-u [[ location ]]
-w [[ warning ]] -c [[ critical ]]
[[ extra ]]

If extra is left blank, the check command is unchanged from sensu-resources.yaml, ignoring values from location and warning and critical. I can kind of work around this one by setting default: " " for extra, as the extra space is harmless to check-cert.

In an internal conversation, the team suggested that instead of doing --warning "[[ warning ]]", to do [[ warning ]] and have it include the warning flag with a default.

However, they provided the following feedback:

Technically, yes, that should work, and I think I'll take this approach for now. However, I believe it presents a poor user experience, since one must carefully position the insertion cursor, instead of just, you know, clicking anywhere in the box and typing what you want. See the attached screeenshot for an example of how this would look.

See https://github.com/sensu/catalog#integration-api-specification for additional details.

Add support for adding integration dashboard configuration files (e.g. for Sumo Logic, Grafana, Kibana, and others) in JSON format to a dashboards/ subdirectory. This should work similarly to the existing support for embedded images in the img/ subdirectory (see #7).

No validation should be applied to contents of the dashboards/ subdirectory other than ensuring that all files are valid JSON.

Document the following API routes w/ final routes and example responses (the following are WIP/draft routes/contents):

• Root API URL (e.g. https://cdn.catalogs.sensu.io/api/v1

• GET /api/v1/status

  Returns the latest content version (used by the Sensu web app to determine the "latest" API subpath):

  {
    "sha256": "b6ca19bee2ea529f39a61c433701f4d9eff9a77dc03f7889ad77347f8e95c32e",
    "last_updated": 1635944238
  }

• GET /api/v1/integrations

  List all integrations.

  [
    {
      "org": "sensu",
      "name": "nginx-healthcheck",
      "logo_url": "https://catalog.sensu.io/api/v1/sensu/nginx-healthcheck/logo.png",
      "class": "supported",
      "provider": "agent/check",
      "supported_platforms": [
        "darwin",
        "linux",
        "windows"
      ],
      "short_description": "NGINX monitoring",
      "tags": [
        "http",
        "nginx",
        "web"
      ]
    },
    {
      "org": "sensu",
      "name": "nginx-metrics",
      "logo_url": "https://catalog.sensu.io/api/v1/sensu/nginx-metrics/logo.png",
      "class": "supported",
      "provider": "agent/check",
      "supported_platforms": [
        "darwin",
        "linux",
        "windows"
      ],
      "short_description": "NGINX metrics",
      "tags": [
        "http",
        "nginx",
        "web"
      ]
    }
  ]

• GET /api/v1/integrations/:org

  List all integrations for a given org (integration namespace).

• GET /api/v1/integrations/:org/:id

  Get integration details for a given integration (latest version).

  {
    "namespace": "sensu",
    "name": "nginx-healthcheck",
    "version": "0.1.0",
    "logo_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/logo.png",
    "resources_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/sensuresources",
    "class": "supported",
    "provider": "agent/check",
    "supported_platforms": [
      "darwin",
      "linux",
      "windows"
    ],
    "short_description": "NGINX monitoring",
    "readme_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/README",
    "tags": [
      "http",
      "nginx",
      "web"
    ],
    "contributors": [
      "@calebhailey",
      "@jspaleta",
      "@thoward",
      "@majormoses"
    ],
    "prompts": [],
    "resource_updates": []
  }

• GET /api/v1/integrations/:org/:id/versions

  List all available versions for a given integration.

  [
    "0.1.0",
    "0.2.0"
  ]

• GET /api/v1/integrations/:org/:id/logo.png

  URL to integration logo.

• GET /api/v1/integrations/:org/:id/:version/sensu-integration.json

  URL to specific version of an integration.

  {
    "namespace": "sensu",
    "name": "nginx-healthcheck",
    "version": "0.1.0",
    "logo_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/logo.png",
    "resources_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/sensuresources",
    "class": "supported",
    "provider": "agent/check",
    "supported_platforms": [
      "darwin",
      "linux",
      "windows"
    ],
    "short_description": "NGINX monitoring",
    "readme_url": "https://cdn.catalogs.sensu.io/api/v1/sensu/nginx-healthcheck/0.1.0/README",
    "tags": [
      "http",
      "nginx",
      "web"
    ],
    "contributors": [
      "@calebhailey",
      "@jspaleta",
      "@thoward",
      "@majormoses"
    ],
    "prompts": [],
    "resource_updates": []
  }

• GET /api/v1/integrations/:org/:id/:version/sensu-resources.json

  URL to JSON representation of valid Sensu resources (e.g. CheckConfig, Handler, Asset, etc)

The new prompts and resource_patches fields are being parsed by catalog-api, but they are only available in the GET /<release_sha256>/v1/catalog.json endpoint, and not in the individual GET /<release_sha256>/v1/<namespace>/<name>.json or GET /<release_sha256>/v1/<namespace>/<name>/<version>.json endpoints.

Changes:

• Remove the prompts and resource_patches fields from GET /<release_sha256>/v1/catalog.json
• Add the prompts and resource_patches fileds to GET /<release_sha256>/v1/<namespace>/<name>.json
• Add the prompts and resource_patches fileds to GET /<release_sha256>/v1/<namespace>/<name>/<version>.json

In order to allow various methods of handling multiple catalog version endpoints. E.g. production.json and staging.json at the root of the S3 bucket. We can add a command line flag to control the name of the filename.

We may also want to consider renaming the default of version.json to latest.json, production.json or stable.json.

For the sake of images referenced in a README, also upload the integration's img/ directory. Assuming the files are kosher, maybe we could even just upload the entire integration folder?

Provide support for configuring "related integrations", to be displayed on the integration detail view.

Possible implementation:

related_integrations:
  - system/network-interface-monitoring
  - system/disk-monitoring
  - system/filesystem-monitoring
  - system/process-monitoring

It would be lovely to have a docker image of this so that we can easily spin up demo environments.

Add a new integrations/v1.Integration "secret" prompt type

---
api_version: integrations/v1
type: Integration
metadata:
  name: example
  namespace: example
spec:
  ...: ...
  prompts:
    - type: secret
      name: sumologic_url
      title: Sumo Logic Source URL
      description: Select the Sumo Logic source URL (Sensu Secret)
      required: true
      filter: "example" # (optional filter on secret names)
      env_var_fallback: true

Notes:

• TBD if we should add a env_var_fallback: true field to allow users to add sensitive configuration as environment variables?

• variable is a hash with two keys: secret and env_var

  {
    "secret": "",
    "env_var": ""
  }

• integrations/v1.Integrations resource_patches can access secret input values via varname.secret or varname.env_var

We should have a way to validate the current state of integrations in a directory. This will give Catalog maintainers some insight into whether a PR contains errors. This command should ignore tags & use the current state of the repository (clean or dirty) and walk through the integrations directory structure to perform validation.