canonical / grafana-k8s-operator Goto Github PK

This charmed operator automates the operational procedures of running Grafana, an open-source visualization toolkit, on Kubernetes.

Home Page: https://charmhub.io/grafana-k8s

License: Apache License 2.0

Python 100.00%

cos data-visualization grafana juju kubernetes observability hacktoberfest juju-charm

grafana-k8s-operator's Introduction

Grafana Operator for k8s

Charmed Grafana (grafana-k8s) is a charm for Grafana.

This charm imposes configurable resource limits on the workload, can be readily integrated with various data sources such as prometheus or loki, automatically generates topology dropdowns, and comes with built-in alert rules for self-monitoring.

It is an essential part of the COS Lite bundle.

Usage

The Grafana Operator may be deployed on a Kubernetes Juju model using the command line via:

juju deploy grafana-k8s

At install time, Grafana does not contain any data sources or dashboards, but Prometheus is commonly used, and is deployable with Juju. The Grafana Operator may also accept additional datasources over Juju relations with charms which support the grafana-datasource interface, such as Loki log visualization.

For example:

juju deploy prometheus-k8s
juju relate prometheus-k8s grafana-k8s

The Grafana Operator includes a Charm library which may be used by other Charms to easily provide datasources. Currently, Prometheus and Loki are tested, with datasource integration built into those charms, but any Grafana datasource which does not require the addition of a plugin should be supported. See the documentation for the charms.grafana_k8s.v0.grafana_source to learn more.

Web Interface

The Grafana dashboard may be accessed on port 3000 on the IP address of the Grafana unit. This unit and its IP address can be retrieved using the juju status command.

The default password is randomized at first install, and can be retrieved with:

juju run grafana-k8s/0 get-admin-password # juju >=3
###
JUJU_FEATURES=actions-v2 juju run grafana-k8s/0 get-admin-password # juju 2

View the dashboard in a browser:

juju status to check the IP of the running Grafana application
Navigate to http://IP_ADDRESS:3000
Log in with the username admin, and the password you got from the get-admin-password action.

To manually set the admin password, see the official docs.

Additionally, Grafana can be accessed via the Kubernetes service matching the Juju application name in the namespace matching the Juju model's name.

Integration with other charms/adding external dashboards

The grafana-k8s charm does not support directly relating to Reactive charms over the dashboards interface, and it does not support adding dashboards via an action similar to the Reactive Grafana Charm as a design goal. For scenarios where Reactive charms which provide dashboards should be integrated, the COS Proxy charm can be deployed in a Reactive model, and related to grafana-k8s to provide a migration path.

Dashboards which are not bundled as part of a charm can be added to grafana-k8s with the COS Config Charm, which can keep a git repository holding your infrastructure-as-code configuration. See the COS Config documentation for more information.

To Bundle Dashboards As Part of Your Charm

See the documentation for the charm.grafana_k8s.v0.grafana_dashboard library. Generally, this only requires adding a grafana-dashboard interface to your charm and putting the dashboard templates into a configurable path.

High Availability Grafana

This charm is written to support a high-availability Grafana cluster, but a database relation is required (MySQL or Postgresql).

If HA is not required, there is no need to add a database relation.

NOTE: HA should not be considered for production use.

Relations

grafana_datasource - An input for grafana-k8s datasources
grafana_dashboard - an input for LZMA compressed base64 encoded dashboard data

OCI Images

This charm is periodically updated to the latest version of the ubuntu/grafana image.

Contributing

See the Juju SDK docs for guidelines on configuring a development environment and best practices for authoring.

grafana-k8s-operator's People

Contributors

Stargazers

Watchers

grafana-k8s-operator's Issues

Upgrade grafana image to the last stable version of 8.5.x

Enhancement Proposal

Upgrade to 8.5 so we can add Prometheus Alertmanager datasource

Documentation mentions storage pool

https://github.com/canonical/grafana-operator/blob/master/README.md
currently says to create a storage pool:
juju create-storage-pool operator-storage kubernetes storage-class=microk8s-hostpath

however, it doesn't ever make use of that in the rest of the README.

Add the "All" option for Juju Unit in the Dashboard Filters

Enhancement Proposal

Some dashboards would likely want to be able to use juju topology down to the application level, but omit the unit selector, for instance to create lists of units and their statuses. Currently, that option is missing.

Grafana fails with hook failed: "grafana-dashboard-relation-created"

Bug Description

In testrun https://solutions.qa.canonical.com/v2/testruns/f12b5292-2b3e-4d63-a466-f5c2c4366f57/ the cos-stack fais to install with the juju status:

Model       Controller       Cloud/Region                Version  SLA          Timestamp
controller  foundations-k8s  kubernetes_cloud/us-east-1  2.9.35   unsupported  12:19:06Z

Model "controller" is empty.
Model  Controller       Cloud/Region                Version  SLA          Timestamp
cos    foundations-k8s  kubernetes_cloud/us-east-1  2.9.35   unsupported  12:19:06Z

App           Version  Status   Scale  Charm             Channel  Rev  Address                                                                 Exposed  Message
alertmanager  0.23.0   waiting      1  alertmanager-k8s  edge      33  10.152.183.222                                                          no       waiting for container
catalogue              active       1  catalogue-k8s     edge       3  10.152.183.119                                                          no       
grafana                waiting    0/1  grafana-k8s       edge      45  10.152.183.97                                                           no       installing agent
loki                   waiting      1  loki-k8s          edge      45  10.152.183.153                                                          no       installing agent
prometheus             waiting      1  prometheus-k8s    edge      75  10.152.183.67                                                           no       installing agent
traefik                waiting      1  traefik-k8s       edge      89  ae8a8e7eac33646d4ada3fe220bc3831-239371593.us-east-1.elb.amazonaws.com  no       installing agent

Unit             Workload  Agent  Address         Ports  Message
alertmanager/0*  active    idle   192.168.133.73         
catalogue/0*     active    idle   192.168.218.70         
grafana/0*       error     lost   192.168.133.71         hook failed: "grafana-dashboard-relation-created"
loki/0*          waiting   idle   192.168.218.72         Waiting for resource limit patch to apply
prometheus/0*    waiting   idle   192.168.218.71         Waiting for resource limit patch to apply
traefik/0*       error     idle   192.168.133.72         hook failed: "metrics-endpoint-relation-joined"

To Reproduce

We got to this state by deploying the cos-stack on charmed kubernetes on jammy.

Environment

The k8s is ck8s 1.24 on jammy, cos is latest/edge

Relevant log output

Nothing in the unit logs indicates any errors:


unit-grafana-0: 12:18:11 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-0: 12:18:11 WARNING cmd developer feature flags enabled: "actions-v2"
unit-grafana-0: 12:18:11 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-0: 12:18:11 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-0: 12:18:11 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-0: 12:18:11 INFO juju.api cannot resolve "a703653953140478c9ac07832662e677-1313776419.us-east-1.elb.amazonaws.com": lookup a703653953140478c9ac07832662e677-1313776419.us-east-1.elb.amazonaws.com: operation was canceled
unit-grafana-0: 12:18:11 INFO juju.api connection established to "wss://controller-service.controller-foundations-k8s.svc.cluster.local:17070/model/81fcf692-fbf2-46ef-8d2a-48bf98f3f492/api"
unit-grafana-0: 12:18:11 INFO juju.worker.apicaller [81fcf6] "unit-grafana-0" successfully connected to "controller-service.controller-foundations-k8s.svc.cluster.local:17070"
unit-grafana-0: 12:18:11 INFO juju.api cannot resolve "a703653953140478c9ac07832662e677-1313776419.us-east-1.elb.amazonaws.com": lookup a703653953140478c9ac07832662e677-1313776419.us-east-1.elb.amazonaws.com: operation was canceled
unit-grafana-0: 12:18:11 INFO juju.api connection established to "wss://controller-service.controller-foundations-k8s.svc.cluster.local:17070/model/81fcf692-fbf2-46ef-8d2a-48bf98f3f492/api"
unit-grafana-0: 12:18:11 INFO juju.worker.apicaller [81fcf6] "unit-grafana-0" successfully connected to "controller-service.controller-foundations-k8s.svc.cluster.local:17070"
unit-grafana-0: 12:18:11 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-0: 12:18:11 INFO juju.worker.logger logger worker started
unit-grafana-0: 12:18:11 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-0: 12:18:11 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-0: 12:18:11 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-0: 12:18:11 INFO juju.worker.leadership grafana/0 promoted to leadership of grafana
unit-grafana-0: 12:18:11 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-0
unit-grafana-0: 12:18:11 INFO juju.worker.uniter unit "grafana/0" started
unit-grafana-0: 12:18:11 INFO juju.worker.uniter resuming charm install
unit-grafana-0: 12:18:11 INFO juju.worker.uniter.charm downloading ch:amd64/focal/grafana-k8s-45 from API server
unit-grafana-0: 12:18:11 INFO juju.downloader downloading from ch:amd64/focal/grafana-k8s-45
unit-grafana-0: 12:18:12 INFO juju.downloader download complete ("ch:amd64/focal/grafana-k8s-45")
unit-grafana-0: 12:18:12 INFO juju.downloader download verified ("ch:amd64/focal/grafana-k8s-45")
unit-grafana-0: 12:18:27 INFO juju.worker.uniter hooks are retried true
unit-grafana-0: 12:18:27 INFO juju.worker.uniter found queued "install" hook
unit-grafana-0: 12:18:30 INFO unit.grafana/0.juju-log Running legacy hooks/install.
unit-grafana-0: 12:18:32 INFO unit.grafana/0.juju-log Successfully patched the Kubernetes service!
unit-grafana-0: 12:18:32 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:35 INFO juju.worker.uniter.operation ran "catalogue-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:36 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:37 INFO juju.worker.uniter.operation ran "ingress-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:39 INFO juju.worker.uniter.operation ran "grafana-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:40 INFO juju.worker.uniter.operation ran "grafana-source-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 12:18:41 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-0: 12:18:42 ERROR juju.worker.uniter.operation hook "grafana-dashboard-relation-created" (via hook dispatching script: dispatch) failed: signal: terminated
unit-grafana-0: 12:18:42 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-grafana-0: 12:18:42 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook

Additional context

Crashdumps and config can be found here:
https://oil-jenkins.canonical.com/artifacts/f12b5292-2b3e-4d63-a466-f5c2c4366f57/index.html

Liveness probe fails

The liveness probe fails with the charm container:

Containers:
  charm:
    Container ID:  containerd://6f23ccbc1c403570f0faf800fe692358f3ce488801db7db189b7126ce46ce50d
    Image:         jujusolutions/charm-base:ubuntu-20.04
    Image ID:      docker.io/jujusolutions/charm-base@sha256:e16ca5a76c033e6c48f4ae57c73d9c691c647a014925359bc435d588357d4d88
    Port:          <none>
    Host Port:     <none>
    Command:
      /charm/bin/containeragent
    Args:
      unit
      --data-dir
      /var/lib/juju
      --charm-modified-version
      0
      --append-env
      PATH=$PATH:/charm/bin
    State:          Running
      Started:      Mon, 02 Aug 2021 11:22:36 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 02 Aug 2021 11:16:11 +0200
      Finished:     Mon, 02 Aug 2021 11:17:30 +0200
    Ready:          False
    Restart Count:  33
    Liveness:       http-get http://:3856/liveness delay=30s timeout=1s period=10s #success=1 #failure=2
    Readiness:      http-get http://:3856/readiness delay=30s timeout=1s period=10s #success=1 #failure=2
    Startup:        http-get http://:3856/startup delay=30s timeout=1s period=10s #success=1 #failure=2
    Environment:

It might have to do with the fact that MicroK8s is running on my laptop and coming in and out of suspend, but then again, if MicroK8s is alive enough to churn pods for failed liveness checks, why isn't the charm container? :-) (Also, I have a Prometheus-K8s and an Alertmanager-K8s charms running in the same controller, and those have no liveness issues.)

Rename repo

Per the naming guidelines recently published, this repo should be renamed grafana-k8s-operator. Will need a ticket into IS.

The get-admin-password actions returns incorrect information during deployment

Bug Description

The get-admin-password action returns "Admin password has been changed by an administrator" during deployment. It also reports this message in the admin-password field, which seems a bit counter-intuitive and might make it less straightforward to use in scripts.

To Reproduce

juju deploy grafana-k8s --channel edge
juju run-action grafana-k8s/0 get-admin-password --wait

Environment

Using grafana-k8s from latest/edge.

Relevant log output

michele@boombox:~/git/prometheus-operator$ juju run-action grafana/0 get-admin-password --wait
unit-grafana-0:
  UnitId: grafana/0
  id: "4"
  results:
    admin-password: Admin password has been changed by an administrator
  status: completed
  timing:
    completed: 2022-03-15 08:56:52 +0000 UTC
    enqueued: 2022-03-15 08:56:50 +0000 UTC
    started: 2022-03-15 08:56:51 +0000 UTC

Additional context

No response

The grafana_dashboard library does not document the required modifications to metadata.yaml

Bug Description

The grafana_dashboard library does not document the required modifications to metadata.yaml. For reference, see the ingress_per_unit library.

To Reproduce

Navigate to https://charmhub.io/grafana-k8s/libraries/grafana_dashboard
Make your best impression of https://media.giphy.com/media/g01ZnwAUvutuK8GIQn/giphy.gif

Environment

CharmHub.io

Relevant log output

Live capture of Michele trying to integrate `grafana_dashboard` in a charm: https://media.giphy.com/media/g01ZnwAUvutuK8GIQn/giphy.gif

Additional context

No response

Charm goes into error state on removal

Bug Description

When I destroy the COS Lite model, grafana goes into error state:

App           Version  Status   Scale  Charm             Channel  Rev  Address         Exposed  Message
grafana       8.2.6    error        1  grafana-k8s       edge      45  10.152.183.227  no       hook failed: "grafana-dashboard-relation-broken"

Unit        Workload  Agent  Address      Ports  Message
grafana/0*  error     idle   10.1.13.151         hook failed: "grafana-dashboard-relation-broken" for alertmanager:grafana-dashboard

To Reproduce

Deploy the COS Lite bundle (without traefik, in the meanwhile).
juju destroy-model --destroy-storage.

Environment

NTA.

Relevant log output

Traceback (most recent call last):
  File "./src/charm.py", line 1153, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event)  # noqa
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1311, in _on_grafana_dashboard_relation_broken
    self._remove_all_dashboards_for_relation(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1456, in _remove_all_dashboards_for_relation
    if self._get_stored_dashboards(relation.id):
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1489, in _get_stored_dashboards
    return self.get_peer_data("dashboards").get(str(relation_id), {})
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1504, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 10:29:11.019 ERROR juju.worker.uniter.operation hook "grafana-source-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1
unit-grafana-0: 10:29:10.013 DEBUG unit.grafana/0.juju-log grafana-source:12: Operator Framework 1.5.3+1.g26626e4 up and running.
unit-grafana-0: 10:29:10.156 DEBUG unit.grafana/0.juju-log grafana-source:12: Re-emitting <GrafanaDashboardsChanged via GrafanaCharm/GrafanaDashboardConsumer[grafana-dashboard]/on/dashboards_changed[237]>.
unit-grafana-0: 10:29:10.169 DEBUG unit.grafana/0.juju-log grafana-source:12: Pebble API is not ready; ConnectionError: [Errno 2] No such file or directory
unit-grafana-0: 10:29:10.182 INFO unit.grafana/0.juju-log grafana-source:12: Initializing dashboard provisioning path
unit-grafana-0: 10:29:10.196 WARNING unit.grafana/0.juju-log grafana-source:12: Could not push default dashboard configuration. Pebble shutting down?
unit-grafana-0: 10:29:10.209 DEBUG unit.grafana/0.juju-log grafana-source:12: Pebble API is not ready; ConnectionError: [Errno 2] No such file or directory
unit-grafana-0: 10:29:10.221 DEBUG unit.grafana/0.juju-log grafana-source:12: Cannot connect to Pebble yet, deferring event
unit-grafana-0: 10:29:10.233 DEBUG unit.grafana/0.juju-log grafana-source:12: Deferring <GrafanaDashboardsChanged via GrafanaCharm/GrafanaDashboardConsumer[grafana-dashboard]/on/dashboards_changed[237]>.
unit-grafana-0: 10:29:10.260 DEBUG unit.grafana/0.juju-log grafana-source:12: Re-emitting <GrafanaDashboardsChanged via GrafanaCharm/GrafanaDashboardConsumer[grafana-dashboard]/on/dashboards_changed[241]>.
unit-grafana-0: 10:29:10.273 DEBUG unit.grafana/0.juju-log grafana-source:12: Pebble API is not ready; ConnectionError: [Errno 2] No such file or directory
unit-grafana-0: 10:29:10.285 INFO unit.grafana/0.juju-log grafana-source:12: Initializing dashboard provisioning path
unit-grafana-0: 10:29:10.299 WARNING unit.grafana/0.juju-log grafana-source:12: Could not push default dashboard configuration. Pebble shutting down?
unit-grafana-0: 10:29:10.311 DEBUG unit.grafana/0.juju-log grafana-source:12: Pebble API is not ready; ConnectionError: [Errno 2] No such file or directory
unit-grafana-0: 10:29:10.323 DEBUG unit.grafana/0.juju-log grafana-source:12: Cannot connect to Pebble yet, deferring event
unit-grafana-0: 10:29:10.335 DEBUG unit.grafana/0.juju-log grafana-source:12: Deferring <GrafanaDashboardsChanged via GrafanaCharm/GrafanaDashboardConsumer[grafana-dashboard]/on/dashboards_changed[241]>.
unit-grafana-0: 10:29:10.359 DEBUG unit.grafana/0.juju-log grafana-source:12: Re-emitting <GrafanaDashboardsChanged via GrafanaCharm/GrafanaDashboardConsumer[grafana-dashboard]/on/dashboards_changed[245]>.

Additional context

No response

Grafana Operator does not correctly add datasource if a second prometheus is added

To reproduce:

Deploy two prometheus sidecar charms and a single Grafana, then add relations between both prometheus sources and grafana.

Refresh the datasources dashboard to see if anything is listed

Provide "Hello world" example for Dashboard

Unit-based grouping
Model-based grouping
Application-based grouping
Simple UP widget

grafana_dashboard lib raises when processing dashboards exported by Grafana 9

Bug Description

Prior to grafana 9, the datasource key was a plain string, e.g. "datasource": "${prometheusds}".
In grafana 9, the datasource key is a dict:

          "datasource": {
            "type": "prometheus",
            "uid": "${prometheusds}"
          },

To Reproduce

Relate cos-config to grafana and

juju config cos-configuration-k8s \
  git_repo=https://github.com/canonical/cos-configuration-k8s-operator.git \
  git_branch=feature/reinit_grafana_after_startup \
  grafana_dashboards_path=tests/samples/grafana_dashboards

Environment

Model                         Controller  Cloud/Region        Version  SLA          Timestamp
test-grafana-dashboards-k30t  chdv2934    microk8s/localhost  2.9.34   unsupported  12:40:32-05:00

App                    Version  Status   Scale  Charm                  Channel  Rev  Address        Exposed  Message
cos-configuration-k8s           active       1  cos-configuration-k8s             0  10.152.183.72  no       
grafana                9.2.1    waiting      1  grafana-k8s            edge      52  10.152.183.57  no       waiting for units to settle down

Relevant log output

File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1272, in _on_grafana_dashboard_relation_changed
    changes = self._render_dashboards_and_signal_changed(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1369, in _render_dashboards_and_signal_changed
    content = _convert_dashboard_fields(content, inject_dropdowns)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 599, in _convert_dashboard_fields
    dict_content = _replace_template_fields(dict_content, datasources, existing_templates)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 633, in _replace_template_fields
    if panel["datasource"].lower() in replacements.values():
AttributeError: 'dict' object has no attribute 'lower'

Additional context

No response

Turn dashboard URLs stable to make them suitable for usage in alerts

Password shown on action changes after scaling

Hello!

After deploying the cos-lite bundle:

juju add-model cos
juju deploy cos-lite --channel=edge --trust

I can get the dashboard password with:

juju run-action grafana get-admin-password

If I scale down the grafana app with:

juju remove-unit grafana --num-units 1

and then scale it back up:

juju add-unit grafana --num-units 1

Using get-admin-password shows a different one, but the dashboard is accessible with the old one.

Add default dashboard(s)

There should be at least a default K8s monitoring dashboard that is automatically configured. As soon as a data source (probably prometheus) is added, a user should be able to log into grafana and see a basic dashboard with standard K8s metrics.

refresh_event defaults to a non-existent event.

grafana-k8s-operator/lib/charms/grafana_k8s/v0/grafana_source.py

Line 361 in 9b3de5c

refresh_event = refresh_event or self._charm.on.pebble_ready

There is no such thing as a pebble_ready event. It should be prefixed with the name. If the refresh_event is omitted from the constructor, it breaks with the following error:

unit-prometheus-0: 11:49:35 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 375, in <module>
    main(PrometheusCharm)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 419, in main
    charm = charm_class(framework)
  File "./src/charm.py", line 75, in __init__
    self.grafana_source_consumer = GrafanaSourceProvider(
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 368, in __init__
    refresh_event = refresh_event or self._charm.on.pebble_ready
AttributeError: 'CharmEvents' object has no attribute 'pebble_ready'

Add jsonData support for Grafana Sources

Grafana datasources accept additional configurations, called jsonData, that allow to specify additional configurations.

Of particular interest is the jsonData of the Alertmanager datasource for Grafana 8, where we will need to differentiate between Prometheus' and Cortex' Alertmanager.

Grafana Dashboard with "creative" datasource names for Prometheus and Loki breaks the Grafana operator

Bug Description

The Grafana Dashboard below borks the Grafana operator:

Model  Controller   Cloud/Region        Version  SLA          Timestamp
cos    development  microk8s/localhost  2.9.27   unsupported  17:07:56+01:00

App           Version  Status   Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager           active       1  alertmanager-k8s  edge      13  10.152.183.177  no       
grafana                waiting      1  grafana-k8s       edge      31  10.152.183.43   no       installing agent
loki                   active       1  loki-k8s          edge      15  10.152.183.145  no       
prometheus             active       1  prometheus-k8s    edge      24  10.152.183.70   no       
spring-music           active       1  spring-music                 0  10.152.183.201  no       

Unit             Workload  Agent  Address      Ports  Message
alertmanager/0*  active    idle   10.1.151.86         
grafana/0*       error     idle   10.1.151.87         hook failed: "grafana-dashboard-relation-changed"
loki/0*          active    idle   10.1.151.88         
prometheus/0*    active    idle   10.1.151.89         
spring-music/0*  active    idle   10.1.151.90

To Reproduce

Add the following dashboard to the Spring Music charm, or provide it to Grafana via the COS Configuration charm:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "Dashboard for the Spring Music application, powered by Juju",
  "editable": true,
  "gnetId": 9845,
  "graphTooltip": 1,
  "id": 3,
  "iteration": 1639059314688,
  "links": [],
  "panels": [
    {
      "cacheTimeout": null,
      "datasource": "${promds}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {},
          "mappings": [
            {
              "from": "",
              "id": 1,
              "text": "0",
              "to": "",
              "type": 1,
              "value": ""
            }
          ],
          "max": 100,
          "min": 0,
          "noValue": "0",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "#299c46",
                "value": null
              },
              {
                "color": "rgba(237, 129, 40, 0.89)",
                "value": 1
              },
              {
                "color": "#d44a3a",
                "value": 5
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 7,
        "x": 0,
        "y": 0
      },
      "id": 31,
      "interval": null,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "text": {}
      },
      "pluginVersion": "7.4.1",
      "targets": [
        {
          "expr": "sum(increase(http_server_requests_seconds_count{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\", outcome!=\"SUCCESS\", outcome!=\"REDIRECTION\"}[$__range])) / sum(increase(http_server_requests_seconds_count{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\"}[$__range])) * 100",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "Error Rate",
      "type": "gauge"
    },
    {
      "cacheTimeout": null,
      "datasource": "${promds}",
      "description": "",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {},
          "decimals": 0,
          "mappings": [
            {
              "id": 0,
              "op": "=",
              "text": "0",
              "type": 1,
              "value": "null"
            }
          ],
          "noValue": "0",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": null
              }
            ]
          },
          "unit": "locale"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 17,
        "x": 7,
        "y": 0
      },
      "id": 39,
      "interval": "1m",
      "links": [],
      "maxDataPoints": null,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {},
        "textMode": "auto"
      },
      "pluginVersion": "7.4.1",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\"}[4m]))",
          "format": "time_series",
          "interval": "1m",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "Req / Min",
      "type": "stat"
    },
    {
      "collapsed": false,
      "datasource": null,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 8
      },
      "id": 2,
      "panels": [],
      "title": "Application",
      "type": "row"
    },
    {
      "aliasColors": {
        "Erroneous Requests": "rgb(255, 53, 75)",
        "Error": "red",
        "Success": "blue",
        "Successful Requests": "rgb(47, 117, 46)",
        "{level=\"ERROR\"}": "red",
        "{level=\"INFO\"}": "blue"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${promds}",
      "fieldConfig": {
        "defaults": {
          "color": {},
          "custom": {},
          "thresholds": {
            "mode": "absolute",
            "steps": []
          },
          "unit": "short"
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 9
      },
      "hiddenSeries": false,
      "id": 43,
      "interval": "60s",
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": false,
        "hideEmpty": true,
        "hideZero": false,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": true,
        "values": true
      },
      "lines": true,
      "linewidth": 3,
      "maxDataPoints": null,
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.1",
      "pointradius": 0.5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\", outcome!~\"SUCCESS|REDIRECTION\"}[4m]))",
          "hide": false,
          "instant": false,
          "interval": "1m",
          "intervalFactor": 1,
          "legendFormat": "Error",
          "refId": "B"
        },
        {
          "expr": "sum(rate(http_server_requests_seconds_count{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\", outcome=~\"SUCCESS|REDIRECTION\"}[4m]))",
          "hide": false,
          "interval": "1m",
          "legendFormat": "Success",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Requests/Outcome",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "transformations": [],
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:502",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:503",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "{level=\"ERROR\"}": "red",
        "{level=\"INFO\"}": "blue"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${lokidokids}",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 18
      },
      "hiddenSeries": false,
      "id": 46,
      "interval": null,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": true,
        "values": true
      },
      "lines": true,
      "linewidth": 3,
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.1",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(rate({juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\"}[1m])) by (level)",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Logs by severity",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:594",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:595",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "datasource": "${lokidokids}",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "gridPos": {
        "h": 13,
        "w": 24,
        "x": 0,
        "y": 27
      },
      "id": 45,
      "options": {
        "showLabels": false,
        "showTime": false,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "pluginVersion": "7.4.1",
      "targets": [
        {
          "expr": "{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\",juju_unit=\"$juju_unit\"}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "title": "Logs",
      "type": "logs"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 27,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "juju_lma_51fd2a3a-f900-4606-8c12-dc8f9febe555_prometheus_0",
          "value": "juju_lma_51fd2a3a-f900-4606-8c12-dc8f9febe555_prometheus_0"
        },
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Prometheus Datasource",
        "multi": false,
        "name": "promds",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "current": {
          "selected": false,
          "text": "juju_lma_51fd2a3a-f900-4606-8c12-dc8f9febe555_loki_0",
          "value": "juju_lma_51fd2a3a-f900-4606-8c12-dc8f9febe555_loki_0"
        },
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Loki Datasource",
        "multi": false,
        "name": "lokidokids",
        "options": [],
        "query": "loki",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "spring",
          "value": "spring"
        },
        "datasource": "${promds}",
        "definition": "label_values(up,juju_model)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Juju model",
        "multi": false,
        "name": "juju_model",
        "options": [],
        "query": {
          "query": "label_values(up,juju_model)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "c129a417-465c-4570-8c8b-b6e3402af25d",
          "value": "c129a417-465c-4570-8c8b-b6e3402af25d"
        },
        "datasource": "${promds}",
        "definition": "label_values(up{juju_model=\"$juju_model\"},juju_model_uuid)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Juju model uuid",
        "multi": false,
        "name": "juju_model_uuid",
        "options": [],
        "query": {
          "query": "label_values(up{juju_model=\"$juju_model\"},juju_model_uuid)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "spring-music",
          "value": "spring-music"
        },
        "datasource": "${promds}",
        "definition": "label_values(up{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\"},juju_application)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Juju application",
        "multi": false,
        "name": "juju_application",
        "options": [],
        "query": {
          "query": "label_values(up{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\"},juju_application)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "selected": true,
          "text": "spring-music/0",
          "value": "spring-music/0"
        },
        "datasource": "${promds}",
        "definition": "label_values(up{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\"},juju_unit)",
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": "Juju unit",
        "multi": false,
        "name": "juju_unit",
        "options": [],
        "query": {
          "query": "label_values(up{juju_model=\"$juju_model\",juju_model_uuid=\"$juju_model_uuid\",juju_application=\"$juju_application\"},juju_unit)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "Spring Music",
  "uid": "XwUYhT27k",
  "version": 7
}

Environment

A Juju controller next to you. cos-lite from edge.

Relevant log output

unit-grafana-0: 17:07:17 ERROR unit.grafana/0.juju-log grafana-dashboard:6: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 680, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1029, in _on_grafana_dashboard_relation_changed
    changes = self._render_dashboards_and_signal_changed(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1127, in _render_dashboards_and_signal_changed
    content = _encode_dashboard_content(_convert_dashboard_fields(content))
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 562, in _convert_dashboard_fields
    dict_content = _replace_template_fields(dict_content, datasources, existing_templates)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 596, in _replace_template_fields
    ds = re.sub(r"(\$|\{|\})", "", panel["datasource"])
  File "/usr/lib/python3.8/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Additional context

I took the Spring Music dashboard from the Spring Music charm, replaces prometheusds with promds, and lokids with lokidokids because I am devious.

Does not correctly advertise its external URL

Bug Description

When deploying the bundle and accessing the catalogue, the URL advertised by Grafana is the cluster internal URL, even though ingress is up and working. The URL provided by self.external_url ends up becoming something along the lines of http://grafana-0.grafana-endpoints.cos.svc.cluster.local:3000/cos-grafana, rather than the expected http://<traefik-ip-or-hostname>/cos-grafana

To Reproduce

juju deploy cos-lite --channel edge --trust
Navigate to the Catalogue UI
Hover the Grafana link

Environment

Juju v3.0-rc1
Microk8s v1.25.2

Relevant log output

unit-catalogue-0: 21:53:31 INFO unit.catalogue/0.juju-log catalogue:29: Configuring 3 application entries
unit-grafana-0: 21:53:31 INFO juju.worker.uniter.operation ran "catalogue-relation-created" hook (via hook dispatching script: dispatch)
unit-catalogue-0: 21:53:31 INFO juju.worker.uniter.operation ran "catalogue-relation-joined" hook (via hook dispatching script: dispatch)
unit-catalogue-0: 21:53:32 INFO unit.catalogue/0.juju-log catalogue:29: Configuring 3 application entries
unit-catalogue-0: 21:53:32 INFO juju.worker.uniter.operation ran "catalogue-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 21:53:32 INFO juju.worker.uniter.operation ran "catalogue-relation-joined" hook (via hook dispatching script: dispatch)
unit-catalogue-0: 21:53:32 INFO unit.catalogue/0.juju-log catalogue:29: Configuring 3 application entries
unit-catalogue-0: 21:53:33 INFO juju.worker.uniter.operation ran "catalogue-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 21:53:33 INFO juju.worker.uniter.operation ran "catalogue-relation-changed" hook (via hook dispatching script: dispatch)
unit-prometheus-0: 21:55:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

Removing the relation and adding it back makes no difference.

Dashboards do not survive node restart.

Bug Description

If you restart the k8s node, all the dasboards will be gone until you add any "grafana-dashboard" relation, at which point they are all restored.

To Reproduce

juju deploy grafana-k8s
juju deploy <anything>
juju relate grafana-k8s <anything>
sudo reboot
check for dashboards

Environment

dylan@protostar:~/loki$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

dylan@protostar:~/loki$ snap list
Name        Version        Rev    Tracking       Publisher   Notes
charmcraft  1.7.1          978    latest/stable  canonical✓  classic
core18      20220428       2409   latest/stable  canonical✓  base
core20      20220527       1518   latest/stable  canonical✓  base
docker      20.10.14       1779   latest/stable  canonical✓  -
juju        2.9.32         19681  latest/stable  canonical✓  classic
juju-wait   2.8.4~2.8.4    96     latest/stable  stub        classic
lxd         4.0.9-8e2046b  22753  4.0/stable/…   canonical✓  -
microk8s    v1.24.0        3272   1.24/stable    canonical✓  classic
snapd       2.56           16010  latest/stable  canonical✓  snapd

Relevant log output

Additional context

No response

Charm container stuck in crash loop backoff when using Juju 2.9.35 (VSphere)

Bug Description

The charm container gets stuck in a crash loop when deploying grafana-k8s using juju 2.9.35. The other containers (grafana and litestream) both have a ready status. This does not happen in microk8s, but does happen in Charmed-Kubernetes/Kubernetes core running in VSphere. I had previously opened issue where it was noted that when grafana is deployed the unit is rebooted/restarted, which seemed uncommon, so it might be some vsphere wonkyness at play.

I was able to deploy grafana on 2.9.34 as well as 2.9.33 so it seems related to 2.9.35 changes.I also deployed prometheus-k8s to see if this was an issue affecting other k8s charms, but prometheus did not seem to have problems and went active/idle after a minute or 2.

To Reproduce

Bootstrap a 2.9.35 controller on VSphere
Set model defaults: juju model-defaults vsphere juju-http-proxy=http://squid.internal:3128 apt-http-proxy=http://squid.internal:3128 snap-http-proxy=http://squid.internal:3128 juju-https-proxy=http://squid.internal:3128 apt-https-proxy=http://squid.internal:3128 snap-https-proxy=http://squid.internal:3128 apt-no-proxy=localhost,127.0.0.1,ppa.launchpad.net,launchpad.net juju-no-proxy=localhost,127.0.0.1,0.0.0.0,ppa.launchpad.net,launchpad.net,10.0.8.0/24,10.246.154.0/24
Add model for k8s-core: juju add-model --config enable-os-refresh-update=false --config enable-os-upgrade=false --config logging-config='<root>=DEBUG' --config datastore=vsanDatastore --config primary-network=$YOUR_VLAN_HERE k8s-core vsphere/Boston
Deploy k8s-core: juju deploy kubernetes-core --overlay vsphere-overlay.yaml --trust --debug --channel edge
The overlay file yaml looks like this:

description: Charmed Kubernetes overlay to add native vSphere support.
applications:
  vsphere-integrator:
    annotations:
      gui-x: "600"
      gui-y: "300"
    charm: vsphere-integrator
    num_units: 1
    trust: true
    options:
      datastore: vsanDatastore
      folder: k8s-crew-root
relations:
  - ['vsphere-integrator', 'kubernetes-control-plane']
  - ['vsphere-integrator', 'kubernetes-worker']

Copy kubeconfig: juju scp kubernetes-control-plane/0:config ~/.kube/config
Apply storage class: kubectl apply -f vsphere-storageclass.yaml
The storage class yaml looks like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mystorage
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: zeroedthick

add a k8s-cloud: juju add-k8s $YOUR_K8S_CLOUD --controller $YOUR_CONTROLLER --storage mystorage
Deploy grafana-k8s: juju deploy grafana-k8s --channel edge --trust

Environment

Juju version being used in 2.9.35. Cloud being used to deploy charms into is the Boston vsphere cloud. Kubernetes 1.25 is being deployed as part of the edge kubernetes-core bundle (a slimmed down version of charmed-kubernetes).

As mentioned above this does not happen on 2.9.34 or 2.9.33 juju versions. It is isolated to the newly released 2.9.35.

Relevant log output

# kubectl describe
kubectl describe pods -n stonepreston-cos
Name:             grafana-k8s-0
Namespace:        stonepreston-cos
Priority:         0
Service Account:  grafana-k8s
Node:             juju-ab7ac4-1/10.246.154.153
Start Time:       Mon, 17 Oct 2022 09:58:38 -0500
Labels:           app.kubernetes.io/name=grafana-k8s
                  controller-revision-hash=grafana-k8s-7696977bcb
                  statefulset.kubernetes.io/pod-name=grafana-k8s-0
Annotations:      controller.juju.is/id: 377150c5-51aa-422e-8707-d621b5754511
                  juju.is/version: 2.9.35
                  model.juju.is/id: 82239d5a-57f7-40a4-88e9-a3727951ff02
                  unit.juju.is/id: grafana-k8s/0
Status:           Running
IP:               192.168.102.136
IPs:
  IP:           192.168.102.136
Controlled By:  StatefulSet/grafana-k8s
Init Containers:
  charm-init:
    Container ID:  containerd://8b46c907e0c0aa34f39cd6dd83959bd39a33c55d1ad18246a1731e8a47258192
    Image:         rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.9.35
    Image ID:      rocks.canonical.com/cdk/jujusolutions/jujud-operator@sha256:b5313b7611b82efd9ac96a0d3c8da5e30e87aa11ff3b17ab71ee2c6a68cba758
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/containeragent
    Args:
      init
      --containeragent-pebble-dir
      /containeragent/pebble
      --charm-modified-version
      0
      --data-dir
      /var/lib/juju
      --bin-dir
      /charm/bin
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 17 Oct 2022 09:58:53 -0500
      Finished:     Mon, 17 Oct 2022 09:58:53 -0500
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      grafana-k8s-application-config  Secret  Optional: false
    Environment:
      JUJU_CONTAINER_NAMES:  grafana,litestream
      JUJU_K8S_POD_NAME:     grafana-k8s-0 (v1:metadata.name)
      JUJU_K8S_POD_UUID:      (v1:metadata.uid)
    Mounts:
      /charm/bin from charm-data (rw,path="charm/bin")
      /charm/containers from charm-data (rw,path="charm/containers")
      /containeragent/pebble from charm-data (rw,path="containeragent/pebble")
      /var/lib/juju from charm-data (rw,path="var/lib/juju")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mftr7 (ro)
Containers:
  charm:
    Container ID:  containerd://b975a221c7d6e27d8203e964881a1d8798910b9f9b04591fdebb3124d9f67f33
    Image:         rocks.canonical.com/cdk/jujusolutions/charm-base:ubuntu-20.04
    Image ID:      rocks.canonical.com/cdk/jujusolutions/charm-base@sha256:5ccefd1a92d63baa961680c22a47e01213c99e9c06280c732a1910a5c126f2d2
    Port:          <none>
    Host Port:     <none>
    Command:
      /charm/bin/pebble
    Args:
      run
      --http
      :38812
      --verbose
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 17 Oct 2022 10:05:23 -0500
      Finished:     Mon, 17 Oct 2022 10:05:58 -0500
    Ready:          False
    Restart Count:  5
    Liveness:       http-get http://:38812/v1/health%3Flevel=alive delay=30s timeout=1s period=5s #success=1 #failure=1
    Readiness:      http-get http://:38812/v1/health%3Flevel=ready delay=30s timeout=1s period=5s #success=1 #failure=1
    Environment:
      JUJU_CONTAINER_NAMES:  grafana,litestream
      HTTP_PROBE_PORT:       3856
    Mounts:
      /charm/bin from charm-data (ro,path="charm/bin")
      /charm/containers from charm-data (rw,path="charm/containers")
      /var/lib/juju from charm-data (rw,path="var/lib/juju")
      /var/lib/juju/storage/database/0 from grafana-k8s-database-163432a4 (rw)
      /var/lib/pebble/default from charm-data (rw,path="containeragent/pebble")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mftr7 (ro)
  grafana:
    Container ID:  containerd://937ed7f7d97046e0e19fb918750471946e0b328b2d4db6c179d7765c20a07dfc
    Image:         registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/grafana-image@sha256:cb9b47b4a53ae5f3da0fe40157e8eb20d0e120ae76da4dd9296f4b6c8ee62520
    Image ID:      registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/grafana-image@sha256:cb9b47b4a53ae5f3da0fe40157e8eb20d0e120ae76da4dd9296f4b6c8ee62520
    Port:          <none>
    Host Port:     <none>
    Command:
      /charm/bin/pebble
    Args:
      run
      --create-dirs
      --hold
      --http
      :38813
      --verbose
    State:          Running
      Started:      Mon, 17 Oct 2022 09:59:54 -0500
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:38813/v1/health%3Flevel=alive delay=30s timeout=1s period=5s #success=1 #failure=1
    Readiness:      http-get http://:38813/v1/health%3Flevel=ready delay=30s timeout=1s period=5s #success=1 #failure=1
    Environment:
      JUJU_CONTAINER_NAME:  grafana
      PEBBLE_SOCKET:        /charm/container/pebble.socket
    Mounts:
      /charm/bin/pebble from charm-data (ro,path="charm/bin/pebble")
      /charm/container from charm-data (rw,path="charm/containers/grafana")
      /var/lib/grafana from grafana-k8s-database-163432a4 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mftr7 (ro)
  litestream:
    Container ID:  containerd://5484bef6a363f50d77cbe6671059c9269502b9a72bb29dfb8c8e655467e134a9
    Image:         registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/litestream-image@sha256:8ab4b042f6c84ec51cabd5a9caef7b5394080c88fa1d7c445f201780e39e8ea7
    Image ID:      registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/litestream-image@sha256:8ab4b042f6c84ec51cabd5a9caef7b5394080c88fa1d7c445f201780e39e8ea7
    Port:          <none>
    Host Port:     <none>
    Command:
      /charm/bin/pebble
    Args:
      run
      --create-dirs
      --hold
      --http
      :38814
      --verbose
    State:          Running
      Started:      Mon, 17 Oct 2022 10:00:01 -0500
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:38814/v1/health%3Flevel=alive delay=30s timeout=1s period=5s #success=1 #failure=1
    Readiness:      http-get http://:38814/v1/health%3Flevel=ready delay=30s timeout=1s period=5s #success=1 #failure=1
    Environment:
      JUJU_CONTAINER_NAME:  litestream
      PEBBLE_SOCKET:        /charm/container/pebble.socket
    Mounts:
      /charm/bin/pebble from charm-data (ro,path="charm/bin/pebble")
      /charm/container from charm-data (rw,path="charm/containers/litestream")
      /var/lib/grafana from grafana-k8s-database-163432a4 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mftr7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  grafana-k8s-database-163432a4:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana-k8s-database-163432a4-grafana-k8s-0
    ReadOnly:   false
  charm-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-mftr7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/arch=amd64
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From                     Message
  ----     ------                  ----                    ----                     -------
  Warning  FailedScheduling        9m53s                   default-scheduler        0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Normal   Scheduled               9m51s                   default-scheduler        Successfully assigned stonepreston-cos/grafana-k8s-0 to juju-ab7ac4-1
  Normal   SuccessfulAttachVolume  9m50s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-da4fa45b-6002-442c-9d74-c413b45ac192"
  Normal   Pulled                  9m37s                   kubelet                  Container image "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.9.35" already present on machine
  Normal   Created                 9m36s                   kubelet                  Created container charm-init
  Normal   Started                 9m36s                   kubelet                  Started container charm-init
  Normal   Pulling                 9m35s                   kubelet                  Pulling image "rocks.canonical.com/cdk/jujusolutions/charm-base:ubuntu-20.04"
  Normal   Pulled                  9m18s                   kubelet                  Successfully pulled image "rocks.canonical.com/cdk/jujusolutions/charm-base:ubuntu-20.04" in 17.60974236s
  Normal   Pulling                 9m16s                   kubelet                  Pulling image "registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/grafana-image@sha256:cb9b47b4a53ae5f3da0fe40157e8eb20d0e120ae76da4dd9296f4b6c8ee62520"
  Normal   Pulled                  8m36s                   kubelet                  Successfully pulled image "registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/grafana-image@sha256:cb9b47b4a53ae5f3da0fe40157e8eb20d0e120ae76da4dd9296f4b6c8ee62520" in 39.667093839s
  Normal   Created                 8m36s                   kubelet                  Created container grafana
  Normal   Started                 8m35s                   kubelet                  Started container grafana
  Normal   Pulling                 8m35s                   kubelet                  Pulling image "registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/litestream-image@sha256:8ab4b042f6c84ec51cabd5a9caef7b5394080c88fa1d7c445f201780e39e8ea7"
  Normal   Pulled                  8m28s                   kubelet                  Successfully pulled image "registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/litestream-image@sha256:8ab4b042f6c84ec51cabd5a9caef7b5394080c88fa1d7c445f201780e39e8ea7" in 7.040337768s
  Normal   Created                 8m28s                   kubelet                  Created container litestream
  Normal   Started                 8m28s                   kubelet                  Started container litestream
  Warning  Unhealthy               7m54s (x2 over 7m54s)   kubelet                  Readiness probe failed: HTTP probe failed with statuscode: 502
  Normal   Created                 7m35s (x3 over 9m16s)   kubelet                  Created container charm
  Normal   Pulled                  7m35s (x2 over 8m27s)   kubelet                  Container image "rocks.canonical.com/cdk/jujusolutions/charm-base:ubuntu-20.04" already present on machine
  Normal   Started                 7m34s (x3 over 9m16s)   kubelet                  Started container charm
  Warning  BackOff                 4m34s (x10 over 7m51s)  kubelet                  Back-off restarting failed container


Name:             modeloperator-6bcb5dc5f9-hcgrd
Namespace:        stonepreston-cos
Priority:         0
Service Account:  modeloperator
Node:             juju-ab7ac4-1/10.246.154.153
Start Time:       Mon, 17 Oct 2022 09:58:25 -0500
Labels:           model.juju.is/disable-webhook=true
                  operator.juju.is/name=modeloperator
                  operator.juju.is/target=model
                  pod-template-hash=6bcb5dc5f9
Annotations:      <none>
Status:           Running
IP:               192.168.102.135
IPs:
  IP:           192.168.102.135
Controlled By:  ReplicaSet/modeloperator-6bcb5dc5f9
Containers:
  juju-operator:
    Container ID:  containerd://4fe3543de3f1fd2358a7036420fe3fcc87e30593bf6ea2c403a270c0f1d94d6b
    Image:         rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.9.35
    Image ID:      rocks.canonical.com/cdk/jujusolutions/jujud-operator@sha256:b5313b7611b82efd9ac96a0d3c8da5e30e87aa11ff3b17ab71ee2c6a68cba758
    Port:          17071/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
    Args:
      -c
      export JUJU_DATA_DIR=/var/lib/juju
      export JUJU_TOOLS_DIR=$JUJU_DATA_DIR/tools
      
      mkdir -p $JUJU_TOOLS_DIR
      cp /opt/jujud $JUJU_TOOLS_DIR/jujud
      
      $JUJU_TOOLS_DIR/jujud model --model-uuid=82239d5a-57f7-40a4-88e9-a3727951ff02
      
    State:          Running
      Started:      Mon, 17 Oct 2022 09:58:52 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      HTTP_PORT:          17071
      SERVICE_NAME:       modeloperator
      SERVICE_NAMESPACE:  stonepreston-cos
    Mounts:
      /var/lib/juju/agents/model-82239d5a-57f7-40a4-88e9-a3727951ff02/template-agent.conf from modeloperator (rw,path="template-agent.conf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jhmvb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  modeloperator:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      modeloperator
    Optional:  false
  kube-api-access-jhmvb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  10m    default-scheduler  Successfully assigned stonepreston-cos/modeloperator-6bcb5dc5f9-hcgrd to juju-ab7ac4-1
  Normal  Pulling    10m    kubelet            Pulling image "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.9.35"
  Normal  Pulled     9m39s  kubelet            Successfully pulled image "rocks.canonical.com/cdk/jujusolutions/jujud-operator:2.9.35" in 23.822076221s
  Normal  Created    9m37s  kubelet            Created container juju-operator
  Normal  Started    9m37s  kubelet            Started container juju-operator

# juju debug-log
juju debug-log
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:25:46 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:25:46 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:26:16 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:25:46 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:25:46 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:25:46 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.migrationminion migration phase is now: NONE
^C
stone@stone-desktop:~/ck/charm-kube-ovn$ juju debug-log --replay
controller-0: 09:58:23 INFO juju.worker.apicaller [82239d] "machine-0" successfully connected to "localhost:17070"
controller-0: 09:58:23 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 09:58:23 INFO juju.worker.logger logger worker started
controller-0: 09:58:23 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for stonepreston-cos (82239d5a-57f7-40a4-88e9-a3727951ff02)
controller-0: 09:58:23 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for stonepreston-cos (82239d5a-57f7-40a4-88e9-a3727951ff02)
controller-0: 09:58:32 INFO juju.worker.caasapplicationprovisioner.runner start "grafana-k8s"
model-82239d5a-57f7-40a4-88e9-a3727951ff02: 09:58:53 INFO juju.worker.caasupgrader abort check blocked until version event received
model-82239d5a-57f7-40a4-88e9-a3727951ff02: 09:58:53 INFO juju.worker.caasupgrader unblocking abort check
model-82239d5a-57f7-40a4-88e9-a3727951ff02: 09:58:55 INFO juju.worker.muxhttpserver starting http server on [::]:17071
model-82239d5a-57f7-40a4-88e9-a3727951ff02: 09:58:55 INFO juju.worker.caasadmission ensuring model k8s webhook configurations
unit-grafana-k8s-0: 09:59:13 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 09:59:13 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 09:59:13 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 09:59:13 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 09:59:13 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 09:59:13 INFO juju.worker.uniter.charm downloading ch:amd64/focal/grafana-k8s-45 from API server
unit-grafana-k8s-0: 09:59:13 INFO juju.downloader downloading from ch:amd64/focal/grafana-k8s-45
unit-grafana-k8s-0: 09:59:13 INFO juju.downloader download complete ("ch:amd64/focal/grafana-k8s-45")
unit-grafana-k8s-0: 09:59:14 INFO juju.downloader download verified ("ch:amd64/focal/grafana-k8s-45")
unit-grafana-k8s-0: 09:59:43 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:00:02 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:00:02 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:00:02 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:00:02 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:00:02 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:00:02 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:00:03 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:00:03 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:00:03 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:00:03 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:00:32 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:00:55 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:00:55 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:00:55 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:00:55 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:00:55 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:00:55 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:01:25 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:01:58 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:01:58 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:01:58 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:01:58 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:01:58 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:01:58 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:02:28 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:03:19 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:03:19 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:03:19 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:03:19 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:03:19 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:03:19 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:03:49 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:05:23 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:05:23 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:05:23 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:05:23 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:05:23 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:05:23 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:05:23 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:05:53 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:08:45 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:08:45 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:08:45 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:08:45 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:08:45 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:08:45 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:09:15 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:14:27 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:14:27 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:14:27 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:14:27 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:14:27 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:14:27 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:14:27 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:14:57 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:20:06 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:20:06 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:20:06 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:20:06 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:20:06 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:20:06 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:20:36 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 10:25:46 INFO juju.cmd running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
unit-grafana-k8s-0: 10:25:46 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.upgradesteps upgrade steps for 2.9.35 have already been run.
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.probehttpserver starting http server on [::]:65301
unit-grafana-k8s-0: 10:25:46 INFO juju.api connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.apicaller [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 10:25:46 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 10:25:46 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 10:25:46 INFO juju.worker.uniter.charm detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
unit-grafana-k8s-0: 10:26:16 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM

# kubectl log of charm container
kubectl logs grafana-k8s-0 -c charm -n stonepreston-cos
2022-10-17T15:25:46.086Z [pebble] HTTP API server listening on ":38812".
2022-10-17T15:25:46.086Z [pebble] Started daemon.
2022-10-17T15:25:46.146Z [pebble] POST /v1/services 59.993765ms 202
2022-10-17T15:25:46.147Z [pebble] Started default services with change 19.
2022-10-17T15:25:46.174Z [pebble] Service "container-agent" starting: /charm/bin/containeragent unit --data-dir /var/lib/juju --append-env "PATH=$PATH:/charm/bin" --show-log --charm-modified-version 0
2022-10-17T15:25:46.228Z [container-agent] 2022-10-17 15:25:46 INFO juju.cmd supercommand.go:56 running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
2022-10-17T15:25:46.228Z [container-agent] starting containeragent unit command
2022-10-17T15:25:46.228Z [container-agent] containeragent unit "unit-grafana-k8s-0" start (2.9.35 [gc])
2022-10-17T15:25:46.229Z [container-agent] 2022-10-17 15:25:46 INFO juju.cmd.containeragent.unit runner.go:556 start "unit"
2022-10-17T15:25:46.229Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.upgradesteps worker.go:60 upgrade steps for 2.9.35 have already been run.
2022-10-17T15:25:46.230Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.probehttpserver server.go:157 starting http server on [::]:65301
2022-10-17T15:25:46.247Z [container-agent] 2022-10-17 15:25:46 INFO juju.api apiclient.go:688 connection established to "wss://10.246.154.88:17070/model/82239d5a-57f7-40a4-88e9-a3727951ff02/api"
2022-10-17T15:25:46.261Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.apicaller connect.go:163 [82239d] "unit-grafana-k8s-0" successfully connected to "10.246.154.88:17070"
2022-10-17T15:25:46.283Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2022-10-17T15:25:46.290Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.logger logger.go:120 logger worker started
2022-10-17T15:25:46.297Z [container-agent] 2022-10-17 15:25:46 WARNING juju.worker.proxyupdater proxyupdater.go:282 unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
2022-10-17T15:25:46.313Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.caasupgrader upgrader.go:113 abort check blocked until version event received
2022-10-17T15:25:46.313Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.caasupgrader upgrader.go:119 unblocking abort check
2022-10-17T15:25:46.314Z [container-agent] 2022-10-17 15:25:46 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
2022-10-17T15:25:46.334Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.leadership tracker.go:194 grafana-k8s/0 promoted to leadership of grafana-k8s
2022-10-17T15:25:46.336Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.uniter uniter.go:326 unit "grafana-k8s/0" started
2022-10-17T15:25:46.338Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.uniter uniter.go:631 resuming charm install
2022-10-17T15:25:46.412Z [container-agent] 2022-10-17 15:25:46 INFO juju.worker.uniter.charm manifest_deployer.go:182 detected interrupted deploy of charm "ch:amd64/focal/grafana-k8s-45"
2022-10-17T15:25:56.120Z [pebble] Check "readiness" failure 1 (threshold 3): received non-20x status code 418
2022-10-17T15:26:06.118Z [pebble] Check "readiness" failure 2 (threshold 3): received non-20x status code 418
2022-10-17T15:26:16.119Z [pebble] Check "readiness" failure 3 (threshold 3): received non-20x status code 418
2022-10-17T15:26:16.119Z [pebble] Check "readiness" failure threshold 3 hit, triggering action
2022-10-17T15:26:16.119Z [pebble] Service "container-agent" on-check-failure action is "shutdown", triggering server exit
2022-10-17T15:26:16.119Z [pebble] Server exiting!
2022-10-17T15:26:16.147Z [pebble] Stopping all running services.
2022-10-17T15:26:16.249Z [container-agent] 2022-10-17 15:26:16 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2022-10-17T15:26:21.271Z [pebble] Service "container-agent" stopped

Additional context

No response

Convention over Configuration for Grafana dashboard templates

The consumer library should automatically look into the src/grafana_dashboards/ folder and scoop up as Grafana dashboards everything that follows the filename pattern <name>.json.j2.

The add_dashboard API is kept for programmatic use-cases.

Lock Grafana into single node or HA mode from the start

Based on the team review of this charm, we need to lock the setting of Grafana into HA or single node mode because if it starts in single node mode and then scales up, we would need to migrate the sqlite3 database to the mysql or postgres database.

However, we need to have goal-state exposed in OF to accomplish this. Blocked for now (as of Nov. 12, 2020).

Allow opt-out of automatic injection of topology filters

Only run libcheck for PRs when the remote is a local feature branch

Libcheck will always fail on PRs from forks, as they don't have access to the secrets. We should probably make it conditional on the remote branch being local to the repo.

Originally posted by @simskij in #74 (comment)

`Error: ImagePullBackOff` sends charm to error state on config-changed

I noticed that when the charm is "on hold" due to "Error: ImagePullBackOff" then grafana goes into error state.

Shouldn't it go into Waiting or Maintenance?

$ juju show-status-log grafana/0
Time                   Type       Status       Message
09 Feb 2022 21:04:03Z  juju-unit  allocating   Error: ImagePullBackOff
09 Feb 2022 21:04:03Z  workload   waiting      agent initializing
09 Feb 2022 21:04:09Z  juju-unit  executing    running install hook
09 Feb 2022 21:04:15Z  juju-unit  executing    running grafana-peers-relation-created hook
09 Feb 2022 21:04:17Z  juju-unit  executing    running leader-elected hook
09 Feb 2022 21:04:20Z  juju-unit  executing    running database-storage-attached hook
09 Feb 2022 21:04:22Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:04:24Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:04:29Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:04:31Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:04:38Z  juju-unit  allocating   Back-off pulling image "registry.jujucharms.com/charm/h71m6jk2jeap1qu5lv9nv5mplqayr91q34lqp/grafana-image@sha256:f80dd9655394540852f4996d3a47deab36cb34b7c556fc002e1621bf926f3708"
09 Feb 2022 21:04:38Z  workload   maintenance  installing charm software
09 Feb 2022 21:04:41Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:04:42Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:05:02Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:05:03Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:05:44Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:05:45Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:07:07Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:07:08Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:09:52Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:09:53Z  juju-unit  error        hook failed: "config-changed"
09 Feb 2022 21:14:53Z  juju-unit  executing    running config-changed hook
09 Feb 2022 21:14:54Z  juju-unit  error        hook failed: "config-changed"

$ juju status
Model               Controller  Cloud/Region        Version  SLA          Timestamp
cos-lite-load-test  uk8s        microk8s/localhost  2.9.24   unsupported  21:20:18Z

App               Version  Status   Scale  Charm                         Store     Channel  Rev  OS          Address         Message
alertmanager-k8s           waiting    0/1  alertmanager-k8s              charmhub  edge       9  kubernetes  10.152.183.230  installing agent
grafana                    waiting      1  grafana-k8s                   charmhub  edge      20  kubernetes  10.152.183.83   installing agent
loki                       waiting    0/1  loki-k8s                      charmhub  edge      15  kubernetes  10.152.183.29   installing agent
prometheus                 active       1  prometheus-k8s                charmhub  edge      19  kubernetes  10.152.183.63   
scrape-config              active       1  prometheus-scrape-config-k8s  charmhub  edge      21  ubuntu      10.152.183.137  
scrape-target              active       1  prometheus-scrape-target-k8s  charmhub  edge       8  ubuntu      10.152.183.38   

Unit                 Workload     Agent       Address     Ports  Message
alertmanager-k8s/0*  maintenance  allocating  10.1.66.16         Waiting for pod startup to complete
grafana/0*           error        idle        10.1.66.17         hook failed: "config-changed"
loki/0*              waiting      allocating  10.1.66.18         HTTPConnectionPool(host='localhost', port=3100): Max retries exceeded with url: /loki/api/v1/status/buildinfo (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4d041a65e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
prometheus/0*        active       idle        10.1.66.13         
scrape-config/0*     active       idle        10.1.66.14         
scrape-target/0*     active       idle        10.1.66.15

unit-grafana-0: 19:25:30 INFO unit.grafana/0.juju-log Updated Grafana's base configuration
unit-grafana-0: 19:25:30 DEBUG unit.grafana/0.juju-log Pebble API is not ready; ConnectionError: [Errno 2] No such file or directory
unit-grafana-0: 19:25:30 ERROR unit.grafana/0.juju-log Could not push datasource config. Pebble refused connection. Shutting down?
unit-grafana-0: 19:25:30 INFO unit.grafana/0.juju-log Updated Grafana's datasource configuration
unit-grafana-0: 19:25:30 ERROR unit.grafana/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 67, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 1144, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 81, in http_open
    return self.do_open(_UnixSocketConnection, req, socket_path=self.socket_path)
  File "/usr/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 657, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 426, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 141, in _on_config_changed
    self._configure()
  File "./src/charm.py", line 196, in _configure
    if self.container.get_plan().services != self._build_layer().services:
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/model.py", line 1189, in get_plan
    return self._pebble.get_plan()
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 1428, in get_plan
    resp = self._request('GET', '/v1/plan', {'format': 'yaml'})
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 1115, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/pebble.py", line 1157, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory

Need to swap names of in GrafanaSourceProvider and GrafanaSourceConsumer

WHen we switched the roles of the Grafana relations to align it with "Provider has the server socket" rule, we have not aligned the names of the objects in the grafana_source.py library.

Reuse dashboards across Juju applications with the same charm and version

Grafana dashboards in Grafana should be shared across all versions of the same charm.

Dashboards in Grafana should be identified by <charm_name>/<charm_version>, so that we can use the grouping inside the dashboard to handle multiple instances of the same charm and allow quick comparison of, e.g., performance of various Cassandra clusters in one go.

charm goes to error state - hook failed: "config-changed"

Bug Description

During deployment as part of cos-lite we encountered

Model  Controller              Cloud/Region      Version  SLA          Timestamp
cos    foundations-kubernetes  kubernetes_cloud  2.9.34   unsupported  16:01:10Z

App           Version  Status   Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager           waiting    0/1  alertmanager-k8s  edge      31  10.152.183.89   no       installing agent
grafana                waiting      1  grafana-k8s       edge      43  10.152.183.121  no       installing agent
loki                   waiting      1  loki-k8s          edge      45  10.152.183.225  no       installing agent
prometheus             waiting      1  prometheus-k8s    edge      71  10.152.183.154  no       installing agent
traefik                waiting      1  traefik-k8s       edge      87  10.246.167.235  no       installing agent

Unit             Workload  Agent  Address          Ports  Message
alertmanager/0*  unknown   lost                           agent lost, see 'juju show-status-log alertmanager/0'
grafana/0*       error     idle   192.168.252.133         hook failed: "config-changed"
loki/0*          waiting   idle                           Waiting for resource limit patch to apply
prometheus/0*    waiting   idle   192.168.252.134         Waiting for resource limit patch to apply
traefik/0*       waiting   idle   192.168.254.7           waiting for service: 'traefik'

To Reproduce

This was a failure in SQA unattended deployment, we deploy cos-lite with edge channel

Environment

This was a failure in SQA unattended deployment, we deploy cos-lite with edge channel

Relevant log output

No way to catch logs as the deployments are constantly replaced in SQA.
Here is a link to artifacts
https://oil-jenkins.canonical.com/artifacts/4dd077fa-63bf-4d76-83d1-5a573d974790/index.html

juju crashdump for cos is https://oil-jenkins.canonical.com/artifacts/4dd077fa-63bf-4d76-83d1-5a573d974790/generated/generated/cos/juju-crashdump-cos-2022-09-26-16.01.25.tar.gz

Additional context

This was a failure in SQA unattended deployment, we deploy cos-lite with edge channel

json error in provided dashboard sends the charm into error state

I had a stray comma in the json dashboard provided, which resulted in:

unit-grafana-0: 02:40:24 ERROR unit.grafana/0.juju-log grafana-dashboard:9: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 657, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 426, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 938, in _on_grafana_dashboard_relation_changed
    self._render_dashboards_and_emit_event(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1020, in _render_dashboards_and_emit_event
    content = _encode_dashboard_content(_inject_dashboard_dropdowns(content))
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 522, in _inject_dashboard_dropdowns
    dict_content = json.loads(content)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)

Bug reference: OPENG-616

Customize the Grafana UI to include COS branding

Automate label selector injection in dashboard panels when topology filters is enabled

Code analysis failed in grafana_dashboard lib

Bug Description

For code analysis, I use pylint.

It shows me this error in the grafana_dashboard library:

lib/charms/grafana_k8s/v0/grafana_dashboard.py:999:8: E0401: Unable to import 'jinja2' (import-error)
lib/charms/grafana_k8s/v0/grafana_dashboard.py:1000:8: E0401: Unable to import 'jinja2.exceptions' (import-error)
lib/charms/grafana_k8s/v0/grafana_dashboard.py:1322:12: E0401: Unable to import 'jinja2' (import-error)

To Reproduce

python3 -m pip install pylint
pylint -E lib/

Environment

No need to run the charm.

Relevant log output

lib/charms/grafana_k8s/v0/grafana_dashboard.py:999:8: E0401: Unable to import 'jinja2' (import-error)
lib/charms/grafana_k8s/v0/grafana_dashboard.py:1000:8: E0401: Unable to import 'jinja2.exceptions' (import-error)
lib/charms/grafana_k8s/v0/grafana_dashboard.py:1322:12: E0401: Unable to import 'jinja2' (import-error)

Additional context

No response

Suggested feature: Add convenience action for providing all login information

This is purely a suggestion for a convenience function.

The get-admin-password action is great for getting the grafana dashboard password, but whenever I use it I'm really looking for:

password
url
login (I realise it is fixed, but for completeness...)

There are ways of getting the other info programatically (could use juju to get the unit IP), but I think in most cases when someone wants password they probably also want to know the dashboard url to use the password on. Would be great if they were all returned together (either in get-admin-password, or in a similar get-dashboard-login-details or whatever)

Specify required plugins over grafana_dashboard

It would be cool if charms could ask for plugins.

Flaky integration tests

Bug Description

test_selfmonitoring_dashboard::test_remove occasionally fails.

After removing prom,

await ops_test.model.applications[prometheus_app_name].remove()

The test makes an assertion

relation_removed_dashboards = await get_grafana_dashboards(ops_test, grafana_app_name, 0)
assert relation_removed_dashboards == []

That fails. Occasionally.
https://github.com/canonical/grafana-k8s-operator/actions/runs/3177380221/jobs/5177760631#step:5:801

To Reproduce

Run itests.

Environment

GitHub runner

Relevant log output

Traceback (most recent call last):
  File "/home/runner/work/grafana-k8s-operator/grafana-k8s-operator/tests/integration/test_selfmonitoring_dashboard.py", line 88, in test_remove
    assert relation_removed_dashboards == []
AssertionError: assert [{'id': 1, 'i...eta': 0, ...}] == []
  Left contains one more item: {'id': 1, 'isStarred': False, 'slug': '', 'sortMeta': 0, ...}
  Full diff:
    [
  -  ,
  +  {'id': 1,
  +   'isStarred': False,
  +   'slug': '',...

Additional context

No response

Inject Juju topology label filters for widgets that use a datasource of type prometheus

The PromQL transform utility allows to inject label filters into PromQL queries. This, coupled with the already-existing Juju topology variables that are automatically injected by the grafana-dashboard library, will result in automatically applying the values of those dashboard variables to the widgets without any additional work by charm authors.

Missing OCI image from the readme

following the instructions in the readme:

ubuntu@primary:~/grafana-operator$ charmcraft build
Created 'grafana.charm'.
ubuntu@primary:~/grafana-operator$ juju deploy ./grafana.charm
ERROR local charm missing OCI images for: grafana-image

Provide authentication support for general Oauth or Anonymous access

Enhancement Proposal

In some cases we'd like to be able to enable anonymous read-only access to grafana in K8s, similar to how we currently can do with the machine version of the grafana charm.

Typically we'd put a proxy in front of this with SSO authentication tied to a particular team, but taking this approach means that we don't have to create specific users (and share/manage those credentials) for those who just want to browse existing graphs. That SSO proxy would happen outside of this charm, fwiw.

The generated grafana source contains duplicate schemes

Bug Description

Relating a Prometheus unit to Grafana incorrectly writes a data source to disk that includes dual scheme parts.

Bug is here

To Reproduce

juju deploy prometheus-k8s --channel=edge
juju deploy grafana-k8s --channel=edge
juju relate prometheus-k8s:grafana-source grafana-k8s

Environment

N/A

Relevant log output

root@grafana-0:/etc/grafana/provisioning/datasources# cat datasources.yaml 
apiVersion: 1
datasources:
- access: proxy
  isDefault: 'false'
  name: juju_cos_1c9d8fff-d8e0-4b2e-81fc-8739256dec47_prometheus_0
  orgId: '1'
  type: prometheus
  url: http://http://prometheus-0.prometheus-endpoints.cos.svc.cluster.local:9090
- access: proxy
  isDefault: 'false'
  name: juju_cos_1c9d8fff-d8e0-4b2e-81fc-8739256dec47_loki_0
  orgId: '1'
  type: loki
  url: http://10.1.151.82:3100
deleteDatasources: []
root@grafana-0:/etc/grafana/provisioning/datasources#

Additional context

  - endpoint: grafana-source
    related-endpoint: grafana-source
    application-data:
      grafana_source_data: '{"model": "cos", "model_uuid": "1c9d8fff-d8e0-4b2e-81fc-8739256dec47",
        "application": "prometheus", "type": "prometheus"}'
    related-units:
      prometheus/0:
        in-scope: true
        data:
          egress-subnets: 10.152.183.231/32
          grafana_source_host: http://prometheus-0.prometheus-endpoints.cos.svc.cluster.local:9090
          ingress-address: 10.152.183.231
          private-address: 10.152.183.231

Grafana stuck on waiting - installing agent

Bug Description

In test run https://solutions.qa.canonical.com/v2/testruns/d4676f04-a422-47f1-a066-40a974baeebe, the deployment blocks with:

Model  Controller    Cloud/Region              Version  SLA          Timestamp
cos    popocatepetl  microk8s_cloud/localhost  2.9.37   unsupported  12:08:08Z

App           Version  Status   Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.23.0   active       1  alertmanager-k8s  edge      37  10.152.183.182  no       
catalogue              active       1  catalogue-k8s     edge       4  10.152.183.190  no       
grafana       9.2.1    waiting      1  grafana-k8s       edge      54  10.152.183.180  no       installing agent
loki          2.4.1    active       1  loki-k8s          edge      47  10.152.183.27   no       
prometheus    2.33.5   active       1  prometheus-k8s    edge      86  10.152.183.243  no       
traefik                active       1  traefik-k8s       edge      94  10.244.40.91    no       

Unit             Workload     Agent  Address      Ports  Message
alertmanager/0*  active       idle   10.1.35.156         
catalogue/0*     active       idle   10.1.35.145         
grafana/0        maintenance  idle                       installing charm software
loki/0*          active       idle   10.1.35.157         
prometheus/0*    active       idle   10.1.35.158         
traefik/0*       active       idle   10.1.35.155

In the microk8s crashdump the pod logs for grafana are missing. There are some mentions of grafana in the microk8s logs, but nothing that indicates a clear failure to me.

Other logs and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/d4676f04-a422-47f1-a066-40a974baeebe/index.html

To Reproduce

Deploy microk8s 1.26, add metallb, deploy cos

Environment

microk8s 1.26
cos-lite latest/edge

Relevant log output

https://oil-jenkins.canonical.com/artifacts/d4676f04-a422-47f1-a066-40a974baeebe/index.html

Additional context

No response

Charm does not go active/idle if restart_grafana is never called

Bug Description

When deploying the edge grafana-k8s charm on top of Charmed Kubernetes, I noticed the grafana charm will not ever go active. It gets stuck in a waiting status.

It seems setting active status only happens in the restart_grafana method. If this method never ends up being called because it passes the various checks in _configure, then it seems it will be stuck in waiting forever.

Pod has a ready status, all its containers are up and running, and there is not really any concerning logs other than the litestream logs I included down below. The grafana container seems fine. The juju debug-log also appears normal without any stacktraces or errors.

I tried this on both a VSphere-deployed CK 1.25 and microk8s. Microk8s did not face this issue, but CK did.

I added some log statements to the restart_grafana method, and saw that it was not being called. I forced a restart to be called by removing the check in a locally built grafana-k8s operator, and the charm did go active.

To Reproduce

Deploy the edge grafana-k8s operator in a CK-backed k8s-cloud (in my case I am using CK 1.25/stable deployed in a VSphere env)

juju deploy grafana-k8s --channel edge

Environment

I am running things in our CK VSphere. My controller is version 2.9.32. CK deployed is 1.25/stable along with the vsphere integrator so storage is possible as described here

The grafana-k8s charm is being deployed from the edge channel, version 44.

Relevant log output

# Litestream container logs
[litestream] /var/lib/grafana/grafana.db: init: no wal files available, clearing generation
2022-09-30T17:36:25.142Z [litestream] /var/lib/grafana/grafana.db: sync error, retrying: ensure wal exists: no such table: _litestream_seq

# Juju debug-log
unit-grafana-k8s-0: 14:31:47 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:31:49 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:34:31 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-grafana-k8s-0: 14:34:31 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 14:34:33 INFO juju.worker.uniter.operation ran "database-storage-detaching" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:34:35 INFO juju.worker.uniter.operation ran "stop" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:34:37 INFO juju.worker.uniter.operation ran "remove" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:34:37 INFO juju.worker.uniter unit "grafana-k8s/0" shutting down: agent should be terminated
controller-0: 14:34:47 INFO juju.worker.caasapplicationprovisioner.runner stopped "grafana-k8s", err: <nil>
controller-0: 14:35:42 INFO juju.worker.caasapplicationprovisioner.runner start "grafana-k8s"
unit-grafana-k8s-0: 14:35:55 INFO juju.cmd running containerAgent [2.9.32 c360f3d92b40458cf15512a7fe5eddb0e7ae57b2 gc go1.18.3]
unit-grafana-k8s-0: 14:35:55 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.upgradesteps upgrade steps for 2.9.32 have already been run.
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.probehttpserver starting http server on [::]:3856
unit-grafana-k8s-0: 14:35:55 INFO juju.api connection established to "wss://10.246.154.149:17070/model/09841705-6c15-491b-87db-1cd68a04cab0/api"
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.apicaller [098417] "unit-grafana-k8s-0" successfully connected to "10.246.154.149:17070"
unit-grafana-k8s-0: 14:35:55 INFO juju.api connection established to "wss://10.246.154.149:17070/model/09841705-6c15-491b-87db-1cd68a04cab0/api"
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.apicaller [098417] "unit-grafana-k8s-0" successfully connected to "10.246.154.149:17070"
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 14:35:55 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 14:35:55 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.uniter resuming charm install
unit-grafana-k8s-0: 14:35:55 INFO juju.worker.uniter.charm downloading ch:amd64/focal/grafana-k8s-44 from API server
unit-grafana-k8s-0: 14:35:55 INFO juju.downloader downloading from ch:amd64/focal/grafana-k8s-44
unit-grafana-k8s-0: 14:35:56 INFO juju.downloader download complete ("ch:amd64/focal/grafana-k8s-44")
unit-grafana-k8s-0: 14:35:56 INFO juju.downloader download verified ("ch:amd64/focal/grafana-k8s-44")
unit-grafana-k8s-0: 14:36:03 INFO juju.worker.uniter hooks are retried true
unit-grafana-k8s-0: 14:36:03 INFO juju.worker.uniter found queued "install" hook
unit-grafana-k8s-0: 14:36:05 INFO unit.grafana-k8s/0.juju-log Running legacy hooks/install.
unit-grafana-k8s-0: 14:36:06 INFO unit.grafana-k8s/0.juju-log Successfully patched the Kubernetes service!
unit-grafana-k8s-0: 14:36:07 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:09 INFO juju.worker.uniter.operation ran "grafana-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:09 INFO juju.worker.uniter found queued "leader-elected" hook
unit-grafana-k8s-0: 14:36:11 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:12 WARNING unit.grafana-k8s/0.juju-log Could not read configuration from the Grafana workload container: not-found - stat /etc/grafana/grafana-config.ini: no such file or directory
unit-grafana-k8s-0: 14:36:12 INFO unit.grafana-k8s/0.juju-log Updated Grafana's base configuration
unit-grafana-k8s-0: 14:36:12 WARNING unit.grafana-k8s/0.juju-log Could not read configuration from the Grafana workload container: not-found - stat /etc/grafana/provisioning/datasources/datasources.yaml: no such file or directory
unit-grafana-k8s-0: 14:36:12 INFO unit.grafana-k8s/0.juju-log Updated Grafana's datasource configuration
unit-grafana-k8s-0: 14:36:12 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests=None), actual=ResourceRequirements(limits=None, requests=None)
unit-grafana-k8s-0: 14:36:13 INFO juju.worker.uniter.operation ran "grafana-pebble-ready" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:15 INFO juju.worker.uniter.operation ran "litestream-pebble-ready" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:17 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:18 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests=None), actual=ResourceRequirements(limits=None, requests=None)
unit-grafana-k8s-0: 14:36:19 INFO unit.grafana-k8s/0.juju-log Restarted replication
unit-grafana-k8s-0: 14:36:19 INFO unit.grafana-k8s/0.juju-log Kubernetes resources for app 'grafana-k8s', container 'grafana' patched successfully: ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:36:19 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
unit-grafana-k8s-0: 14:36:20 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:20 INFO juju.worker.uniter found queued "start" hook
unit-grafana-k8s-0: 14:36:21 INFO unit.grafana-k8s/0.juju-log Running legacy hooks/start.
unit-grafana-k8s-0: 14:36:23 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:25 INFO juju.worker.uniter.operation ran "stop" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:36:25 INFO juju.worker.uniter unit "grafana-k8s/0" shutting down: agent should be terminated
unit-grafana-k8s-0: 14:36:45 INFO juju.cmd running containerAgent [2.9.32 c360f3d92b40458cf15512a7fe5eddb0e7ae57b2 gc go1.18.3]
unit-grafana-k8s-0: 14:36:45 INFO juju.cmd.containeragent.unit start "unit"
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.upgradesteps upgrade steps for 2.9.32 have already been run.
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.probehttpserver starting http server on [::]:3856
unit-grafana-k8s-0: 14:36:45 INFO juju.api connection established to "wss://10.246.154.149:17070/model/09841705-6c15-491b-87db-1cd68a04cab0/api"
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.apicaller [098417] "unit-grafana-k8s-0" successfully connected to "10.246.154.149:17070"
unit-grafana-k8s-0: 14:36:45 INFO juju.api connection established to "wss://10.246.154.149:17070/model/09841705-6c15-491b-87db-1cd68a04cab0/api"
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.apicaller [098417] "unit-grafana-k8s-0" successfully connected to "10.246.154.149:17070"
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.logger logger worker started
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.leadership grafana-k8s/0 promoted to leadership of grafana-k8s
unit-grafana-k8s-0: 14:36:45 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.caasupgrader unblocking abort check
unit-grafana-k8s-0: 14:36:45 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-grafana-k8s-0
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.uniter unit "grafana-k8s/0" started
unit-grafana-k8s-0: 14:36:45 INFO juju.worker.uniter hooks are retried true
unit-grafana-k8s-0: 14:36:46 INFO juju.worker.uniter.charm downloading ch:amd64/focal/grafana-k8s-44 from API server
unit-grafana-k8s-0: 14:36:46 INFO juju.downloader downloading from ch:amd64/focal/grafana-k8s-44
unit-grafana-k8s-0: 14:36:46 INFO juju.downloader download complete ("ch:amd64/focal/grafana-k8s-44")
unit-grafana-k8s-0: 14:36:46 INFO juju.downloader download verified ("ch:amd64/focal/grafana-k8s-44")
unit-grafana-k8s-0: 14:36:53 INFO juju.worker.uniter found queued "upgrade-charm" hook
unit-grafana-k8s-0: 14:36:55 INFO unit.grafana-k8s/0.juju-log Running legacy hooks/upgrade-charm.
unit-grafana-k8s-0: 14:36:57 WARNING unit.grafana-k8s/0.juju-log Could not read configuration from the Grafana workload container: not-found - stat /etc/grafana/grafana-config.ini: no such file or directory
unit-grafana-k8s-0: 14:36:57 INFO unit.grafana-k8s/0.juju-log Updated Grafana's base configuration
unit-grafana-k8s-0: 14:36:57 WARNING unit.grafana-k8s/0.juju-log Could not read configuration from the Grafana workload container: not-found - stat /etc/grafana/provisioning/datasources/datasources.yaml: no such file or directory
unit-grafana-k8s-0: 14:36:57 INFO unit.grafana-k8s/0.juju-log Updated Grafana's datasource configuration
unit-grafana-k8s-0: 14:36:57 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:36:58 INFO unit.grafana-k8s/0.juju-log Restarted grafana-k8s
unit-grafana-k8s-0: 14:36:58 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:36:58 INFO unit.grafana-k8s/0.juju-log Initializing dashboard provisioning path
unit-grafana-k8s-0: 14:36:59 INFO unit.grafana-k8s/0.juju-log Restarted grafana-k8s
unit-grafana-k8s-0: 14:37:00 INFO juju.worker.uniter.operation ran "upgrade-charm" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:37:00 INFO juju.worker.uniter found queued "config-changed" hook
unit-grafana-k8s-0: 14:37:01 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:37:02 INFO unit.grafana-k8s/0.juju-log Restarted replication
unit-grafana-k8s-0: 14:37:02 INFO unit.grafana-k8s/0.juju-log Kubernetes resources for app 'grafana-k8s', container 'grafana' patched successfully: ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:37:03 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:37:03 INFO juju.worker.uniter reboot detected; triggering implicit start hook to notify charm
unit-grafana-k8s-0: 14:37:04 INFO unit.grafana-k8s/0.juju-log Running legacy hooks/start.
unit-grafana-k8s-0: 14:37:06 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:37:07 INFO unit.grafana-k8s/0.juju-log reqs=ResourceRequirements(limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-k8s-0: 14:37:08 INFO juju.worker.uniter.operation ran "grafana-pebble-ready" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:37:10 INFO juju.worker.uniter.operation ran "litestream-pebble-ready" hook (via hook dispatching script: dispatch)
unit-grafana-k8s-0: 14:37:12 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)

Additional context

No response

Update admin password command in README

Enhancement Proposal

There has been a significant rework of the Juju Actions CLI (see here for details). The charm README suggests the following Juju command to get the admin password:

juju run-action grafana-k8s/0 get-admin-password --wait

However, this doesn't work in Juju 3.0+ as the old Actions CLI is deprecated. The correct command is instead

juju run grafana/0 get-admin-password

It would be helpful to add this command to the README for Juju 3.0+ users.

Juju 2.9 users can opt in to the new Actions CLI by setting the feature flag:

export JUJU_FEATURES=actions-v2

So I'm not sure if we want to keep the old command as well, or remove it and advise Juju 2.9 users to set the feature flag.

Multiple errors in documentation and on charmhub.io

Multiple issues with the documentation for the VM charm in charmhub.io

The documentation URL for the VM charm points to the k8 repo. Is this the right place to file bugs also for the VM charm? Should I really file errors about the VM charm in here?
The documentation on how to deploy is for k8 charms. So, I get confused what this charm is about and how to use it.
The documentation states it can relate to charmcraft-prometheus, but it cant. "ERROR: No relations found" (See separate issue: )

Perform proper acceptance tests of Grafana with all of its datasources and dashboard relations, with and without CMRs

Grafana Dashboard: the usage of the `*.tmpl` glob causes dashboards not to show up

Bug Description

In previous versions of the grafana_dashboard library we have relied heavily on Jinja to template the dashboards files. But we want to move away from that model, and reach one in which one can reasonably get a dashboard from the Grafana Dashboards website, add it to the src/grafana_dashboards folder, and see it pop up in Grafana.

When downloading from the Grafana Dashboards website, the file extension if .json. However, the library looks for the .tmpl glob, without logging any warnings or debug messages for files in the folder that it ignored (!!!), sending me down a solid 10 minutes of FUD, in which ofc I first blamed charmcraft, then myself, then charmcraft again, and then I looked at the code, and started blaming the Product Manager that forgot that detail.

The glob is actually documented, but the mismatch with my expectations (I can download from Grafana Dashboards and be happy) and the lack of debug sent me for a spin.

We should:

Relax the glob to try and import all files under src/grafana-dashboard.
Perform basic validation.
Log errors for the ones that are invalid.
[Optional] We may consider exposing an event to the charm about invalid dashboards, but tbh, since we are embedding those files as effectively static resources in the charm, it does not strike me as a valid usecase for the charm to, say, go in Blocked status or something.

To Reproduce

Add this file to a charm with the grafana_dashboard library correctly initialized
charmcraft pack
juju deploy
juju relate with Grafana K8s
Open Grafana
Despair

Environment

Planet Earth, Sol system, Milk way

Relevant log output

_There is no log output and that is one of the problems_

Additional context

No response

Add alert rules, relations and dashboards for self-monitoring

Adapt to the variable names existing in the dashboard

Bug Description

Today, we expect the Grafana dashboards to be uploaded to Grafana via the grafana_dashboard library to use prometheusds and lokids as names for the datasources of type prometheus and loki respectively.

This is not an assumption that works for many dashboards on the Grafana Dashboards website and it introduces unnecessary friction for the charm author and Juju admins wanting to deploy stock Grafana dashboards through the COS Configuration charm.

We should revisit our logic to do the following when importing a dashboard:

Look up the datasource variables in .__inputs (using jq notation)
Rename all entries with type == 'datasource' and pluginId == 'prometheus' to DS_PROMETHEUS
Rename all entries with type == 'datasource' and pluginId == 'loki' to DS_LOKI
Remove duplicates in the datasources [1]
Update all widgets to use the new datasource names :-)

[1] it might be that for some unspecified reason the user might have linked multiple Prometheus datasources in a dashboard but that is not a use-case we should support.

To Reproduce

Try to add this Traefik 2 dashboard to the traefik-k8s-operator charm
Despair

Environment

This plane of reality

Relevant log output

Michele wails in MatterMost about it a lot

Additional context

No response

Log a warning if ingested more than one dashboard with the same id

Enhancement Proposal

Currently, dashboards are silently omitted if the uid is not unique.

We could encounter dashboards with the same id in the following scenarios:

An existing dashboard is modified and exported several times (exported uid and version remain the same).
A dashboard is manually duplicated and modified, forgetting to change the id.
A dashboard is manually duplicated and modified, and id is manually changed to something else, which then in turn collides with another random dashboard.

The SQLite streaming is currently hardcoded to amd64

Note that is is architecture specific, could we do something like (best guess/draft/probably doesn't work):

static-sqlite3:
plugin: dump
source: .
override-pull:
curl -sLo sqlite-static
https://github.com/CompuRoot/static-sqlite3/releases/latest/download/sqlite3-$CRAFT_TARGET_ARCH
chmod +x sqlite-static


> This obviously assumes they actually distribute arm builds, but worth consideration

_Originally posted by @jnsgruk in https://github.com/canonical/grafana-k8s-operator/pull/83#discussion_r942115851_

canonical / grafana-k8s-operator Goto Github PK

grafana-k8s-operator's Introduction

Grafana Operator for k8s

Usage

Web Interface

Integration with other charms/adding external dashboards

To Bundle Dashboards As Part of Your Charm

High Availability Grafana

Relations

OCI Images

Contributing

grafana-k8s-operator's People

Contributors

Stargazers

Watchers

Forkers

grafana-k8s-operator's Issues

Enhancement Proposal

Enhancement Proposal

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Bug Description

To Reproduce

Environment

Relevant log output

Additional context

Enhancement Proposal