[Bug] Go panic when syncing many dashboards about grafana-operator HOT 4 CLOSED

That's certainly something interesting to investigate, thanks for reporting!

from grafana-operator.

The issue could be the error handling here : if the error is GetDashboardByUIDNotFound, we don't return, but instead continue to run resp.GetPayload().

This happens in onDashboardDeleted, so if the dashboard isn't found, we can return without an error here.

from grafana-operator.

@smuda Well, it's hard to say without seeing it in a lab, but it could potentially be in at least two cases:

• dashboard uid got updated, then you should see a log message dashboard uid got updated, deleting dashboards with the old uid;
• maybe some delays in synchronization of etcd (=stale reads) if you have multiple replicas. When the dashboard controller receives an event, it tries to fetch a CR from API-server, and if API-server replies that the CR does not exist, onDashboardDeleted gets called to clean up the dashboard from Grafana instances. etcd has a concept of quorum read, and I'm not sure if API-server relies on it when interacting with etcd.

from grafana-operator.

Probably a stupid question, but as this occurred during the first population of the dashboards to Grafana, why is onDashboardDeleted being called?

from grafana-operator.