Many of the dashboard widgets with error information in this configuration are using a hardcoded "errors" caption in the charts, like so (from the Observatorium API dashboard):

This hides the details about which errors is this. It would be greatly useful if we could see the error code or any other error attribute that helps identifying it:

This happens in Observatorium's and Thanos' dashboards, across gRPC and HTTP errors charts.

We should add a proper runbook for the ThanosReceiveTrafficBelowThreshold alert that got added after updating thanos-mixin:

https://github.com/rhobs/configuration/pull/104/files#diff-a67300a31eb947d3669517e20fa9bd33fb3f3139141797c864b072d2df1eac67R365

https://github.com/rhobs/configuration/blob/main/docs/sop/observatorium.md#thanosreceivetrafficbelowthreshold

We have an incident where users were not sending cluster IDs so the series was not unique across clusters. I think it would be useful for everyone if we would validate some of it on runtime? 🤔

E.g if you have _id label name in one of labels as per: https://rhobs-handbook.netlify.app/services/rhobs/#metric-label-restrictions

It isn't possible to configure the trace namespace without regenerating the templates.

This means that oc process can't alter the namespaces. They aren't configurable.

resources/services/observatorium-traces-template.yaml has { name: 'OBSERVATORIUM_TRACES_NAMESPACE', value: 'observatorium-traces' }, in the Jsonnet.

It gets substituted in the correct places by resources/services/observatorium-template.yaml

- --experimental.traces.read.endpoint-template=http://observatorium-jaeger-{tenant}-query.${OBSERVATORIUM_TRACES_NAMESPACE}.svc.cluster.local:16686/
...
--experimental.traces.read.endpoint-template=http://observatorium-jaeger-{tenant}-query.${OBSERVATORIUM_TRACES_NAMESPACE}.svc.cluster.local:16686/

The problem is that the substitution happens when the Templates are being made. Rather than being something that is fixed during template generation, the traces namespace SHOULD become an OpenShift template parameter similar to NAMESPACE, OPENTELEMETRY_OPERATOR_NAMESPACE or JAEGER_OPERATOR_NAMESPACE.

cc @pavolloffay

The opa-ams sidecar of the dev/test Observatorium API is sending error spans to the internal Jaeger. The errors are 404s when doing GET :8082/

I suspect the reason is that the sidecar is started with --web.healthchecks.url=http://127.0.0.1:8082, but that URL 404s.
See https://github.com/rhobs/configuration/blob/main/resources/services/observatorium-template.yaml#L296

I don't know what the correct health check URL is. Note that OPA AMS is started with --ams.url=${OCM_BASE_URL}
(This is not overridden anywhere on dev/test, as can be verified with kubectl -n observatorium-testing exec deployment/observatorium-observatorium-api -c opa-ams -- ps -ef.)

Mention lack of referring feature for Alerts and Rules.

Some issues observed while deploying RHOBS on OCP cluster using launch script

1. Deployment script under the test folder gets failed while adding rules This happened when deploying observatorium-template.yaml.
2. Older image of observatorium-api is used in observatorium-template.yaml due to which observatorium-api remains in CrashLoopBackOff state.
3. Older image of Thanos is used in observatorium-metrics-template.yaml due to which Thanos remains in CrashLoopBackOff state.
4. The PVC which were supposed to be created in observatorium-metrics-template.yaml were all in pending state due to the gp2 storage class not available on the OCP cluster.

W0309 12:40:53.415964 4005930 shim_kubectl.go:55] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "template.openshift.io/v1" for your resource

Currently we annotate all alerts with dashboard URL in format https://grafana.app-sre.devshift.net/d/<dashboard_ID>/* - however, for alerts which do not have a dashboard we are returning link to non-existing dashboard URL https://grafana.app-sre.devshift.net/d/no-dashboard/*. We should not provide dashboard annotation if there is none.

Idea is that we want to see loaded rules. We have to also ensure this will be filtered by tenant.

Makefile commands that worked in the past such as XARGS=gxargs SED=gsed make all now fail:

(re)installing /Users/snible/go/bin/jsonnetfmt-v0.18.0
# golang.org/x/sys/unix
../../../go/pkg/mod/golang.org/x/[email protected]/unix/syscall_darwin.1_13.go:25:3: //go:linkname must refer to declared function or variable
../../../go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.1_13.go:27:3: //go:linkname must refer to declared function or variable

See See https://stackoverflow.com/questions/71507321/go-1-18-build-error-on-mac-unix-syscall-darwin-1-13-go253-golinkname-mus for discussion. The solution suggested, go get -u golang.org/x/sys, isn't quite accurate for our situation.

To reproduce without our Makefile, I did

cd .bingo
GOBIN=$(go env GOPATH)/bin
go build -mod=mod -modfile=jsonnet.mod -o=${GOBIN}/jsonnet-v0.18.0 "github.com/google/go-jsonnet/cmd/jsonnet"

This fails, but if I then do go get -modfile=jsonnet.mod -u golang.org/x/sys and repeat the go build it succeeds.

I am working on a PR; expect it soon.

I was trying to update some manifests lately and encountered two different set of issues:

Need fixing

ARM64 related issues with Golang based tools

Mostly caused by the jsonnet tools that have to be updated to be arm64 compatible. This is easy to solve.

Need updated documentation

GNU vs BSD tools

It's not documented that macOS users have to install gnu-sed and findutils (for GNU xargs) and update the Makefile to make use of them.

Jsonnet dependencies update

The instructions provided at the moment do not work, because make vendor doesn't exist anymore.