CPU pressure - Shared resources guide about blueprint HOT 15 CLOSED

currently only people with get access to opf-monitoring can access grafana (i.e. people in the operate-first ocp group), due to the change here -- we should expand these to additional users, (subject to ongoing discussions regarding user policies of course)

from blueprint.

@HumairAK can we extend that to data-science group then? 🙂

from blueprint.

Based on investigation:

• Dashboard: https://grafana-route-opf-monitoring.apps.zero.massopen.cloud/d/GGgK4Q9Mk/tcoufal-testing-total-cpu-usage-vs-request?orgId=1
• CPU request is much higher than any workload we run
• CPU tiers are too generous in JH
• Why does the openshift-storage consume so much
• Completed and Failed pods resource requests are really not considered by the scheduler, despite it appears that deleting them makes the cluster unstuck (further investigation needed)
• There's conflicting metrics:
  • kube_pod_container_resource_limits_cpu_cores shows the Pod's CPU request as it is defined in the resource
  • kube_pod_resource_request shows the effective Pod's resource request that is considered by the scheduler
• Prioritize https://github.com/operate-first/SRE/issues/229

New user-focused dashboards (convert to declarative and provide upstream):

• https://grafana-route-opf-monitoring.apps.zero.massopen.cloud/d/2ZPZ1LrGz/tcoufal-testing-kubeflow-pipeline-info?orgId=1
• https://grafana-route-opf-monitoring.apps.zero.massopen.cloud/d/fuJBFErMz/tcoufal-testing-jh-pod-usage?orgId=1

from blueprint.

@tumido I get a 403 permission denied on these dashboards as i try to login through moc-sso. Can you please provide access?

from blueprint.

thanks! if these are the user oriented dashboards which @tumido presented, we(data science users) would very much benefit from having access to these 🙂

from blueprint.

yes, these are the user oriented dashboards (the two linked at the end) - we'll convert them to permanent dashboards soon. I still have some fiddling to do. Right now, they are just prototypes, stored in the Grafana runtime..

from blueprint.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from blueprint.

/remove-lifecycle stale

from blueprint.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from blueprint.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

from blueprint.

/remove-lifecycle rotten

from blueprint.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from blueprint.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

from blueprint.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

from blueprint.

@sesheta: Closing this issue.

from blueprint.