Add optional ServiceMonitor about dask-kubernetes HOT 4 CLOSED

Right now we propagate labels from the top-level DaskCluster and DaskWorkerGroup objects down to their child resources. So if you set foo=bar on the DaskCluster then you could use the foo=bar,dask.org/component=scheduler selector to select the scheduler service.

from dask-kubernetes.

I quickly threw together a ServiceMonitor that will scrape metrics for all Dask schedulers deployed with the operator.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dask-operator-scheduler
  labels:
    dask.org/component: "scheduler"
spec:
  selector:
    matchLabels:
      dask.org/component: "scheduler"
  targetLabels:
    - dask.org/cluster-name
  endpoints:
  - interval: 15s
    port: http-dashboard

I think it would make sense to optionally include this in the Operator Helm Chart with all the same templating and customisation as is available in the basic helm chart. This is a slightly different approach because one ServiceMonitor would be registered which would select all Dask clusters created by the operator. Compared to the Helm Chart which creates one ServiceMonitor per cluster.

I also had a play around with a PodMonitor for scraping worker metrics but the operator currently doesn't configure worker ports by default so that will take more work.

from dask-kubernetes.

@jacobtomlinson Wow., thank you! I already forked the repo and wanted to create PR for that feature. It would be also nice to have specs customizable in make_scheduler_spec / make_worker_spec to include some extra matadata labels when creating KubeCluster instance.

from dask-kubernetes.

Ok now that #688 is merged we can also create a PodMonitor to scrape worker metrics.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: dask-operator-workers
  labels:
    dask.org/component: "worker"
spec:
  selector:
    matchLabels:
      dask.org/component: "worker"
  podTargetLabels:
    - dask.org/cluster-name
    - dask.org/workergroup-name
  podMetricsEndpoints:
  - interval: 15s
    port: http-dashboard

from dask-kubernetes.