[BUG] Cannot drop metric container about rancher HOT 4 CLOSED

I tried this on rancher v2.8.4. After creating an RKE2 downstream cluster, I installed monitoring on it and added the following to values.yaml before creation:

cAdvisorMetricRelabelings:
    - action: drop
      regex: container_blkio_device_usage_total
      sourceLabels:
      - __name__

• After successful installation, I clicked on "Prometheus Rules" in the following page and searched for the metric.
  

• The container_blkio_device_usage_total doesn't show up, as can be seen below:
  

• After removing the aforementioned "drop" rule, I do see the container_blkio_device_usage_total in suggestions:
  

@joshmeranda @alexandreLamarre can one of you validate if I'm looking at this correctly?

from rancher.

drop'ing the metric again still shows it when trying to query. But the value isn't increasing (as mentioned in the original issue).

from rancher.

Looks like the cAdvisor relabellings only apply to the kubelet service monitor, which is disabled in k3s. So this issue is likely specific to other distributions, like RKE2.

from rancher.

Works as expected in 103.1.1+up45.31.1

with:

kubelet:
  serviceMonitor:
    cAdvisorMetricRelabelings:
      // ....
      - action: drop
      regex: container_blkio_device_usage_total
      sourceLabels:
      - __name__

from rancher.