Eviction manager: When stats calls failed, we evict the pod that we couldn't get stats for. about kubernetes HOT 10 OPEN

/sig node
?

from kubernetes.

/kind failing-test

from kubernetes.

/triage accepted

This seems to be one of the causes for eviction test flakiness.

I've seen in mostly in disk stats.

from kubernetes.

Some ideas could be to use QoS in case of stats failure. We could say that BestEffort gets evicted before Burstable and Burstable before Guaranteed. This could help in the case of stats failure so we can control the eviction a bit better.

An interesting thing about this is this doesn't help disk based eviction as ephemeral-storage is never considered guaranteed (guaraneteed is only for cpu/memory).

from kubernetes.

Pid stats is another area where we are seeing failures but that is related to all pid stats seemed to be returning 0 so any stats gathering is not working.

from kubernetes.

cc @pacoxu

from kubernetes.

/priority important-longterm

from kubernetes.

Yeah overloading the compare prioritization with error checking appears on face to be problematic. I'd try to split that out into two separate funcs one for get stats another for the compare. The inability to get stats for pod a may be due to a block of the resource by pod b that will be ignored because of the failure it caused?

from kubernetes.

Brought this up to sig-node.

We should look into ways to populate cache more aggressively. Qos could be an option but it is worth investigating why stats gathering fails.

from kubernetes.

/remove-kind failing-test

from kubernetes.