kubelet unbalanced affinity pod in different numa node about kubernetes HOT 11 OPEN

sig/k8s-infra

from kubernetes.

/sig node

from kubernetes.

would the cpumanager policy option distribute-cpus-across-numa help here? https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy-options

from kubernetes.

@ffromani thanks a lot. my understanding is distribute-cpus-accors-numa is used to affinity cpu to different numa for one pod/container. for example: if one pod/container needs 16 cpus, w/o distribute-cpus-accross-numa, 16cpus will firstly be affinity in numa0 and numa1, then it maybe 14cpus in numa0 if there is enough cpus in numa0 and 2 cpus in numa1. w/ distribute-cpus-accorss-numa, 8 cpus are affinity in numa0 although there is enough cpus availabe in numa0 but another 8 cpus are affinity into numa1. then cpus in pod/conainer are distributed in numa0 and numa1.

but this issue described in this ticket is; if there are 8 pods. those 8 pods will be affinity into numa0 firstly if there is enough cpu available in nuam1. then then numa1, that will result in different cpu load in different numa nodes and then different cpu frequency when turboboost enabled.

from kubernetes.

but this issue described in this ticket is; if there are 8 pods. those 8 pods will be affinity into numa0 firstly if there is enough cpu available in nuam1. then then numa1, that will result in different cpu load in different numa nodes and then different cpu frequency when turboboost enabled.

right, so the ask is about distributing the pods (containers) across NUMA nodes more evenly it seems. This is indeed not yet covered by options or combinations thereof.

from kubernetes.

/triage accepted

from kubernetes.

right, so the ask is about distributing the pods (containers) across NUMA nodes more evenly it seems. This is indeed not yet covered by options or combinations thereof.

yes @ffromani . frankly cpuManagePolicyOption/distribute-cpus-accors-numa is conflict with topoloygManagerPolicy/single-numa-node. but single-numa-node can provide more high performance.

w/o evenly distribute pod/container into different numa, cpu frequency will be different when turboboost enabled and then result in different cpu usage although incoming traffic is same.
I think evenly distribute pod/container into different numa can overall increase performance.

So will you create a feature to support it?

Thanks a lot

from kubernetes.

So will you create a feature to support it?

Unfortunately I don't have capacity atm. But I agree this is a gap (triage accepted)
Addressing this gap will require an enancement proposal, design etc. etc. It's 1.32 material in the best case IMO.

from kubernetes.

So will you create a feature to support it?

Unfortunately I don't have capacity atm. But I agree this is a gap (triage accepted) Addressing this gap will require an enancement proposal, design etc. etc. It's 1.32 material in the best case IMO.

Thanks a lot @ffromani. we agreed it is gap and triage accepted then there will be followup by someone since you have no capacity atm. correct?

from kubernetes.

Thanks a lot @ffromani. we agreed it is gap and triage accepted then there will be followup by someone since you have no capacity atm. correct?

Not automatically. Someone has to volunteer pick this work up. If you wish to push forward, by all means do it!

from kubernetes.

/cc

from kubernetes.