kube-controller-manager Master Election Time Exceeds the Lease Time about kubernetes HOT 7 OPEN

Based on the code inference, we infer that the following situation occurs:
1、The lease queried before the controller-manager request of the master1 node times out is the penultimate renewal of the controller-manager before the master2 node is powered off, not the last renewal.
2、After controller-manager of master2 updates the lease, controller-manager of master1 fails to obtain the latest lease due to the persistent connection of http2.
3、When the http2 connection is disconnected and reconnected, the latest lease information is obtained.
4、However, the lease in the memory is not the latest lease. Therefore, le.observedTime is updated to the current time based on the code.

from kubernetes.

/sig apps

from kubernetes.

/triage accepted

from kubernetes.

If update observedTime to renewTime, does that fix the problem?

observedTime is necessary .Utilizing observedTime contributes to ensuring consistency, robustness, and correctness in the election process.

from kubernetes.

The log entries you've shared indicate a situation where master1, already recognized as the leader, is inexplicably initiating another leadership acquisition attempt.

Consider testing version 1.30 to check if the problem remains unresolved. The following code should alleviate your issue to some degree:

from kubernetes.

The log entries you've shared indicate a situation where master1, already recognized as the leader, is inexplicably initiating another leadership acquisition attempt.

Consider testing version 1.30 to check if the problem remains unresolved. The following code should alleviate your issue to some degree:

kubernetes/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go

Lines 319 to 342 in 85fad5b

	func (le *LeaderElector) tryAcquireOrRenew(ctx context.Context) bool {
	now := metav1.NewTime(le.clock.Now())
	leaderElectionRecord := rl.LeaderElectionRecord{
	HolderIdentity: le.config.Lock.Identity(),
	LeaseDurationSeconds: int(le.config.LeaseDuration / time.Second),
	RenewTime: now,
	AcquireTime: now,
	}
	// 1. fast path for the leader to update optimistically assuming that the record observed
	// last time is the current version.
	if le.IsLeader() && le.isLeaseValid(now.Time) {
	oldObservedRecord := le.getObservedRecord()
	leaderElectionRecord.AcquireTime = oldObservedRecord.AcquireTime
	leaderElectionRecord.LeaderTransitions = oldObservedRecord.LeaderTransitions
	err := le.config.Lock.Update(ctx, leaderElectionRecord)
	if err == nil {
	le.setObservedRecord(&leaderElectionRecord)
	return true
	}
	klog.Errorf("Failed to update lock optimitically: %v, falling back to slow path", err)
	}

I think you may have missed my point. What I want to know is why observedTime is used instead of renewTime to determine if the lease has expired.

from kubernetes.

I think you may have missed my point. What I want to know is why observedTime is used instead of renewTime to determine if the lease has expired.

No, I know what you mean. As I said above, you can't do it this way.

from kubernetes.