发生了什么？
1、我们关闭了运行pods的worker节点
2、一个pod保持在Running状态，再也没有改变过

我们期望会发生什么？

我们期望pod会被更新为Terminating。

我们如何尽可能精确地重现它？

由于我们找到了“唯一可能的原因”，其概率极低，因此不太可能重现。

我们需要了解其他信息吗？

pod定义如下，但似乎没有问题：

tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 2
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 2

在controller-manager的日志中发现了奇怪的日志序列

I0604 15:28:49.239218       1 timed_workers.go:113] Adding TimedWorkerQueue item dpa/infrastructure-0 at 2024-06-04 15:28:49.239118259 +0800 CST m=+252387.993120493 to be fired at 2024-06-04 15:28:51.239118259 +0800 CST m=+252389.993120493
......
I0604 15:28:51.240319       1 taint_manager.go:106] "NoExecuteTaintManager is deleting pod" pod="dpa/infrastructure-0"
I0604 15:28:51.259631       1 taint_manager.go:416] "Noticed pod deletion" pod="dpa/infrastructure-0"
I0604 15:28:51.259642       1 timed_workers.go:132] Cancelling TimedWorkerQueue item dpa/infrastructure-0 at 2024-06-04 15:28:51.259639403 +0800 CST m=+252390.013641637
......
I0604 17:38:00.725378       1 timed_workers.go:113] Adding TimedWorkerQueue item dpa/infrastructure-0 at 2024-06-04 17:38:00.725348125 +0800 CST m=+260139.479350359 to be fired at 2024-06-04 17:38:02.725348125 +0800 CST m=+260141.479350359
W0604 17:38:00.725393       1 timed_workers.go:118] Trying to add already existing work for &{NamespacedName:dpa/infrastructure-0}. Skipping.
......

我们注意到大约在15:28左右重启了一次节点，这触发了预期的pod dpa/infrastructure-0的驱逐。但是当我们再次在大约17:38关闭节点时，污点管理器由于在workersMap中找到了相同的键而跳过了工作。
控制器管理器从未重启或更改领导者。
日志很荒谬，但在查看代码后，我们确实找到了一种方法，即在取消后保留键在workersMap中：

func (q *TimedWorkerQueue) getWrappedWorkerFunc(key string) func(ctx context.Context, args *WorkArgs) error {
	return func(ctx context.Context, args *WorkArgs) error {
		err := q.workFunc(ctx, args)
		q.Lock()
		defer q.Unlock()
		if err == nil {
			// To avoid duplicated calls we keep the key in the queue, to prevent
			// subsequent additions.
			q.workers[key] = nil
		} else {
			delete(q.workers, key)
		}
		return err
	}
}

在此部分的代码生成了一个WorkerFunc,将在工作中调用，污点管理器将运行q.workFunc以从apiserver中删除pod,这将触发podUpdate并导致CancelWork:

func (q *TimedWorkerQueue) CancelWork(key string) bool {
	q.Lock()
	defer q.Unlock()
	worker, found := q.workers[key]
	result := false
	if found {
		klog.V(4).Infof("Cancelling TimedWorkerQueue item %v at %v", key, time.Now())
		if worker != nil {
			result = true
			worker.Cancel()
		}
		delete(q.workers, key)
	}
	return result
}

如果且仅当CancelWork在WorkerFunc之前获取了锁，那么q.workers[key]将不会像预期的那样被删除，而是q.workers[key] = nil被删除，导致后续工作的跳过。

Kubernetes版本

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.3+77d8f59014e1f3-dirty", GitCommit:"77d8f59014e1f3ca907c1b4a4f57900539b88dc8", GitTreeState:"dirty", BuildDate:"2024-04-29T13:26:15Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.3+77d8f59014e1f3-dirty", GitCommit:"77d8f59014e1f3ca907c1b4a4f57900539b88dc8", GitTreeState:"dirty", BuildDate:"2024-04-29T13:23:44Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/arm64"}

云提供商

无

OS版本

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

安装工具

容器运行时(CRI)和版本(如适用)

相关插件(CNI,CSI等)和版本(如适用)