kubernetes 我在k8s集群上遇到了大量的CPU限制--我是否有高限制低配额的Linux内核错误?

cmssoen2  于 2022-12-11  发布在  Kubernetes
关注(0)|答案(3)|浏览(122)

我在Kubernetes集群(k8s v1.18.12,运行4.9.0-11-amd 64 #1 SMP Debian 4.9.189-3+ deb 9 u2(2019-11-11)x86_64 GNU/Linux)中遇到了很多CPU节流问题(参见下面的nginx图表,其他pod通常为25%到50%)。
由于backports的原因,我不知道我的集群是否包含https://lkml.org/lkml/2019/5/17/581中描述的Linux内核bug。我该如何查找?有简单的方法来检查或测量吗?
如果我有错误,修复的最佳方法是什么?或者我应该采取其他缓解措施,例如不使用CFS配额(--cpu-cfs-quota=false或无CPU限制)或减少cfs_period_uscfs_quota_us
nginx的CPU节流百分比(水平缩放约15:00,移除约19:30的CPU限制):

ee7vknir

ee7vknir1#

由于修复程序被后移植到许多旧的内核版本,我不知道如何轻松地查找是否有修复程序,例如4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux
但是,您可以测量CFS是工作顺畅还是限制过多,如https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1中所述:
1.您可以运行给定的cfs.go,并为其睡眠和迭代设置合适的设置,以及CFS设置,例如docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
1.检查是否所有burn都花费了5ms。如果不是,则CFS限制过多。这可能是由于原始错误198197(请参阅https://bugzilla.kernel.org/show_bug.cgi?id=198197)或错误198197修复程序引入的回归(详细信息请参阅https://lkml.org/lkml/2019/5/17/581)。
https://github.com/kubernetes/kops/issues/8954中也采用了这种测量方法,表明Linux内核4.9.0-11-amd64的节流太多(然而,使用比Debian 4.9.189-3+deb9u2 (2019-11-11)更早的Debian 4.9.189-3+deb9u1 (2019-09-20))。

kxeu7u2r

kxeu7u2r2#

Recently, I'm working on debuging the cpu throttling issue, with the following 5 tests, I've tested out the bug in kernel (Linux version 4.18.0-041800rc4-generic)
This test case is intended to hit 100% throttling for the test 5000ms / 100 ms periods = 50 periods. A kernel without this bug should be able to have a CPU usage stats about 500ms.
Maybe you can try these tests to check whether your kernel will be throttlled.
[Multi Thread Test 1]

./runfibtest 1; ./runfibtest

From <https://github.com/indeedeng/fibtest>

[Result]

Throttled

./runfibtest 1
Iterations Completed(M): 465 
Throttled for: 52 
CPU Usage (msecs) = 508

./runfibtest 8
Iterations Completed(M): 283 
Throttled for: 52 
CPU Usage (msecs) = 327

[Multi Thread Test 2]

docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/fibtest:latest /bin/sh -c "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"

[Result]

Throttled

Iterations Completed(M): 192 
Throttled for: 53 
CPU Usage (msecs) = 227
nr_periods 58
nr_throttled 56
throttled_time 10136239267
267434463
209397643 2871651 8044402 4226146 5891926 5532789 27939741 4104364

[Multi Thread Test 3]

docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/stress-ng:cpu-delay /bin/sh -c "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"

Result

Throttled

nr_periods 56
nr_throttled 53
throttled_time 7893876370
379589091
330166879 3073914 6581265 724144 706787 5605273 29455102 3849694

For the following kubernetes test, we can use "kubectl logs pod-name" to get the result once the job is done
[Multi Thread Test 4]

apiVersion: batch/v1
kind: Job
metadata:
  name: fibtest
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: fibtest
        image: hipeteryang/fibtest
        command: ["/bin/bash", "-c", "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"]
        resources:
          requests:
            cpu: "50m"
          limits:
            cpu: "100m"
      restartPolicy: Never

Result

Throttled

Iterations Completed(M): 195 
Throttled for: 52 
CPU Usage (msecs) = 230
nr_periods 56
nr_throttled 54
throttled_time 9667667360
255738571
213621103 4038989 2814638 15869649 4435168 5459186 4549675 5437010

[Multi Thread Test 5]

apiVersion: batch/v1
kind: Job
metadata:
  name: stress-ng-test
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: stress-ng-test
        image: hipeteryang/stress-ng:cpu-delay
        command: ["/bin/bash", "-c", "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu
"]
        resources:
          requests:
            cpu: "50m"
          limits:
            cpu: "100m"
      restartPolicy: Never

Result

Throttled

nr_periods 53
nr_throttled 50
throttled_time 6827221694
417331622
381601924 1267814 8332150 3933171 13654620 184120 6376208 2623172
fnatzsnv

fnatzsnv3#

CFS错误已在Linux 5.4中修复,执行kubectl describe nodes | grep Kernel或转到任何Kubernetes节点,执行uname -sr将告诉您正在运行的内核版本。

相关问题