kubernetes K8s水平pod自动缩放不工作

我正在按照this tutorial来试用k8s水平吊舱自动缩放。
我有以下K8的载货单：

apiVersion: v1 
kind: Service 
metadata: 
  name: springboot-k8s-svc
spec:
  selector:
    app: spring-boot-k8s
  ports:
    - protocol: "TCP"
      port: 8080 
      targetPort: 8080 
  type: NodePort 
---
apiVersion: apps/v1
kind: Deployment 
metadata:
  name: spring-boot-k8s
spec:
  selector:
    matchLabels:
      app: spring-boot-k8s
  replicas: 1 
  template:
    metadata:
      labels:
        app: spring-boot-k8s
    spec:
      containers:
        - name: spring-boot-k8s
          image: springboot-k8s-example:1.0 
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

目前我的minikube正在运行以下内容：

$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   4h52m

我启动了我的虚拟 Spring Boot 应用程序：

$ kubectl apply -f deployment-n-svc.yaml 
service/springboot-k8s-svc created
deployment.apps/spring-boot-k8s created

此应用程序似乎按需适当启动：

$ kubectl get all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/spring-boot-k8s-bccc4c557-7wbrn   1/1     Running   0          5s

NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/kubernetes           ClusterIP   10.96.0.1      <none>        443/TCP          4h53m
service/springboot-k8s-svc   NodePort    10.99.136.27   <none>        8080:30931/TCP   5s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/spring-boot-k8s   1/1     1            1           5s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/spring-boot-k8s-bccc4c557   1         1         1       5s

当我尝试到达REST端点时，我得到了所需的输出：

$ curl http://192.168.49.2:30931/message
OK!

现在我尝试自动缩放应用程序：

$ kubectl autoscale deployment spring-boot-k8s --min=1 --max=5 --cpu-percent=10
horizontalpodautoscaler.autoscaling/spring-boot-k8s autoscaled

开始观看hpa刚刚启动如下所示的命令。它似乎已经开始：

$ watch -n 1 kubectl get hpa

Every 1.0s: kubectl get hpa

NAME                                                  REFERENCE                    TARGETS         MINPODS   MAXPODS   REPLIC
AS   AGE
horizontalpodautoscaler.autoscaling/spring-boot-k8s   Deployment/spring-boot-k8s   <unknown>/10%   1         5         1
     8m10s

然后，我尝试使用apache bench HTTP load test utility在spring Boot 服务器上创建负载，以检查k8s是否增加了pod的数量：

$ab -n 1000000 -c 100 http://192.168.49.2:30931/message

然而，这并没有增加豆荚的数量。我错过了什么？

附言：

当我用Ctrl+C杀死ab命令时，它会给出以下输出（注意每个请求大约需要5s的处理时间）：

$ ab -n 1000000 -c 100 http://192.168.49.2:32215/message
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.49.2 (be patient)
^C

Server Software:        
Server Hostname:        192.168.49.2
Server Port:            32215

Document Path:          /message
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   35.650 seconds
Complete requests:      601
Failed requests:        0
Total transferred:      81736 bytes
HTML transferred:       2404 bytes
Requests per second:    16.86 [#/sec] (mean)
Time per request:       5931.751 [ms] (mean)
Time per request:       59.318 [ms] (mean, across all concurrent requests)
Transfer rate:          2.24 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.5      0       7
Processing:  5001 5004   4.7   5003    5022
Waiting:     5000 5004   3.9   5002    5019
Total:       5001 5006   5.4   5003    5024

Percentage of the requests served within a certain time (ms)
  50%   5003
  66%   5005
  75%   5007
  80%   5009
  90%   5013
  95%   5020
  98%   5023
  99%   5023
 100%   5024 (longest request)

更新

正如注解中所问，以下是一些其他命令的输出：

$ kubectl describe hpa spring-boot-k8s
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                  spring-boot-k8s
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 03 Feb 2023 01:58:06 +0530
Reference:                                             Deployment/spring-boot-k8s
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 10%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                   Age                  From                       Message
  ----     ------                   ----                 ----                       -------
  Warning  FailedGetResourceMetric  35m (x500 over 16h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

注意上面写的：autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler .
它还说：no metrics returned from resource metrics API。我认为指标服务器正在运行：

$ kubectl get deployment -n kube-system
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
coredns          1/1     1            1           304d
metrics-server   1/1     1            1           40h

这似乎是它不起作用的原因。但可能是什么原因呢？
除此之外，CPU利用率也没有增加太多，直到昨天晚上，我还在watch kubectl top node命令的输出中看到最大3%的CPU利用率：

NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   141m         1%     1127Mi          7%

但现在显示以下错误：

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

当我运行kubectl get pod时，它给出了以下输出：

$ kubectl get pod
NAME                               READY   STATUS    RESTARTS      AGE
spring-boot-k8s-556b578645-pwdhs   1/1     Running   3 (49m ago)   31d

但是，即使用不同的方法指定了这个pod的名称，我也无法得到它的利用率。

$ kubectl top pod pod/spring-boot-k8s -n efault --containers
error: invalid resource name "pod/spring-boot-k8s": [may not contain '/']

$ kubectl top pod spring-boot-k8s -n efault --containers
Error from server (NotFound): pod "spring-boot-k8s" not found

$ kubectl top pod spring-boot-k8s-556b578645-pwdhs -n efault --containers
Error from server (NotFound): pod "spring-boot-k8s-556b578645-pwdhs" not found

$ kubectl top pod pod/spring-boot-k8s-556b578645-pwdhs -n efault --containers
error: invalid resource name "pod/spring-boot-k8s-556b578645-pwdhs": [may not contain '/']

我的指标服务器已在运行：

$ kubectl get all -n kube-system | grep metric
pod/metrics-server-6b76bd68b6-f68f4    1/1     Running   68 (54m ago)   38d
service/metrics-server   ClusterIP   10.103.98.14   <none>        443/TCP                  38d
deployment.apps/metrics-server   1/1     1            1           38d
replicaset.apps/metrics-server-6b76bd68b6   1         1         1       38d

此外，我还可以看到kubectl top node的节点利用率：

$ kubectl top node
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
minikube   132m         1%     734Mi           4%

但我无法看到Pod的CPU利用率：

$ kubectl top pod
error: Metrics not available for pod default/spring-boot-k8s-556b578645-pwdhs, age: 747h9m12.416340419s

结果发现，here和here中存在一个bug。
简单的解决方案是使用内务处理间隔参数停止和启动minikube：

$ minikube stop
$ minikube start --extra-config=kubelet.housekeeping-interval=10s

在此之后，我能够得到pod CPU利用率：

$ kubectl top pod
NAME                               CPU(cores)   MEMORY(bytes)   
spring-boot-k8s-556b578645-pwdhs   2m           97Mi

这也使得k8s hpa能够在负载增加时自动调整pod。需要注意的是，可能需要一些时间才能在kubectl get pod命令输出中真正反映pod副本数量的增加。

kubernetes K8s水平pod自动缩放不工作

1条答案

相关问题

热门标签

最新问答