我是新的AKS、ACR和DevOps管道,我正在尝试设置CI/CD管道。
我有一个资源组设置,其中既有AKS和ACR。AKS使用的是Standard_B2s
,并且在这一点上只有一个节点,因为我只是随便玩玩。
映像会在提交给master时自动部署到ACR--还没有弄清楚如何设置测试--但是当涉及到部署到AKS时,我总是得到:
##[error]error: deployment "client-deployment" exceeded its progress deadline
我已经更改了我的client.yaml
,以包括一个小时的progressDeadlineSeconds
,因为10,15和20分钟不起作用:
apiVersion: apps/v1
kind: Deployment
metadata:
name: client-deployment
spec:
progressDeadlineSeconds: 3600
replicas: 1
selector:
matchLabels:
component: client
template:
metadata:
labels:
component: client
spec:
containers:
- name: client
image: testappcontainers.azurecr.io/testapp-client
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: client-cluster-ip-service
spec:
type: ClusterIP
selector:
component: client
ports:
- port: 3000
targetPort: 3000
我刚刚修改了Pipelines为我生成的azure-pipelines.yml
,我现在有如下内容:
# Docker
# Build and push an image to Azure Container Registry
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key>
imageRepository: 'testapp'
containerRegistry: 'testappcontainers.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)'
tag: '$(Build.BuildId)'
imagePullSecret: <imagePullSecret_key>
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker@2
displayName: Build and push client image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)-client
dockerfile: $(dockerfilePath)/client/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
- upload: manifests
artifact: manifests
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
jobs:
- deployment: Deploy
displayName: Deploy job
pool:
vmImage: $(vmImageName)
environment: 'testapp.default'
strategy:
runOnce:
deploy:
steps:
- task: KubernetesManifest@0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
- task: KubernetesManifest@0
displayName: Deploy to Kubernetes cluster
inputs:
action: deploy
manifests: |
$(Pipeline.Workspace)/manifests/client.yaml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository):$(tag)
下面是失败的任务的日志:
##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task : Deploy to Kubernetes
Description : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version : 0.162.1
Author : Microsoft Corporation
Help : https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_<token>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers<key>-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=<token>
##[debug]<token> data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=<token>
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]<token> data authorizationType = AzureSubscription
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/
##[debug]<token> auth param serviceAccountCertificate = ***
##[debug]<token> auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]<token> data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] apply
##[debug] -f
##[debug] /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] rollout
##[debug] status
##[debug] Deployment/client-deployment
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] get
##[debug] service/client-cluster-ip-service
##[debug] -o
##[debug] json
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"
},
"creationTimestamp": "name": "client-cluster-ip-service",
"namespace": "default",
"resourceVersion": "1234045",
"selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
"uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
},
"spec": {
"clusterIP": "10.0.181.220",
"ports": [
{
"port": 3000,
"protocol": "TCP",
"targetPort": 3000
}
],
"selector": {
"component": "client"
},
"sessionAffinity": "None",
"type": "ClusterIP"
},
"status": {
"loadBalancer": {}
}
}
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster
然后在Azure CLI中,它显示部署在那里,但没有可用的Pod:
eox-dev@Azure:~$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
client-deployment 0/1 1 0 3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name: client-deployment
Namespace: default
CreationTimestamp: Tue, 31 Dec 2019 15:50:30 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...
Selector: component=client
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: component=client
Containers:
client:
Image: testappcontainers.azurecr.io/testapp-client
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: client-deployment-5688bdc69c (1/1 replicas created)
Events: <none>
我做错了什么?
7条答案
按热度按时间o8x7eapl1#
来自服务器的错误(BadRequest):容器“client”在pod“client-deployment-5688 bdc 69 c-hxlcf”中正在等待启动:尝试提取图像但失败
根据我的经验,这与
imagePullSecrets
和Kubernetes namespace
更相关。在您的
Create imagePullSecret
和Deploy to Kubernetes cluster
任务中,我看到您没有提供task参数的值:namespace
。这将导致创建一个新的命名空间,名称为default
,因为您未指定命名空间。并且,
createSecret
action生成的kubernetes secret是为每个命名空间分离的。一句话,不同的namespace有不同的secret值:Secret存储在给定的命名空间中,并且只能由同一命名空间中的pod访问。
现在,让我们回到构建编译过程。
在你的yml定义中,
Create imagePullSecret
将为新的命名空间default
创建一个secret
,它是由任务自动创建的,因为你没有提供给定的命名空间值。然后,在下一个任务
Deploy to Kubernetes cluster
中,由于同样的原因,这里任务将重新创建一个新的命名空间default
(**注意:**这与上一个不一样)。此外,您可以从日志中看到此进度:此时,从上一个任务生成的
secret
将不可用于当前名称空间。但是,如您所知,ACR
是一个私有容器注册表,我们的系统必须验证kubernetes secret
是否可用。此外,在Deploy to Kubernetes集群任务中,您将仓库指定为
$(imageRepository)
,这与您将镜像推送到**$(imageRepository)-client
**的仓库不相同。这也可以在您的日志中检查:
这就是为什么你的kubernetes中没有可用的节点,你也无法拉取镜像。
为避免此问题,请确保在
KubernetesManifest@0
任务中提供namespace
值。secret to imagePull每个命名空间的Secret
nuypyhwy2#
我面临着类似的问题,即使提供命名空间仍然得到错误“超过其进度截止日期等待部署“XX-service”推出完成:有0个已更新的复制副本可用...”
由于无法提取图像
正在添加“imagePullPolicy:Always”in .yaml修复了我的推出拉问题
kyxcudwk3#
我认为可能导致此问题的原因(特别是在AKS中)是集群在部署之前对容器运行健康检查或活动检查。群集发现应用程序未运行,并由于CrashLoopBackOff而进入等待状态
这可以通过在API应用程序上的“/”上创建一个返回200的默认GET路由来解决。这将表明您的容器是健康的或活的
wwwo4jvm4#
我也遇到过类似的问题,因为异常和就绪探测失败。在解决这些问题后,成功完成了推出。
11dmarpk5#
在我的例子中,这是工人节点的问题。所以我试着从我的pod被分配到的节点中提取图像。我开始知道该节点有一些性能相关的问题,所以我只是添加了一个高规格的新节点,并将Pod分配给该节点。部署也取得了成功。
0wi1tuuw6#
把我的2美分扔在这里因为我们的事
事实证明,这个错误可以显示任何可能阻止k8s集群运行和/或 * 获取 * 给定映像的原因。
我们的问题不在于我们的容器,而是从容器注册表中获取它。
只需在群集上运行以下命令以查看是否存在ImagePullBackoff问题,然后运行另一个命令以获取更多信息:
在此之后,我们可以确认从Azure容器注册表中提取时存在问题。为了防止将来发生这种情况,我们正在利用容器注册表功能,如Zone Redundancy和Georeplication。
zujrkrfu7#
我遇到了这个错误,但在我的情况下,我没有在requirements.txt中包含一些必要的模块以进行正确的部署。对于django,运行
pip freeze -r requirements.txt
也可以使用其他工具来获取所需的导入,例如pipreqs