**前言:**这发生在kubernetes 1.17.2中。
大约一个月前,我通过以下方式续订了所有证书(因为它们将在25天后到期)
'sudo kubeadm alpha证书全部续订'
然后我将更新后的/etc/kuberenetes.admin.conf复制到$HOME/.kube/config。
25天内一切正常,然后证书到期,大多数事情继续正常工作。我可以创建,应用,删除,获取日志/描述pod/节点,引流节点等。
但我再也不能
kubectl执行-it bash
在浏览了S/O所提供的内容后,我确定了一些不是问题所在的东西:
1.所有工作节点上的Kubelet客户端证书正确且未过期。
1.控制平面服务已全部重新启动并且运行正常(API服务器、控制器管理器和调度程序)
1.所有工作机上的计算机都和主机上的一致。
我还重新启动了所有的kubelet服务,但没有效果。
使用-v=3标志运行exec会出现一系列错误,例如写入$HOME/.kube/cache/discovery/(ip and port)/things的权限被拒绝。
我检查了一下,$HOME/.kube/cache/discovery下的所有内容都归root所有,这让我很惊讶,但我不确定会发生什么。
我很肯定这与证书过期有关,但据我所知,没有引用的证书过期。
以下是exec -it -v命令的输出:
I0926 09:10:38.207383 143977 cached_discovery.go:130] failed to write cache to /home/username/.kube/cache/discovery/ip_port/servergroups.json due to open /home/username/.kube/cache/discovery/ip_port/servergroups.json.951747418: permission denied
I0926 09:10:38.218182 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1beta1/serverresources.json.358037648: permission denied
I0926 09:10:38.223832 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apps/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apps/v1/serverresources.json.233202372: permission denied
I0926 09:10:38.270628 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1/serverresources.json.491603985: permission denied
I0926 09:10:38.286187 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/v1/serverresources.json.246617561: permission denied
I0926 09:10:38.288167 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1beta1/serverresources.json.050273651: permission denied
I0926 09:10:38.290221 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1beta1/serverresources.json.663074870: permission denied
I0926 09:10:38.292567 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1/serverresources.json.861438680: permission denied
I0926 09:10:38.294598 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta2/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta2/serverresources.json.153396170: permission denied
I0926 09:10:38.296621 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/extensions/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/extensions/v1beta1/serverresources.json.374028172: permission denied
I0926 09:10:38.298626 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/certificates.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/certificates.k8s.io/v1beta1/serverresources.json.055142558: permission denied
I0926 09:10:38.302635 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1/serverresources.json.349584095: permission denied
I0926 09:10:38.304796 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1/serverresources.json.828927593: permission denied
I0926 09:10:38.306933 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/events.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/events.k8s.io/v1beta1/serverresources.json.749609603: permission denied
I0926 09:10:38.309070 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/batch/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/batch/v1beta1/serverresources.json.891187629: permission denied
I0926 09:10:38.311296 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1beta1/serverresources.json.095032423: permission denied
I0926 09:10:38.313529 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1beta1/serverresources.json.681854769: permission denied
I0926 09:10:38.315781 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/policy/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/policy/v1beta1/serverresources.json.623984267: permission denied
I0926 09:10:38.317914 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v1/serverresources.json.105061109: permission denied
I0926 09:10:38.319951 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/batch/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/batch/v1/serverresources.json.877278959: permission denied
I0926 09:10:38.321926 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/node.k8s.io/v1beta1/serverresources.json due to mkdir /home/username/.kube/cache/discovery/ip_port/node.k8s.io/v1beta1: permission denied
I0926 09:10:38.324019 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1/serverresources.json.303121412: permission denied
I0926 09:10:38.325963 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1/serverresources.json.187370454: permission denied
I0926 09:10:38.328049 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1beta1/serverresources.json.531871096: permission denied
I0926 09:10:38.330015 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1/serverresources.json.215681985: permission denied
I0926 09:10:38.332101 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta1/serverresources.json.552182828: permission denied
I0926 09:10:38.334021 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/discovery.k8s.io/v1beta1/serverresources.json due to mkdir /home/username/.kube/cache/discovery/ip_port/discovery.k8s.io/v1beta1: permission denied
I0926 09:10:38.336229 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1beta1/serverresources.json.897633413: permission denied
I0926 09:10:38.338151 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1/serverresources.json.866995967: permission denied
I0926 09:10:38.340325 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1beta1/serverresources.json.530593673: permission denied
I0926 09:10:38.342279 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1beta1/serverresources.json.523148963: permission denied
I0926 09:10:38.344700 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1beta1/serverresources.json.398974408: permission denied
I0926 09:10:38.346887 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1/serverresources.json.075738247: permission denied
I0926 09:10:38.349032 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1/serverresources.json.925550673: permission denied
I0926 09:10:38.351282 143977 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0926 09:10:38.351323 143977 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0926 09:10:38.361244 143977 cached_discovery.go:130] failed to write cache to /home/username/.kube/cache/discovery/ip_port/servergroups.json due to open /home/username/.kube/cache/discovery/ip_port/servergroups.json.078017807: permission denied
I0926 09:10:38.383656 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apps/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apps/v1/serverresources.json.925277627: permission denied
I0926 09:10:38.391307 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/v1/serverresources.json.640309234: permission denied
I0926 09:10:38.434800 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1beta1/serverresources.json.467193527: permission denied
I0926 09:10:38.436832 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1/serverresources.json.950885546: permission denied
I0926 09:10:38.438719 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta2/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta2/serverresources.json.319412588: permission denied
I0926 09:10:38.440611 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1beta1/serverresources.json.196545918: permission denied
I0926 09:10:38.442429 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1beta1/serverresources.json.098460512: permission denied
I0926 09:10:38.444233 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1beta1/serverresources.json.979104658: permission denied
I0926 09:10:38.446167 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/extensions/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/extensions/v1beta1/serverresources.json.167910548: permission denied
I0926 09:10:38.448055 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1/serverresources.json.930787558: permission denied
I0926 09:10:38.449955 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authentication.k8s.io/v1/serverresources.json.931838216: permission denied
I0926 09:10:38.451917 143977 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0926 09:10:38.453789 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1/serverresources.json.694079164: permission denied
I0926 09:10:38.455842 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiextensions.k8s.io/v1/serverresources.json.500010219: permission denied
I0926 09:10:38.457690 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/rbac.authorization.k8s.io/v1beta1/serverresources.json.018542512: permission denied
I0926 09:10:38.459606 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v1/serverresources.json.062236495: permission denied
I0926 09:10:38.461493 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/scheduling.k8s.io/v1beta1/serverresources.json.838471257: permission denied
I0926 09:10:38.463525 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1beta1/serverresources.json.157864947: permission denied
I0926 09:10:38.466064 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/authorization.k8s.io/v1/serverresources.json.241774237: permission denied
I0926 09:10:38.468064 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1/serverresources.json.946278103: permission denied
I0926 09:10:38.469925 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1beta1/serverresources.json.481704225: permission denied
I0926 09:10:38.472079 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/apiregistration.k8s.io/v1beta1/serverresources.json.777763835: permission denied
I0926 09:10:38.474016 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/events.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/events.k8s.io/v1beta1/serverresources.json.236970469: permission denied
I0926 09:10:38.476174 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/storage.k8s.io/v1/serverresources.json.613842783: permission denied
I0926 09:10:38.478043 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/admissionregistration.k8s.io/v1/serverresources.json.335527145: permission denied
I0926 09:10:38.480113 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/discovery.k8s.io/v1beta1/serverresources.json due to mkdir /home/username/.kube/cache/discovery/ip_port/discovery.k8s.io/v1beta1: permission denied
I0926 09:10:38.482040 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/policy/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/policy/v1beta1/serverresources.json.558002310: permission denied
I0926 09:10:38.483961 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/autoscaling/v2beta1/serverresources.json.393956776: permission denied
I0926 09:10:38.485988 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/coordination.k8s.io/v1beta1/serverresources.json.744410394: permission denied
I0926 09:10:38.487968 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/networking.k8s.io/v1/serverresources.json.612010332: permission denied
I0926 09:10:38.489993 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/batch/v1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/batch/v1/serverresources.json.856159470: permission denied
I0926 09:10:38.492015 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/node.k8s.io/v1beta1/serverresources.json due to mkdir /home/username/.kube/cache/discovery/ip_port/node.k8s.io/v1beta1: permission denied
I0926 09:10:38.494009 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/batch/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/batch/v1beta1/serverresources.json.591595375: permission denied
I0926 09:10:38.496019 143977 cached_discovery.go:87] failed to write cache to /home/username/.kube/cache/discovery/ip_port/certificates.k8s.io/v1beta1/serverresources.json due to open /home/username/.kube/cache/discovery/ip_port/certificates.k8s.io/v1beta1/serverresources.json.749766009: permission denied
F0926 09:10:38.522153 143977 helpers.go:114] error: unable to upgrade connection: Unauthorized
任何帮助都将不胜感激。
1条答案
按热度按时间46qrfjad1#
我遇到了完全相同的情况,主节点重新启动帮助了我。