本文是这次升级的完整流程记录将升级流程与遇到的问题都整理进来了。后续升级时如果遇到什么新问题也都会更新到本文中。升级前的集群状态# kubectl get node -o wide NAME STATUS ROLES AGE VERSION OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME x.x.x Ready control-plane 342d v1.31.14 Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://1.7.27 x.x.x Ready control-plane 342d v1.31.14 Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://1.7.27 x.x.x Ready control-plane 342d v1.31.14 Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://1.7.27Kubernetes 升级1.版本升级规划K8s 官方文档明确说明Skipping MINOR versions when upgrading is unsupported也就是说从v1.31.14升级到v1.35.6不能一步到位必须逐个版本升级共经历四跳v1.31.14 - v1.32.x - v1.33.x - v1.34.x - v1.35.62.版本升级流程由于不能跳版本v1.31 - v1.35本质上是把单次升级流程重复执行四遍以下步骤针对每个控制平面节点每完成一个版本升级后所有节点同步升级再进入下一个版本升级流程。2.1.腾空节点只对当前升级的节点进行kubectl drain xxx \ --ignore-daemonsets \ --delete-emptydir-data \ --timeout300s2.2.备份 ETCD在所有控制平面节点上执行。ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /root/backup/etcd-$(date %F-%H%M).db执行成功后输出一堆info日志最终看到Snapshot saved at ...就备份成功了{level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:65,msg:created temporary db file,path:/backup/etcd-xxxx-xx-xx-xxxx.db.part} {level:info,ts:xxxx-xx-xx,logger:client,caller:v3v3.5.21/maintenance.go:212,msg:opened snapshot stream; downloading} {level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:88,msg:fetched snapshot,endpoint:https://127.0.0.1:2379,size:xx MB,took:xx second ago} {level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:97,msg:saved,path:/backup/etcd-xxxx-xx-xx-xxxx.db} Snapshot saved at /backup/etcd-x.x.x-xxxx.db2.3.更新软件源当前环境使用清华源。每次版本升级都需要把旧版本源注释或清理换成新版本源并下载对应的仓库公钥。在所有控制平面节点更新# 下载前先将旧版本公钥改名备份一下,等升级结束后再清理 curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \ | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # 配置新版本源 # $K8S_VERSION: 具体要升级的版本 cat /etc/apt/sources.list.d/kubernetes.list deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/$K8S_VERSION/deb/ / # 更新源 apt-get update # 列出当前可用版本 apt-cache madison kubeadm | head -n 1 # 示例输出 kubeadm | 1.32.13-1.1 | http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.32/deb Packages apt-cache madison kubelet | head -n 1 apt-cache madison kubectl | head -n 1 # 确认组件未被 apt 锁定(hold) apt-mark showhold | grep -Ei kubeadm|kubelet|kubectl2.4.验证升级计划检查可升级到哪些版本并验证当前集群是否可升级。该命令只能在存在 kubeconfig 文件admin.conf的控制平面节点上运行。# 安装目标版本的 kubeadm apt-get install -y kubeadm1.32.13-1.1 # 确认版本 kubeadm version # 验证升级计划: kubeadm 会列出当前版本/目标版本/各组件变更 kubeadm upgrade plan # 输出内容(忽略输出的版本信息) [preflight] Running pre-flight checks. [upgrade/config] Reading configuration from the kubeadm-config ConfigMap in namespace kube-system... [upgrade/config] Use kubeadm init phase upload-config --config your-config-file to re-upload it. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: 1.32.13 [upgrade/versions] kubeadm version: v1.33.13 I0622 18:27:10.934338 3715038 version.go:261] remote version is much newer: v1.36.2; falling back to: stable-1.33 [upgrade/versions] Target version: v1.33.13 [upgrade/versions] Latest version in the v1.32 series: v1.32.13 Components that must be upgraded manually after you have upgraded the control plane with kubeadm upgrade apply: COMPONENT NODE CURRENT TARGET kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 Upgrade to the latest stable version: COMPONENT NODE CURRENT TARGET kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-proxy 1.32.13 v1.33.13 CoreDNS v1.11.3 v1.12.0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.33.13