关于openelbl作为调度器实现k8s的Service的loadbalancer模式Kubernetes 原生不提供 LoadBalancer 类型 Service 的具体实现尤其在裸机或私有云环境。需要借助外部组件如 OpenELB来充当 4/7 层调度器提供负载均衡能力并能自动判断后端 Pod 的健康状态。本文档记录了在 K8s 集群中部署 OpenELB 并测试 LoadBalancer Service 的完整过程及问题解决方法。1.安装OpenELB1.1下载资源清单wgethttps://raw.githubusercontent.com/openelb/openelb/release-0.6/deploy/openelb.yaml1.2应用[rootk8s-master01 ~]# kubectl apply -f openelb.yaml1.3 检验[rootk8s-master01 ~]# kubectl get pod -n openelb-systemNAME READY STATUS RESTARTS AGE openelb-admission-create-8ndq20/1 ContainerCreating012s openelb-admission-patch-pn9n60/1 ContainerCreating012s openelb-controller-5cdcf556d4-rtvtg0/1 ContainerCreating012s openelb-speaker-4plq60/1 ContainerCreating012s openelb-speaker-b4qjf0/1 ContainerCreating012s openelb-speaker-c5hwc0/1 ContainerCreating012s2.配置 EIP弹性 IP 地址池2.1 EIP 资源配置文件apiVersion:network.kubesphere.io/v1alpha2kind:Eipmetadata:name:eip-sample-poolannotations:eip.openelb.kubesphere.io/is-default-eip:truespec:address:192.168.64.100-192.168.64.150# 可用的 IP 范围priority:100namespaces:-test-defaultnamespaceSelector:kubesphere.io/workspace:workspacedisable:falseprotocol:layer2# 使用二层协议ARP/NDPinterface:ens160# 指定网络接口2.2 应用 故障解决[rootk8s-master01 ~]# kubectl apply -f eip.ymlError from server(InternalError): error when creatingeip.yml:Internal error occurred: failed calling webhookvalidate.eip.network.kubesphere.io:failed to call webhook: Posthttps://openelb-controller.openelb-system.svc:443/validate-network-kubesphere-io-v1alpha2-eip?timeout10s:dial tcp10.7.129.16:443: connect: connection refused原因是 OpenELB 的 Admission Webhook 无法正常提供服务根本原因在于相关 Pod 未正常运行。检测相关pod 查看openelb-controller[rootk8s-master01 ~]# kubectl get pod -n openelb-systemNAME READY STATUS RESTARTS AGE openelb-admission-create-8ndq20/1 ImagePullBackOff06m32s openelb-admission-patch-pn9n60/1 ErrImagePull06m32s openelb-controller-5cdcf556d4-rtvtg0/1 ContainerCreating06m32s openelb-speaker-4plq60/1 ContainerCreating06m32s openelb-speaker-b4qjf0/1 ContainerCreating06m32s openelb-speaker-c5hwc1/1 Running06m32s[rootk8s-master01 ~]# kubectl describe pod -n openelb-system openelb-controller-5cdcf556d4-rtvtg....... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m default-scheduler Successfully assigned openelb-system/openelb-controller-5cdcf556d4-rtvtg to k8s-master01 Warning FailedMount 48s(x11 over 7m)kubelet MountVolume.SetUp failedforvolumewebhook-cert:secretopenelb-admissionnot found解决问题临时解决方案由于 Webhook 配置存在问题导致 EIP 创建被拦截。可临时删除该 ValidatingWebhookConfiguration 以绕过校验[rootk8s-master01 ~]# kubectl get validatingwebhookconfiguration | grep openelbopenelb-admission18m3s[rootk8s-master01 ~]#[rootk8s-master01 ~]# kubectl delete validatingwebhookconfiguration openelb-admissionvalidatingwebhookconfiguration.admissionregistration.k8s.ioopenelb-admissiondeleted验证[rootk8s-master01 ~]# kubectl apply -f eip.ymleip.network.kubesphere.io/eip-sample-pool created[rootk8s-master01 ~]# kubectl get eipNAME CIDR USAGE TOTAL eip-sample-pool192.168.64.100-192.168.64.1503.部署测试 LoadBalancer Service3.1 编辑资源清单[rootk8s-master01 ~]# cat test_loadbalancer.ymlapiVersion: apps/v1 kind: Deployment metadata: name: loadbalancer-deploy spec: replicas:3selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: loadbalancer-container image: hb.reg.com/library/myapp:1.0# 替换为实际可用镜像这里是我是私人仓库ports: - containerPort:80--- apiVersion: v1 kind: Service metadata: name: loadbalancer-svc spec: type: LoadBalancer ports: - port:80targetPort:80selector: app: myapp3.2应用[rootk8s-master01 ~]# kubectl apply -f test_loadbalancer.yml3.3检测[rootk8s-master01 ~]# kubectl get eip,svc,podNAME CIDR USAGE TOTAL eip.network.kubesphere.io/eip-sample-pool192.168.64.100-192.168.64.150 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE service/kubernetes ClusterIP10.0.0.1none443/TCP 9d service/loadbalancer-svc LoadBalancer10.10.18.79192.168.64.10080:32452/TCP 52s NAME READY STATUS RESTARTS AGE pod/loadbalancer-deploy-5c6f5f784d-lkvj21/1 Running052s pod/loadbalancer-deploy-5c6f5f784d-tls6p1/1 Running052s pod/loadbalancer-deploy-5c6f5f784d-wtm7b1/1 Running052s可用看到service的ip就是就是eipIP地址池设定的范围[rootk8s-master01 ~]# curl 10.10.18.79hello jock|welcome to nginx!|version1.0[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-wtm7b[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-tls6p[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-lkvj2[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-wtm7b[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-tls6p[rootk8s-master01 ~]# curl 10.10.18.79/hostname.htmlloadbalancer-deploy-5c6f5f784d-lkvj2[rootk8s-master01 ~]# curl 192.168.64.100/hostname.htmlloadbalancer-deploy-5c6f5f784d-lkvj2[rootk8s-master01 ~]# curl 192.168.64.100/hostname.htmlloadbalancer-deploy-5c6f5f784d-wtm7b[rootk8s-master01 ~]# curl 192.168.64.100/hostname.htmlloadbalancer-deploy-5c6f5f784d-tls6p可用看到3个pod轮询可以在浏览器里面查看该EXTERNAL-IP用了两个浏览器谷歌/微软本来应该有轮询效果的是浏览器的机制问题加载也不会变化。浏览器开启 HTTP Keep‑Alive 复用 TCP 连接K8s Service 轮询只在新建 TCP 连接时生效同一个连接内所有请求固定发给同一个 Pod所以刷新看不到轮询。一些Tips值得注意的是即使我们现在查看相关pod[rootk8s-master01 ~]# kubectl get pod -n openelb-systemNAME READY STATUS RESTARTS AGE openelb-admission-create-8ndq20/1 ImagePullBackOff035m openelb-admission-patch-pn9n60/1 ImagePullBackOff035m openelb-controller-5cdcf556d4-rtvtg0/1 ContainerCreating035m openelb-speaker-4plq61/1 Running035m openelb-speaker-b4qjf1/1 Running035m openelb-speaker-c5hwc1/1 Running035m当前的 Speaker 已正常运行即使 admission 和 controller 有问题你的 LoadBalancer Service 仍然可以工作。因为Speaker负责实际流量转发已经正常工作Controller主要负责 IP 分配和状态管理Admission负责资源校验已通过删除 webhook 绕过如果你只需要基础功能可以暂时忽略这些错误。但对于生产环境建议解决所有组件的健康状态。具体来说结合本次实验Speaker是OpenELB的核心数据平面组件它负责响应ARP请求将外部IP如你实验中的192.168.64.100绑定到节点MAC地址并通过iptables/IPVS规则将到达该IP的流量负载均衡到后端Pod——这正是你实验中curl 192.168.64.100/hostname.html能够成功且轮询返回三个不同Pod名称的根本原因而Controller仅负责初始的IP池管理和状态更新你的EIP在创建时已分配好IP就不再需要它Admission仅负责配置校验你删除Webhook后配置已生效就无需它这两个控制平面组件只在资源创建或变更时起作用不影响已运行服务的流量转发所以即便它们都异常只要Speaker在运行LoadBalancer Service就能正常工作。
)
