大概的配置参数:
node故障后,pod会迁移到 正常的node上, 迁移时间大概8分钟左右, 如果是微服务,注册到nacos,服务不受影响,但是对于其他的服务,
请求中会有大量失败。
需要几个流程:
kubelet 自身会定期更新状态到 apiserver,通过kubelet的参数 node-status-update-frequency 配置上报频率,默认 10s 上报一次。
kube-controller-manager定期去探测kubelet的运行状态,默认5s问kubelet,使用--node-monitor-grace-period参数
kube-controller-manager询问kubelet,如果5分钟没有回复,认为kubelet有问题,node-monitor-grace-period,是kubelet参数node-status-update-frequency的整数倍
如果有问题,故障node节点被设置成污点
- node.kubernetes.io/unreachable:NoExecute
- node.kubernetes.io/unreachable:NoSchedule
节点被设置了污点,pod节点上还是running,apiserver的参数--default-unreachable-toleration-seconds是驱逐掉时间。
vim /var/lib/kubelet/config.yaml
memorySwap: {}
#修改kubelet向apiserver 汇报时间,由10秒变成4秒 node-status-update-frequency: 4s nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s shutdownGracePeriod: 0s shutdownGracePeriodCriticalPods: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s
vim /etc/kubernetes/manifests/kube-apiserver.yaml
spec:containers:- command:- kube-apiserver- --advertise-address=192.168.148.131- --allow-privileged=true- --authorization-mode=Node,RBAC- --client-ca-file=/etc/kubernetes/pki/ca.crt- --enable-admission-plugins=NodeRestriction- --enable-bootstrap-token-auth=true- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key- --etcd-servers=https://127.0.0.1:2379- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key- --requestheader-allowed-names=front-proxy-client- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt- --requestheader-extra-headers-prefix=X-Remote-Extra-- --requestheader-group-headers=X-Remote-Group- --requestheader-username-headers=X-Remote-User- --secure-port=6443- --service-account-issuer=https://kubernetes.default.svc.cluster.local- --service-account-key-file=/etc/kubernetes/pki/sa.pub- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key- --service-cluster-ip-range=10.96.0.0/12- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
#当node节点为notready或者 unreachable时,5秒钟驱逐掉,默认300秒- --default-unreachable-toleration-seconds=5- --default-not-ready-toleration-seconds=5
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
spec:containers:- command:- kube-controller-manager- --allocate-node-cidrs=true- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf- --bind-address=127.0.0.1- --client-ca-file=/etc/kubernetes/pki/ca.crt- --cluster-cidr=10.244.0.0/16- --cluster-name=kubernetes- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key- --controllers=*,bootstrapsigner,tokencleaner- --kubeconfig=/etc/kubernetes/controller-manager.conf- --leader-elect=true- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt- --root-ca-file=/etc/kubernetes/pki/ca.crt- --service-account-private-key-file=/etc/kubernetes/pki/sa.key- --service-cluster-ip-range=10.96.0.0/12- --use-service-account-credentials=true- --v=5
# kube-controller-manager 修改成2秒主动探测,默认值5秒- --node-monitor-period=2s
# 将一个node标记为不健康之前允许其无响应的上线- --node-monitor-grace-period=12s
验证脚本,中间关闭node, 查看pod漂移时间
while true; do echo `date +%F-%T`; kubectl get node ;echo "####################################################################"; kubectl describe nodes node2 | grep -A5 Taint;echo "";echo "##############################################################"; kubectl get pod -o wide; sleep 5; done