每天一點基礎K8S--K8S中的排程策略---汙點(Taints)和容忍度(Tolerations)
阿新 • • 發佈：2022-11-30
汙點和容忍度
之前的實驗測試了排程策略中的nodeName、nodeSelector、節點親和性和pod親和性。
有時，為了實現部分pod不能執行在特定的節點上，可以將節點打上汙點。此時容忍這個汙點的POD還是可以被排程到該節點上
環境中一共有2個master節點，2個worker節點 [root@master-worker-node-1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master-worker-node-1 Ready control-plane 4d20h v1.25.3 master-worker-node-2 Ready control-plane 4d19h v1.25.3 only-worker-node-3 Ready worker 4d19h v1.25.3 only-worker-node-4 Ready worker 4d19h v1.25.3
正常情況下，新建的pod只能在only-worker-node-3和only-worker-node-4上執行 [root@master-worker-node-1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-affinity-base-pod 1/1 Running 29 (20m ago) 10h 10.244.31.11 only-worker-node-3 <none> <none> test-node-affinity-2 1/1 Running 4 (140m ago) 16h 10.244.54.8 only-worker-node-4 <none> <none> test-node-affinity-3 1/1 Running 4 (122m ago) 15h 10.244.54.9 only-worker-node-4 <none> <none> test-pod-affinity-by-labelselector 1/1 Running 27 (13m ago) 9h 10.244.54.10 only-worker-node-4 <none> <none> test-pod-affinity-by-labelselector-2 1/1 Running 27 (89s ago) 9h 10.244.31.12 only-worker-node-3 <none> <none> test-prefered-1 1/1 Running 9 (5m36s ago) 9h 10.244.54.11 only-worker-node-4 <none> <none> test-prefered-2 1/1 Running 8 (55m ago) 8h 10.244.31.13 only-worker-node-3 <none> <none>
但是使用kubeadm搭建環境時，etcd、kube-apiserver、kube-controller-manager、kube-scheduler卻可以被排程到這兩個節點上。 [root@master-worker-node-1 ~]# kubectl get pods -n kube-system -o wide | grep master-worker calico-node-49qt2 1/1 Running 0 4d16h 192.168.122.106 master-worker-node-2 <none> <none> calico-node-q2wpg 1/1 Running 0 4d16h 192.168.122.89 master-worker-node-1 <none> <none> etcd-master-worker-node-1 1/1 Running 5 4d20h 192.168.122.89 master-worker-node-1 <none> <none> etcd-master-worker-node-2 1/1 Running 0 4d19h 192.168.122.106 master-worker-node-2 <none> <none> kube-apiserver-master-worker-node-1 1/1 Running 32 4d20h 192.168.122.89 master-worker-node-1 <none> <none> kube-apiserver-master-worker-node-2 1/1 Running 1 (4d19h ago) 4d19h 192.168.122.106 master-worker-node-2 <none> <none> kube-controller-manager-master-worker-node-1 1/1 Running 10 (4d16h ago) 4d20h 192.168.122.89 master-worker-node-1 <none> <none> kube-controller-manager-master-worker-node-2 1/1 Running 0 4d19h 192.168.122.106 master-worker-node-2 <none> <none> kube-proxy-7gjz9 1/1 Running 0 4d20h 192.168.122.89 master-worker-node-1 <none> <none> kube-proxy-c4d2m 1/1 Running 0 4d19h 192.168.122.106 master-worker-node-2 <none> <none> kube-scheduler-master-worker-node-1 1/1 Running 8 (4d16h ago) 4d20h 192.168.122.89 master-worker-node-1 <none> <none> kube-scheduler-master-worker-node-2 1/1 Running 0 4d19h 192.168.122.106 master-worker-node-2 <none> <none>
正常建立的pod不能排程到master節點是因為該節點上有汙點，而且新建的POD不能容忍這個汙點。
[root@master-worker-node-1 ~]# kubectl describe nodes master-worker-node-1 | grep ^Taint
Taints: node-role.kubernetes.io/control-plane:NoSchedule
[root@master-worker-node-1 ~]# kubectl describe nodes master-worker-node-2 | grep ^Taint
Taints: node-role.kubernetes.io/control-plane:NoSchedule
kubeadm搭建環境時建立的pod卻可以排程到master-node，是因為這些pod具有容忍度，能夠容忍node的汙點
[root@master-worker-node-1 ~]# kubectl describe pods -n kube-system kube-scheduler-master-worker-node-1 | grep ^Tolerations
Tolerations: :NoExecute op=Exists
汙點和容忍度裡面的effect欄位
NoSchedule
Do not allow new pods to schedule onto the node unless they tolerate the taint, but allow all pods submitted to Kubelet without going through the scheduler to start, and allow all already-running pods to continue running. Enforced by the scheduler.
不允許將新建的pod排程到含有該節點，除非新建的POD允許該汙點。但是允許讓停止的pod再次在該節點執行，已經執行的pod不會被排程。
PreferNoSchedule
Like TaintEffectNoSchedule, but the scheduler tries not to schedule new pods onto the node, rather than prohibiting new pods from scheduling onto the node entirely. Enforced by the scheduler.
對於新建的pod，在排程的時候會盡力避免該node，對於該節點上已經有的pod不受影響。
NoExecute
Evict any already-running pods that do not tolerate the taint.
將會把該節點不能容忍這個汙點的所有pod（包括執行的），全部驅逐。
汙點taints
[root@master-worker-node-1 pod]# cat test-taints.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-busybox
labels:
func: test
spec:
containers:
- name: test-busybox
image: busybox:1.28
imagePullPolicy: IfNotPresent
command: ["/bin/sh","-c","sleep 123456"]
正常被排程到only-worker-node-4
[root@master-worker-node-1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-busybox 1/1 Running 0 11m 10.244.54.12 only-worker-node-4 <none> <none>
給only-worker-node-4新增一個NoScheduler的taints
[root@master-worker-node-1 pod]# kubectl taint node only-worker-node-4 test-taints:NoSchedule
node/only-worker-node-4 tainted
[root@master-worker-node-1 pod]# kubectl describe node only-worker-node-4 | grep ^Taints
Taints: test-taints:NoSchedule
pod還是執行在了only-worker-node-4上
[root@master-worker-node-1 ~]# kubectl get pods -w -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-busybox 1/1 Running 0 25m 10.244.54.12 only-worker-node-4 <none> <none>
給only-worker-node-4打上NoExecute標籤
[root@master-worker-node-1 pod]# kubectl taint nodes only-worker-node-4 test-taints:NoExecute
node/only-worker-node-4 tainted
[root@master-worker-node-1 pod]# kubectl describe nodes only-worker-node-4 | grep ^Taints
Taints: test-taints:NoExecute
pod將被終止
[root@master-worker-node-1 ~]# kubectl get pods -w -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-busybox 1/1 Running 0 25m 10.244.54.12 only-worker-node-4 <none> <none>
test-busybox 1/1 Terminating 0 37m 10.244.54.12 only-worker-node-4 <none> <none>
test-busybox 1/1 Terminating 0 37m 10.244.54.12 only-worker-node-4 <none> <none>
test-busybox 0/1 Terminating 0 37m 10.244.54.12 only-worker-node-4 <none> <none>
test-busybox 0/1 Terminating 0 37m 10.244.54.12 only-worker-node-4 <none> <none>
容忍度
給only-worker-node-3和only-worker-node-4都打上NoSchedule的標籤
[root@master-worker-node-1 pod]# kubectl taint nodes only-worker-node-3 test-taints:NoSchedule
node/only-worker-node-3 tainted
[root@master-worker-node-1 pod]# kubectl taint nodes only-worker-node-4 test-taints:NoSchedule
node/only-worker-node-4 tainted
[root@master-worker-node-1 pod]# cat test-taints-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-busybox
labels:
func: test
spec:
containers:
- name: test-busybox
image: busybox:1.28
imagePullPolicy: IfNotPresent
command: ["/bin/sh","-c","sleep 123456"]
tolerations:
- effect: NoSchedule
key: test-taints
operator: Exists
pod可以正常排程
[root@master-worker-node-1 pod]# kubectl apply -f test-taints-2.yaml
pod/test-busybox created
[root@master-worker-node-1 pod]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-busybox 1/1 Running 0 10s 10.244.31.15 only-worker-node-3 <none> <none>