010.Kubernetes二進位制部署kube-controller-manager
阿新 • • 發佈：2019-11-17
一 部署高可用kube-controller-manager
1.1 高可用kube-controller-manager介紹
本實驗部署一個三例項 kube-controller-manager 的叢集，啟動後將通過競爭選舉機制產生一個 leader 節點，其它節點為阻塞狀態。當 leader 節點不可用時，阻塞的節點將再次進行選舉產生新的 leader 節點，從而保證服務的可用性。 為保證通訊安全，本文件先生成 x509 證書和私鑰，kube-controller-manager 在如下兩種情況下使用該證書：- 與 kube-apiserver 的安全埠通訊;
- 在安全埠(https，10252) 輸出 prometheus 格式的 metrics。
1.2 建立kube-controller-manager證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > kube-controller-manager-csr.json <<EOF
3 {
4 "CN": "system:kube-controller-manager",
5 "hosts": [
6 "127.0.0.1",
7 "172.24.8.71",
8 "172.24.8.72",
9 "172.24.8.73"
10 ],
11 "key": {
12 "algo": "rsa",
13 "size": 2048
14 },
15 "names": [
16 {
17 "C": "CN",
18 "ST": "Shanghai",
19 "L": "Shanghai",
20 "O": "system:kube-controller-manager",
21 "OU": "System"
22 }
23 ]
24 }
25 EOF
26 #建立kube-controller-manager的CA證書請求檔案1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager #生成CA金鑰（ca-key.pem）和證書（ca.pem）
1.3 分發證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 scp kube-controller-manager*.pem root@${master_ip}:/etc/kubernetes/cert/
7 done
1.4 建立和分發kubeconfig
kube-controller-manager 使用 kubeconfig 檔案訪問 apiserver，該檔案提供了 apiserver 地址、嵌入的 CA 證書和 kube-controller-manager 證書： 1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \
4 --certificate-authority=/opt/k8s/work/ca.pem \
5 --embed-certs=true \
6 --server=${KUBE_APISERVER} \
7 --kubeconfig=kube-controller-manager.kubeconfig
8
9 [root@k8smaster01 work]# kubectl config set-credentials system:kube-controller-manager \
10 --client-certificate=kube-controller-manager.pem \
11 --client-key=kube-controller-manager-key.pem \
12 --embed-certs=true \
13 --kubeconfig=kube-controller-manager.kubeconfig
14
15 [root@k8smaster01 work]# kubectl config set-context system:kube-controller-manager \
16 --cluster=kubernetes \
17 --user=system:kube-controller-manager \
18 --kubeconfig=kube-controller-manager.kubeconfig
19
20 [root@k8smaster01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
21
22 [root@k8smaster01 ~]# cd /opt/k8s/work
23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
24 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
25 do
26 echo ">>> ${master_ip}"
27 scp kube-controller-manager.kubeconfig root@${master_ip}:/etc/kubernetes/
28 done
1.5 建立kube-controller-manager的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kube-controller-manager.service.template <<EOF
4 [Unit]
5 Description=Kubernetes Controller Manager
6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
7
8 [Service]
9 WorkingDirectory=${K8S_DIR}/kube-controller-manager
10 ExecStart=/opt/k8s/bin/kube-controller-manager \\
11 --profiling \\
12 --cluster-name=kubernetes \\
13 --controllers=*,bootstrapsigner,tokencleaner \\
14 --kube-api-qps=1000 \\
15 --kube-api-burst=2000 \\
16 --leader-elect \\
17 --use-service-account-credentials\\
18 --concurrent-service-syncs=2 \\
19 --bind-address=##MASTER_IP## \\
20 --secure-port=10252 \\
21 --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
22 --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
23 --port=0 \\
24 --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
25 --client-ca-file=/etc/kubernetes/cert/ca.pem \\
26 --requestheader-allowed-names="" \\
27 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
28 --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
29 --requestheader-group-headers=X-Remote-Group \\
30 --requestheader-username-headers=X-Remote-User \\
31 --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
32 --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
33 --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
34 --experimental-cluster-signing-duration=8760h \\
35 --horizontal-pod-autoscaler-sync-period=10s \\
36 --concurrent-deployment-syncs=10 \\
37 --concurrent-gc-syncs=30 \\
38 --node-cidr-mask-size=24 \\
39 --service-cluster-ip-range=${SERVICE_CIDR} \\
40 --pod-eviction-timeout=6m \\
41 --terminated-pod-gc-threshold=10000 \\
42 --root-ca-file=/etc/kubernetes/cert/ca.pem \\
43 --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
44 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
45 --logtostderr=true \\
46 --v=2
47 Restart=on-failure
48 RestartSec=5
49
50 [Install]
51 WantedBy=multi-user.target
52 EOF
1.6 分發systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ ))
4 do
5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${MASTER_IPS[i]}.service
6 done #修正相應IP
7 [root@k8smaster01 work]# ls kube-controller-manager*.service
8 [root@k8smaster01 ~]# cd /opt/k8s/work
9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
11 do
12 echo ">>> ${master_ip}"
13 scp kube-controller-manager-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-controller-manager.service
14 done #分發system
二 啟動並驗證
2.1 啟動kube-controller-manager 服務
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
4 do
5 echo ">>> ${master_ip}"
6 ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
7 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
8 done
2.2 檢查kube-controller-manager 服務
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]}
3 do
4 echo ">>> ${master_ip}"
5 ssh root@${master_ip} "systemctl status kube-controller-manager|grep Active"
6 done
2.3 檢視輸出的 metrics
1 [root@k8smaster01 ~]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.24.8.71:10252/metrics |head注意：以上命令在 kube-controller-manager 節點上執行。
2.4 檢視許可權
1 [root@k8smaster01 ~]# kubectl describe clusterrole system:kube-controller-managerClusteRole system:kube-controller-manager 的許可權很小，只能建立 secret、serviceaccount 等資源物件，各 controller 的許可權分散到 ClusterRole system:controller:XXX 中。 當在 kube-controller-manager 的啟動引數中新增 --use-service-account-credentials=true 引數，這樣 main controller 會為各 controller 建立對應的 ServiceAccount XXX-controller。內建的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 許可權。
1 [root@k8smaster01 ~]# kubectl get clusterrole|grep controller如deployment controller：
1 [root@k8smaster01 ~]# kubectl describe clusterrole system:controller:deployment-controller
2.5 檢視當前leader
1 [root@k8smaster01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yamlkubelet 認證和授權：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorizati