K8s-controller-manager叢集.06-3
阿新 • • 發佈：2019-01-03
06-3.部署高可用 kube-controller-manager 叢集
本文件介紹部署高可用 kube-controller-manager 叢集的步驟。
該叢集包含 3 個節點，啟動後將通過競爭選舉機制產生一個 leader 節點，其它節點為阻塞狀態。當 leader 節點不可用後，剩餘節點將再次進行選舉產生新的 leader 節點，從而保證服務的可用性。
為保證通訊安全，本文件先生成 x509 證書和私鑰，kube-controller-manager 在如下兩種情況下使用該證書：
- 與 kube-apiserver 的安全埠通訊時;
- 在安全埠(https，10252) 輸出 prometheus 格式的 metrics；
準備工作
下載最新版本的二進位制檔案、安裝和配置 flanneld 參考：K8s-部署master節點.06
建立 kube-controller-manager 證書和私鑰
建立證書籤名請求：
cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "172.27.129.101", "172.27.129.102", "172.27.129.103" ], "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "4Paradigm" } ] } EOF
- hosts 列表包含所有 kube-controller-manager 節點 IP；
- CN 為 system:kube-controller-manager、O 為 system:kube-controller-manager，kubernetes 內建的 ClusterRoleBindings system:kube-controller-manager 賦予 kube-controller-manager 工作所需的許可權。
生成證書和私鑰：
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \ -ca-key=/etc/kubernetes/cert/ca-key.pem \ -config=/etc/kubernetes/cert/ca-config.json \ -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
將生成的證書和私鑰分發到所有 master 節點：
source /opt/k8s/bin/environment.sh
for master_ip in ${MASTER_IP[@]}
do
echo ">>> ${master_ip}"
scp kube-controller-manager*.pem [email protected]${master_ip}:/etc/kubernetes/cert/
done
建立和分發 kubeconfig 檔案
kubeconfig 檔案包含訪問 apiserver 的所有資訊，如 apiserver 地址、CA 證書和自身使用的證書；
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.pem \
--client-key=kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
分發 kubeconfig 到所有 master 節點：
source /opt/k8s/bin/environment.sh
for master_ip in ${MASTER_IP[@]}
do
echo ">>> ${master_ip}"
scp kube-controller-manager.kubeconfig [email protected]${master_ip}:/etc/kubernetes/
done
建立和分發 kube-controller-manager systemd unit 檔案
source /opt/k8s/bin/environment.sh
cat > kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/k8s/bin/kube-controller-manager \\
--port=0 \\
--secure-port=10252 \\
--bind-address=127.0.0.1 \\
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--service-cluster-ip-range=${SERVICE_CIDR} \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
--experimental-cluster-signing-duration=8760h \\
--root-ca-file=/etc/kubernetes/cert/ca.pem \\
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
--leader-elect=true \\
--feature-gates=RotateKubeletServerCertificate=true \\
--controllers=*,bootstrapsigner,tokencleaner \\
--horizontal-pod-autoscaler-use-rest-clients=true \\
--horizontal-pod-autoscaler-sync-period=10s \\
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
--use-service-account-credentials=true \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/var/log/kubernetes \\
--v=2
Restart=on
Restart=on-failure
RestartSec=5
User=k8s
[Install]
WantedBy=multi-user.target
EOF
--port=0：關閉監聽 http /metrics 的請求，同時--address引數無效，--bind-address引數有效；--secure-port=10252、--bind-address=0.0.0.0: 在所有網路介面監聽 10252 埠的 https /metrics 請求；--kubeconfig：指定 kubeconfig 檔案路徑，kube-controller-manager 使用它連線和驗證 kube-apiserver；--cluster-signing-*-file：簽名 TLS Bootstrap 建立的證書；--experimental-cluster-signing-duration：指定 TLS Bootstrap 證書的有效期；--root-ca-file：放置到容器 ServiceAccount 中的 CA 證書，用來對 kube-apiserver 的證書進行校驗；--service-account-private-key-file：簽名 ServiceAccount 中 Token 的私鑰檔案，必須和 kube-apiserver 的--service-account-key-file指定的公鑰檔案配對使用；--service-cluster-ip-range：指定 Service Cluster IP 網段，必須和 kube-apiserver 中的同名引數一致；--leader-elect=true：叢集執行模式，啟用選舉功能；被選為 leader 的節點負責處理工作，其它節點為阻塞狀態；--feature-gates=RotateKubeletServerCertificate=true：開啟 kublet server 證書的自動更新特性；--controllers=*,bootstrapsigner,tokencleaner：啟用的控制器列表，tokencleaner 用於自動清理過期的 Bootstrap token；--horizontal-pod-autoscaler-*：custom metrics 相關引數，支援 autoscaling/v2alpha1；--tls-cert-file、--tls-private-key-file：使用 https 輸出 metrics 時使用的 Server 證書和祕鑰；--use-service-account-credentials=true:User=k8s：使用 k8s 賬戶執行；
kube-controller-manager 不對請求 https metrics 的 Client 證書進行校驗，故不需要指定 --tls-ca-file 引數，而且該引數已被淘汰。
分發 systemd unit 檔案到所有 master 節點：
source /opt/k8s/bin/environment.sh
for master_ip in ${MASTER_IP[@]}
do
echo ">>> ${master_ip}"
scp kube-controller-manager.service [email protected]${master_ip}:/etc/systemd/system/
done
kube-controller-manager 的許可權
ClusteRole: system:kube-controller-manager 的許可權很小，只能建立 secret、serviceaccount 等資源物件，各 controller 的許可權分散到 ClusterRole system:controller:XXX 中。
需要在 kube-controller-manager 的啟動引數中新增 --use-service-account-credentials=true 引數，這樣 main controller 會為各 controller 建立對應的 ServiceAccount XXX-controller。
內建的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 許可權。
啟動 kube-controller-manager 服務
source /opt/k8s/bin/environment.sh
for master_ip in ${MASTER_IP[@]}
do
echo ">>> ${master_ip}"
ssh [email protected]${master_ip} "mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes"
ssh [email protected]${master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
done
- 必須先建立日誌目錄；
檢查服務執行狀態
source /opt/k8s/bin/environment.sh
for master_ip in ${MASTER_IP[@]}
do
echo ">>> ${master_ip}"
ssh [email protected]${master_ip} "systemctl status kube-controller-manager|grep Active"
done
確保狀態為 active (running)，否則檢視日誌，確認原因：
$ journalctl -u kube-controller-manager
檢視輸出的 metric
注意：以下命令在 kube-controller-manager 節點上執行。
kube-controller-manager 監聽 10252 埠，接收 https 請求：
$ sudo netstat -lnpt|grep kube-controll tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 18377/kube-controll
$ curl -s --cacert /etc/kubernetes/cert/ca.pem https://127.0.0.1:10252/metrics |head
# HELP ClusterRoleAggregator_adds Total number of adds handled by workqueue: ClusterRoleAggregator
# TYPE ClusterRoleAggregator_adds counter
ClusterRoleAggregator_adds 3
# HELP ClusterRoleAggregator_depth Current depth of workqueue: ClusterRoleAggregator
# TYPE ClusterRoleAggregator_depth gauge
ClusterRoleAggregator_depth 0
# HELP ClusterRoleAggregator_queue_latency How long an item stays in workqueueClusterRoleAggregator before being requested.
# TYPE ClusterRoleAggregator_queue_latency summary
ClusterRoleAggregator_queue_latency{quantile="0.5"} 57018
ClusterRoleAggregator_queue_latency{quantile="0.9"} 57268
- curl --cacert CA 證書用來驗證 kube-controller-manager https server 證書；
測試 kube-controller-manager 叢集的高可用
停掉一個或兩個節點的 kube-controller-manager 服務，觀察其它節點的日誌，看是否獲取了 leader 許可權。
檢視當前的 leader
$ kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master-0002_084534e2-6cc4-11e8-a418-5254001f5b65","leaseDurationSeconds":15,"acquireTime":"2018-06-10T15:40:33Z","renewTime":"2018-06-10T16:19:08Z","leaderTransitions":12}'
creationTimestamp: 2018-06-10T13:59:42Z
name: kube-controller-manager
namespace: kube-system
resourceVersion: "4540"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: 862cc048-6cb6-11e8-96fa-525400ba84c6
可見，當前的 leader 為 k8s-master-0002 節點。
參考
- 關於 controller 許可權和 use-service-account-credentials 引數：https://github.com/kubernetes/kubernetes/issues/48208
- kublet 認證和授權：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization