016.Kubernetes二進位制部署所有節點kube-proxy
阿新 • • 發佈：2019-11-18
一 部署 kube-proxy
kube-proxy 執行在所有節點上，它監聽 apiserver 中 service 和 endpoint 的變化情況，建立路由規則以提供服務 IP 和負載均衡功能。1.1 安裝kube-proxy
提示：k8smaster01節點已下載相應二進位制，可直接分發至node節點。1.2 分發kube-proxy
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 scp kubernetes/server/bin/kube-proxy root@${all_ip}:/opt/k8s/bin/
7 ssh root@${all_ip} "chmod +x /opt/k8s/bin/*"
8 done1.3 建立kube-scheduler證書和私鑰
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > kube-proxy-csr.json <<EOF
3 {
4 "CN": "system:kube-proxy",
5 "key": {
6 "algo": "rsa",
7 "size": 2048
8 },
9 "names": [
10 {
11 "C": "CN",
12 "ST": "Shanghai",
13 "L": "Shanghai",
14 "O": "k8s",
15 "OU": "System"
16 }
17 ]
18 }
19 EOF
20 #建立kube-scheduler的CA證書請求檔案- CN：指定該證書的 User 為 system:kube-proxy；
- 預定義的 RoleBinding system:node-proxier 將User system:kube-proxy 與 Role system:node-proxier 繫結，該 Role 授予了呼叫 kube-apiserver Proxy 相關 API 的許可權；
- 該證書只會被 kube-proxy 當做 client 證書使用，所以 hosts 欄位為空。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy #生成CA金鑰（ca-key.pem）和證書（ca.pem）
1.4 建立和分發kubeconfig
kube-proxy 使用 kubeconfig 檔案訪問 apiserver，該檔案提供了 apiserver 地址、嵌入的 CA 證書和 kube-proxy 證書： 1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \
4 --certificate-authority=/opt/k8s/work/ca.pem \
5 --embed-certs=true \
6 --server=${KUBE_APISERVER} \
7 --kubeconfig=kube-proxy.kubeconfig
8
9 [root@k8smaster01 work]# kubectl config set-credentials kube-proxy \
10 --client-certificate=kube-proxy.pem \
11 --client-key=kube-proxy-key.pem \
12 --embed-certs=true \
13 --kubeconfig=kube-proxy.kubeconfig
14
15 [root@k8smaster01 work]# kubectl config set-context default \
16 --cluster=kubernetes \
17 --user=kube-proxy \
18 --kubeconfig=kube-proxy.kubeconfig
19
20 [root@k8smaster01 work]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
21
22 [root@k8smaster01 ~]# cd /opt/k8s/work
23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
24 [root@k8smaster01 work]# for node_name in ${NODE_NAMES[@]}
25 do
26 echo ">>> ${node_name}"
27 scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/
28 done
1.5 建立kube-proxy 配置檔案
從 v1.10 開始，kube-proxy 部分引數可以配置檔案中配置。可以使用 --write-config-to 選項生成該配置檔案。 1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# cat > kube-proxy-config.yaml.template <<EOF
3 kind: KubeProxyConfiguration
4 apiVersion: kubeproxy.config.k8s.io/v1alpha1
5 clientConnection:
6 burst: 200
7 kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
8 qps: 100
9 bindAddress: ##ALL_IP##
10 healthzBindAddress: ##ALL_IP##:10256
11 metricsBindAddress: ##ALL_IP##:10249
12 enableProfiling: true
13 clusterCIDR: ${CLUSTER_CIDR}
14 hostnameOverride: ##ALL_NAME##
15 mode: "ipvs"
16 portRange: ""
17 kubeProxyIPTablesConfiguration:
18 masqueradeAll: false
19 kubeProxyIPVSConfiguration:
20 scheduler: rr
21 excludeCIDRs: []
22 EOF
解釋：
- bindAddress: 監聽地址；
- clientConnection.kubeconfig: 連線 apiserver 的 kubeconfig 檔案；
- clusterCIDR: kube-proxy 根據 --cluster-cidr 判斷叢集內部和外部流量，指定 --cluster-cidr 或 --masquerade-all 選項後 kube-proxy 才會對訪問 Service IP 的請求做 SNAT；
- hostnameOverride: 引數值必須與 kubelet 的值一致，否則 kube-proxy 啟動後會找不到該 Node，從而不會建立任何 ipvs 規則；
- mode: 使用 ipvs 模式。
1.6 分發配置檔案
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for (( i=0; i < 6; i++ ))
4 do
5 echo ">>> ${ALL_NAMES[i]}"
6 sed -e "s/##ALL_NAME##/${ALL_NAMES[i]}/" -e "s/##ALL_IP##/${ALL_IPS[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${ALL_NAMES[i]}.yaml.template
7 scp kube-proxy-config-${ALL_NAMES[i]}.yaml.template root@${ALL_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yaml
8 done
1.7 建立kube-proxy的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# cat > kube-proxy.service <<EOF
4 [Unit]
5 Description=Kubernetes Kube-Proxy Server
6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
7 After=network.target
8
9 [Service]
10 WorkingDirectory=${K8S_DIR}/kube-proxy
11 ExecStart=/opt/k8s/bin/kube-proxy \\
12 --config=/etc/kubernetes/kube-proxy-config.yaml \\
13 --logtostderr=true \\
14 --v=2
15 Restart=on-failure
16 RestartSec=5
17 LimitNOFILE=65536
18
19 [Install]
20 WantedBy=multi-user.target
21 EOF
1.8 分發kube-proxy systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_name in ${ALL_NAMES[@]}
4 do
5 echo ">>> ${all_name}"
6 scp kube-proxy.service root@${all_name}:/etc/systemd/system/
7 done #分發system
二 啟動並驗證
2.1 啟動kube-proxy 服務
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 ssh root@${all_ip} "mkdir -p ${K8S_DIR}/kube-proxy"
7 ssh root@${all_ip} "modprobe ip_vs_rr"
8 ssh root@${all_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
9 done #啟動服務前必須先建立工作目錄
2.2 檢查kube-proxy 服務
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh
2 [root@k8smaster01 ~]# for all_ip in ${ALL_IPS[@]}
3 do
4 echo ">>> ${all_ip}"
5 ssh root@${all_ip} "systemctl status kube-proxy|grep Active"
6 done
2.3 檢視監聽埠
kube-proxy 監聽 10249 和 10256 埠：- 10249：對外提供 /metrics;
- 10256：對外提供 /healthz 的訪問。
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 ssh root@${all_ip} "sudo netstat -lnpt|grep kube-prox"
7 done
2.4 檢視ipvs 路由規則
1 [root@k8smaster01 ~]# cd /opt/k8s/work
2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
3 [root@k8smaster01 work]# for all_ip in ${ALL_IPS[@]}
4 do
5 echo ">>> ${all_ip}"
6 ssh root@${all_ip} "/usr/sbin/ipvsadm -ln"
7 done
可見所有通過 https 訪問 K8S SVC kubernetes 的請求都轉發到 kube-apiserver 節點的 6443 埠