這個是全新部署，四個節點都要部署

kublet 運行在每個 worker 節點上，接收 kube-apiserver 發送的請求，管理 Pod 容器，
執行交互式命令，如 exec、run、logs 等。
kublet 啟動時自動向 kube-apiserver 註冊節點信息，內置的 cadvisor 統計和監控節點的
資源使用情況。
為確保安全，本文檔只開啟接收 https 請求的安全端口，對請求進行認證和授權，拒絕
未授權的訪問(如 apiserver、heapster)。

先把前面下載好的二進制執行文件分發到所有work節點

[root@k8s-master1 bin]# scp kubelet kubeadm kube-proxy? root@k8s-master1:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB?? 5.4MB/s?? 00:27? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 4.3MB/s?? 00:35? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB?? 3.3MB/s?? 00:15? ?
[root@k8s-master1 bin]# scp kubelet kubeadm kube-proxy? root@k8s-master2:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB? 48.6MB/s?? 00:03? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 7.1MB/s?? 00:21? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB? 24.5MB/s?? 00:02? ?
[root@k8s-master1 bin]# scp kubelet kubeadm kube-proxy? root@k8s-master3:/opt/k8s/bin/
kubelet? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 146MB?? 6.6MB/s?? 00:22? ?
kubeadm? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? 100%? 149MB?? 7.5MB/s?? 00:20? ?
kube-proxy? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%?? 49MB? 12.3MB/s?? 00:04? ?
[root@k8s-master1 bin]# scp kubelet kubeadm kube-proxy? root@k8s-node3:/opt/k8s/bin/
 

創建 kubelet bootstrap kubeconfig 文件
分別按順序跑:
k8s-master1
k8smaster2
k8smaster3

[root@k8s-master1 kubelet]# export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:k8s-master1 --kubeconfig ~/.kube/config)

[root@k8s-master1 kubelet]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.211.127:8443 --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Cluster "kubernetes" set.

[root@k8s-master1 kubelet]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
User "kubelet-bootstrap" set.

[root@k8s-master1 kubelet]# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Context "default" created.

[root@k8s-master1 kubelet]# kubectl config use-context default --kubeconfig=kubelet-bootstrap-k8s-master1.kubeconfig
Switched to context "default".
 

分發 bootstrap kubeconfig 文件到 worker 節點

[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master1.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
Permission denied, please try again.
[email protected]‘s password:
kubelet-bootstrap-k8s-master1.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master2.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
kubelet-bootstrap-k8s-master2.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet-bootstrap-k8s-master3.kubeconfig [email protected]:/etc/kubernetes/kubelet-bootstrap.kubeconfig
[email protected]‘s password:
kubelet-bootstrap-k8s-master3.kubeconfig? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100% 2087? ?? 2.0KB/s?? 00:00? ?
[root@k8s-master1 kubelet]#
 

創建和分發 kubelet 參數配置文件

[root@k8s-master1 kubelet]# cat kubelet.config.json.template
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/cert/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "##NODE_IP##",
"port": 10250,
"readOnlyPort": 0,
"cgroupDriver": "cgroupfs",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"clusterDomain": "${CLUSTER_DNS_DOMAIN}",
"clusterDNS": ["${CLUSTER_DNS_SVC_IP}"]
}
[root@k8s-master1 kubelet]#

address：API 監聽地址，不能為 127.0.0.1，否則 kube-apiserver、heapster 等不
能調用 kubelet 的 API；
readOnlyPort=0：關閉只讀端口(默認 10255)，等效為未指定；
authentication.anonymous.enabled：設置為 false，不允許匿名訪問 10250 端口；
authentication.x509.clientCAFile：指定簽名客戶端證書的 CA 證書，開啟 HTTP 證
書認證；
authentication.webhook.enabled=true：開啟 HTTPs bearer token 認證；
對於未通過 x509 證書和 webhook 認證的請求(kube-apiserver 或其他客戶端)，將
被拒絕，提示 Unauthorized；
authroization.mode=Webhook：kubelet 使用 SubjectAcce***eview API 查詢
kube-apiserver 某 user、group 是否具有操作資源的權限(RBAC)；
featureGates.RotateKubeletClientCertificate、
featureGates.RotateKubeletServerCertificate：自動 rotate 證書，證書的有效期取
決於 kube-controller-manager 的 --experimental-cluster-signing-duration 參數；
需要 root 賬戶運行；

分發並在各節點修改

[root@k8s-master1 kubelet]# scp kubelet.config.json.template root@k8s-master1:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.config.json.template root@k8s-master2:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.config.json.template root@k8s-master3:/etc/kubernetes/kubelet.config.json
kubelet.config.json.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 704? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.config.json.template root@k8s-node3:/etc/kubernetes/kubelet.config.json

修好
##NODE_IP## 改成節點真實IP

${CLUSTER_DNS_DOMAIN}
${CLUSTER_DNS_SVC_IP}

這兩個改成真實的參數，參考見下

[root@k8s-master1 kubelet]# echo ${CLUSTER_DNS_DOMAIN}
cluster.local.
[root@k8s-master1 kubelet]# echo ${CLUSTER_DNS_SVC_IP}
10.254.0.2
[root@k8s-master1 kubelet]#

創建和分發 kubelet systemd unit 文件

[root@k8s-master1 kubelet]# cat kubelet.service.template
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet ? --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig ? --cert-dir=/etc/kubernetes/cert ? --kubeconfig=/etc/kubernetes/kubelet.kubeconfig ? --config=/etc/kubernetes/kubelet.config.json ? --hostname-override=##nodename## ? --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest ? --allow-privileged=true ? --alsologtostderr=true ? --logtostderr=false ? --log-dir=/var/log/kubernetes ? --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
[root@k8s-master1 kubelet]#

註意的地方：

WorkingDirectory=/var/lib/kubelet ? ?##目錄默認沒有，手動去創建
--hostname-override=##nodename## ? ##nodename修改成在節點的名字

分發

[root@k8s-master1 kubelet]# scp kubelet.service.template root@k8s-master1:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.service.template root@k8s-master2:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.service.template root@k8s-master3:/etc/systemd/system/kubelet.service
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]# scp kubelet.service.template root@k8s-node3:/etc/systemd/system/kubelet.service
root@k8s-node3‘s password:
kubelet.service.template? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 100%? 753? ?? 0.7KB/s?? 00:00? ?
[root@k8s-master1 kubelet]#

分別去修改

--hostname-override=##nodename##

創建目錄

mkdir -p /var/lib/kubelet && chown -R k8s /var/lib/kubelet

Bootstrap Token Auth 和授予權限
kublet 啟動時查找配置的 --kubeletconfig 文件是否存在，如果不存在則使用 --bootstrapkubeconfig
向 kube-apiserver 發送證書簽名請求 (CSR)。
kube-apiserver 收到 CSR 請求後，對其中的 Token 進行認證（事先使用 kubeadm 創建
的 token），認證通過後將請求的 user 設置為 system:bootstrap:，group 設置為
system:bootstrappers，這一過程稱為 Bootstrap Token Auth。

[root@k8s-master1 kubernetes]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created

啟動起來了

[root@k8s-master1 kubelet]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
?? Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
?? Active: active (running) since Thu 2018-08-30 04:46:51 EDT; 6s ago
? ?? Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 22228 (kubelet)
?? Memory: 10.3M
?? CGroup: /system.slice/kubelet.service
? ? ? ? ?? └─22228 /opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kub...

Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.374637?? 22228 feature_gate.go:226] feature gates: &{{} map[Ro...true]}
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.390859?? 22228 mount_linux.go:211] Detected OS with systemd
Aug 30 04:46:51 k8s-master1 kubelet[22228]: W0830 04:46:51.396470?? 22228 cni.go:171] Unable to update cni config: No net.../net.d
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406764?? 22228 server.go:376] Version: v1.10.4
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406831?? 22228 feature_gate.go:226] feature gates: &{{} map[Ro...true]}
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406960?? 22228 plugins.go:89] No cloud provider specified.
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.406977?? 22228 server.go:492] No cloud provider specified: "" ...le: ""
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.407001?? 22228 bootstrap.go:58] Using bootstrap kubeconfig to ...g file
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.498673?? 22228 csr.go:105] csr for this node already exists, reusing
Aug 30 04:46:51 k8s-master1 kubelet[22228]: I0830 04:46:51.507675?? 22228 csr.go:113] csr for this node is still valid
Hint: Some lines were ellipsized, use -l to show in full.

kubelet 啟動後使用 --bootstrap-kubeconfig 向 kube-apiserver 發送 CSR 請求，當這個
CSR 被 approve 後，kube-controller-manager 為 kubelet 創建 TLS 客戶端證書、私鑰
和 --kubeletconfig 文件。
註意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --
cluster-signing-key-file 參數，才會為 TLS Bootstrap 創建證書和私鑰。

[root@k8s-master1 kubelet]# kubectl get csr
NAME? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? AGE? ? ?? REQUESTOR? ? ? ? ? ? ? ?? CONDITION
node-csr-4lLI6VjKHHWjZg4je3Ht3mgkyc0kSDALWgqyE6hJGLY?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-BX_rIIl3T80GWXCZqCQISgB2BWKXd_-QuD04IfXyvBU?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-BhI2aoEZzt8UlcSevQr8RQ9tY4ATbawpr3GklGbkdYI?? 54s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-CYy34cOnA7RStasf8ieh9ZF5crDLmTFbvDOZV7UaulI?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-G4fpzkI_gkD9R7LUh1fOHMBllMCTnIzfcWYUhcjbNLQ?? 28s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-GtBzeHjXzw0FThw7SwAQRq7_uWO_LmJutmAKOU19lpM?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-IMzMrDG99ht6FRazQyfq4XFmG0MU0iN7rFj87dJ_LO0?? 6m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-Ne6k_9kYNM5xZPzlMIMOiew6KYbCccgEoGEsD-A2mDI?? 59s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-S3MvbCy6G8vyMmZxPxHtSj7yXWsMKiTFhiEolNhbOcc?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-TobXYGLVUitHRfAJD3cy1uwLbD9xeLRqfVKRWcaqzG8?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-XCHccj91PEcvcgtoYIlUTVwjPntZ1QJ3x0FwaiKiaBQ?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-XWCrqdKkPfKiG20VpU8cn9N8ZRcOWlbfhPr8LMaW_PU?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-_Sp69LiFaATOGVn9fmAnOLHweAWwoVzeP9U0AxtsLPE?? 4m? ? ? ? system:bootstrap:7beznt?? Pending
node-csr-b71vB9tiCT7Ru5q6LQco_nb_hbIABmcDPmNi7fH7Vn8?? 1m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-czzY0kNjKg_6OAcU8m2dRzVt2KR9zY3FQ31t1QE3tXk?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-oA3SifuLsmgSMkZyIN9dJhE66iuMXCzciaLDWH3pl8E?? 57s? ? ?? system:bootstrap:m435c8?? Pending
node-csr-pWeUuvcTZCGqq1sh0KufCNzziyCYfhh-KUB_WAC2lpw?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-uWlqsUKKcVd_HQIMYBHusZS8hJc9yAntfE7qpGNJnSg?? 3m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-wfcltVjp2D_nzjRu7PdnB74L4JlXTFWfaumRnMAEmDg?? 5m? ? ? ? system:bootstrap:3lb82j?? Pending
node-csr-zaniEi7eNGTuzIherUJbNIdPAic1EnB1tKAAGvuzoAc?? 2m? ? ? ? system:bootstrap:m435c8?? Pending
node-csr-zggzAUVrryNXFp49lytoSZYe0qBYOd4Jz5Fa4WODeKQ?? 1m? ? ? ? system:bootstrap:m435c8?? Pending

approve kubelet CSR 請求
可以手動或自動 approve CSR 請求。推薦使用自動的方式，因為從 v1.8 版本開始，可
以自動輪轉approve csr 後生成的證書。

[root@k8s-master1 kubelet]# kubectl certificate approve node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
certificatesigningrequest.certificates.k8s.io "node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk" approved

[root@k8s-master1 kubelet]# kubectl describe csr node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
Name:? ? ? ? ? ? ?? node-csr--67xqDb2wzwOWPy9wzdbQs6XQwIf67skc43jRrpGwLk
Labels:? ? ? ? ? ?? <none>
Annotations:? ? ? ? <none>
CreationTimestamp:? Thu, 30 Aug 2018 04:51:10 -0400
Requesting User:? ? system:bootstrap:m435c8
Status:? ? ? ? ? ?? Approved,Issued
Subject:
? ? ? ?? Common Name:? ? system:node:k8s-node3
? ? ? ?? Serial Number:?
? ? ? ?? Organization:?? system:nodes
Events:? <none>
[root@k8s-master1 kubelet]#

Requesting User ：請求 CSR 的用戶，kube-apiserver 對它進行認證和授權；
Subject ：請求簽名的證書信息；
證書的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kubeapiserver
的 Node 授權模式會授予該證書的相關權限

自動 approve CSR 請求
創建三個 ClusterRoleBinding，分別用於自動 approve client、renew client、renew
server 證書：

[root@k8s-master1 kubelet]# cat csr-crb.yaml
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: auto-approve-csrs-for-group
subjects:
- kind: Group
? name: system:bootstrappers
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
? apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: node-client-cert-renewal
subjects:
- kind: Group
? name: system:nodes
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
? apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
? resources: ["certificatesigningrequests/selfnodeserver"]
? verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
? name: node-server-cert-renewal
subjects:
- kind: Group
? name: system:nodes
? apiGroup: rbac.authorization.k8s.io
roleRef:
? kind: ClusterRole
? name: approve-node-server-renewal-csr
? apiGroup: rbac.authorization.k8s.io
[root@k8s-master1 kubelet]#

[root@k8s-master1 kubelet]# kubectl apply -f csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io "auto-approve-csrs-for-group" created
clusterrolebinding.rbac.authorization.k8s.io "node-client-cert-renewal" created
clusterrole.rbac.authorization.k8s.io "approve-node-server-renewal-csr" created
clusterrolebinding.rbac.authorization.k8s.io "node-server-cert-renewal" created

等待一段時間(1-10 分鐘)，節點的 CSR 都被自動 approve：

[root@k8s-master1 kubelet]# kubectl get csr
NAME? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? AGE? ? ?? REQUESTOR? ? ? ? ? ? ? ?? CONDITION
csr-7685f? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 53s? ? ?? system:node:k8s-master2?? Approved,Issued
csr-8qkxl? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1m? ? ? ? system:node:k8s-node3? ?? Approved,Issued
csr-n56tk? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 44s? ? ?? system:node:k8s-master1?? Approved,Issued
csr-p8h92? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28s? ? ?? system:node:k8s-master3?? Pending

nodes起來了

[root@k8s-master2 kubernetes]# kubectl get nodes
NAME? ? ? ? ? STATUS? ? ROLES? ?? AGE? ? ?? VERSION
k8s-master1?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-master2?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-master3?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
k8s-node3? ?? Ready? ?? <none>? ? 2m? ? ? ? v1.10.4
[root@k8s-master2 kubernetes]#
[root@k8s-master2 kubernetes]# netstat -lnpt|grep kubelet
tcp? ? ? ? 0? ? ? 0 192.168.211.129:10250?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
tcp? ? ? ? 0? ? ? 0 192.168.211.129:4194? ? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
tcp? ? ? ? 0? ? ? 0 127.0.0.1:10248? ? ? ?? 0.0.0.0:*? ? ? ? ? ? ?? LISTEN? ? ? 20752/kubelet? ? ??
[root@k8s-master2 kubernetes]#

cadvisor 和 metrics
cadvisor 統計所在節點各容器的資源(CPU、內存、磁盤、網卡)使用情況，分別在自己
的 http web 頁面(4194 端口)和 10250 以 promehteus metrics 的形式輸出。
瀏覽器訪問 http://192.168.211.128:4194/containers/ 可以查看到 cadvisor 的監控頁面：

啟動服務報錯

[root@k8s-master1 kubernetes]# systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet?

F0830 04:05:24.413219?? 10947 server.go:233] failed to run Kubelet: cannot create certificate signing request: Post http://192.168.211.127/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.211.127:80: getsockopt: connection refused
goroutine 1 [running]:

這個報錯是因為
kubelet-bootstrap.kubeconfig 文件的配置不對導致

部署k8s ssl集群實踐13:work節點配置kubelet