K8S集群tls證書管理
在前文介紹的k8s master高可用實踐方案中，需要對kube-apiserver的證書進行更新，加入VIP和從節點的IP，然後重新下發證書。回顧K8S集群整個搭建過程中，最容易讓人懵圈的也就是配置證書環節，因此本文對K8S集群所用到的證書進行梳理一下。
一、根證書
ca.pem 根證書公鑰文件
ca-key.pem 根證書私鑰文件
ca.csr 證書簽名請求，用於交叉簽名或重新簽名
ca-config.json 使用cfssl工具生成其他類型證書需要引用的配置文件
ca.pem用於簽發後續其他的證書文件，因此ca.pem文件需要分發到集群中的每臺服務器上去。
證書生成命令，默認生成的證書有效期5年
# echo ‘{"CN":"CA","key":{"algo":"rsa","size":2048}}‘ | cfssl gencert -initca - | cfssljson -bare ca -
# echo ‘{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}‘ > ca-config.json
二、flannel證書
證書生成命令，默認生成的證書有效期5年
# cfssl gencert -ca=/etc/ssl/etcd/ca.pem -ca-key=/etc/ssl/etcd/ca-key.pem -config=/etc/ssl/etcd/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
這裏生成證書需要flanneld-csr.json文件
# cat flanneld-csr.json
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "FuZhou",
"L": "FuZhou",
"O": "k8s",
"OU": "System"
}
]
}flannel啟動文件配置
# cat /usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/usr/local/bin/flanneld -etcd-cafile=/etc/ssl/etcd/ca.pem -etcd-certfile=/etc/ssl/flanneld/flanneld.pem -etcd-keyfile=/etc/ssl/flanneld/flanneld-key.pem -etcd-endpoints=https://192.168.115.5,https://192.168.115.6:2379,https://192.168.115.7:2379 -etcd-prefix=/kubernetes/network ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service
三、etcd證書
1、服務端證書
server.pem etcd服務端證書公鑰文件
server-key.pem etcd服務端證書私鑰文件
server.csr 證書簽名請求
證書生成命令，默認生成的證書有效期5年
# export ADDRESS=192.168.115.5,192.168.115.6,192.168.115.7,vm1,vm2,vm3
# export NAME=server
# echo ‘{"CN":"‘$NAME‘","hosts":[""],"key":{"algo":"rsa","size":2048}}‘ | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
server.pem、server-key.pem文件用來etcd集群間通信加解密，因此所有的etcd服務器都需要有這兩個文件
# tail -15 /usr/lib/systemd/system/etcd.service
--initial-cluster-token=etcd-cluster-token --initial-cluster-state=new --cert-file=/etc/ssl/etcd/server.pem --key-file=/etc/ssl/etcd/server-key.pem --peer-cert-file=/etc/ssl/etcd/server.pem --peer-key-file=/etc/ssl/etcd/server-key.pem --trusted-ca-file=/etc/ssl/etcd/ca.pem --peer-trusted-ca-file=/etc/ssl/etcd/ca.pem --peer-client-cert-auth=true --client-cert-auth=true"
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target2、客戶端證書
client.pem etcd客戶端證書公鑰文件
client-key.pem etcd客戶端證書私鑰文件
client.csr 證書簽名請求
證書生成命令，默認生成的證書有效期5年
# export ADDRESS=
# export NAME=client
# echo ‘{"CN":"‘$NAME‘","hosts":[""],"key":{"algo":"rsa","size":2048}}‘ | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
client.pem、client-key.pem文件給etcdctl客戶端用來和etcd服務器進行通信，可根據實際需要進行配置。
# export ETCDCTL_API=3
# etcdctl --write-out=table --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem --cacert=/etc/ssl/etcd/ca.pem --endpoints=https://192.168.115.5:2379,https://192.168.115.6:2379,https://192.168.115.7:2379 member list
四、Master節點證書
Kube-apiserver證書
Kubernetes.pem kube-apiserver證書公鑰文件
Kubernetes-key.pem kube-apiserver證書私鑰文件
kuberentes.csr kube-apiserver證書簽名請求
證書生成命令，默認生成的證書有效期5年
# cfssl gencert -ca=/etc/ssl/etcd/ca.pem -ca-key=/etc/ssl/etcd/ca-key.pem -config=/etc/ssl/etcd/ca-config.json -profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes
這裏生成證書需要k8s-csr.json文件，其中定義了master節點的IP列表等信息
# cat k8s-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.115.4",
"192.168.115.5",
"192.168.115.6",
"192.168.115.7",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "FuZhou",
"L": "FuZhou",
"O": "k8s",
"OU": "System"
}
]
}五、Node節點證書
1、kube-proxy證書
Kube-proxy.pem kube-proxy證書公鑰文件
Kube-proxy-key.pem kube-proxy證書私鑰文件
Kube-proxy.csr kube-proxy證書簽名請求
證書生成命令，默認生成的證書有效期5年
# cfssl gencert -ca=/etc/ssl/etcd/ca.pem -ca-key=/etc/ssl/etcd/ca-key.pem -config=/etc/ssl/etcd/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
這裏生成證書需要kube-proxy-csr.json文件
# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "FuZhou",
"L": "FuZhou",
"O": "k8s",
"OU": "System"
}
]
}2、Kubelet證書
Kubelet-client.crt: kubectl客戶端證書公鑰文件
Kubelet-client.key: kubectl客戶端私鑰文件
Kubelet.crt：kubelet服務端證書公鑰文件
Kubelet.key：kubelet服務端證書私鑰文件> kubelet-client.crt 文件在 kubelet 完成 TLS bootstrapping 後生成，有效期為 1 年。此證書是由 controller manager 簽署的，此後 kubelet 將會加載該證書，用於與 apiserver 建立 TLS 通訊，同時使用該證書的 CN 字段作為用戶名，O 字段作為用戶組向 apiserver 發起其他請求。
kubelet.crt 文件在 kubelet 完成 TLS bootstrapping 後且沒有配置 --feature-gates=RotateKubeletServerCertificate=true 時才會生成；該文件為一個獨立於 apiserver CA 的自簽 CA 證書，有效期為 1 年；被用作 kubelet 10250 api 端口
關於kubelet首次啟動 TLS bootstrapping的介紹（先有雞還是先有蛋問題的解決方案）可參考文檔，https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/
六、配置證書自動續期
默認簽署kubectl客戶端和kubelet服務端證書只有 1 年有效期，如果想要調整證書有效期可以通過設置 kube-controller-manager 的?--experimental-cluster-signing-duration?參數實現，該參數默認值為?8760h0m0s。下面我們來介紹一下如何實現證書到期的自動續簽。這個問題如果處理不當，證書過期之後會出現所有的node節點連接不上的情況。
1、kcm服務，這裏為了方便測試，過期時間修改為30分鐘
# egrep ‘feature|experimental‘ /usr/lib/systemd/system/kube-controller-manager.service
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --experimental-cluster-signing-duration=30m0s # systemctl daemon-reload
# systemctl restart kube-controller-manager2、kubelet服務
配置完成刪掉Kubelet-client.crt、Kubelet-client.key、Kubelet.crt、Kubelet.key四個文件後重啟kubelet服務。
# egrep ‘feature|rotate‘ /usr/lib/systemd/system/kubelet.service
--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true --rotate-certificates=true # systemctl daemon-reload
# systemctl restart kubelet3、手工簽發證書
# kubectl create clusterrolebinding kubelet-clinet --clusterrole=system:node --user=system:anonymous如果缺少對system:anonymous用戶的授權，kubelet啟動的時候會報錯如下：
error: failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:anonymous" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope
# kubectl get csr
# kubectl certificate approve csr-fn946
# kubectl certificate approve csr-kwvg9 
node節點將會重新生成kubectl客戶端和kubelet服務端證書
4、配置自動簽發證書，在大規模集群下，這個配置是必須的
# cat rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
# kubectl create -f rbac.yaml# kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers
# kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
# kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes
這裏需要註意的是刪除掉Kubelet-client.crt、Kubelet-client.key兩個文件之後，啟動kubelet服務，首先會生成一個Kubelet-client.key文件，我們需要對這個證書的crs請求進行approve，否則node節點無法正常啟動。
其次，如果kubelet.kubeconfig文件中配置的client-certificate、client-key目錄位置和kubelet的啟動參數--cert-dir不一致，則kubelet.kubeconfig文件中的配置文件會被自動更新。
K8S集群tls證書管理