利用kubeadm快速搭建Kubernetes學習環境
1. 前言
本文講述如何使用kubeadm快速搭建學習環境,為後面即將開始的Kubernetes的管理及使用做準備,並不是在生產環境中安裝的高可用叢集環境,生產環境的規劃及安裝請關注後面的文件。
2. 安裝環境介紹
2.1 節點規劃
部署k8s叢集的節點按照用途可以劃分為如下2類角色:
-
master:叢集的master節點,叢集的初始化節點,基礎配置不低於2C4G
-
slave:叢集的slave節點,可以多臺,基礎配置不低於2C4G
為了便於比較完整的學習Kubernetes各元件及功能,本文將部署一臺master+2臺slave,節點規劃如下:
主機名 | 節點IP | 角色 | 部署元件 |
k8s-master | 172.16.0.142 | master | etcd, kube-apiserver, kube-controller-manager, kubectl, kubeadm, kubelet, kube-proxy, flannel |
k8s-slave1 | 172.16.0.143 | slave | kubectl, kubelet, kube-proxy, flannel |
k8s-slave2 | 172.16.0.144 | slave | kubectl, kubelet, kube-proxy, flannel |
2.2 元件版本
本學習環境所使用到的Kubernetes的各元件及版本如下:
版本 | 說明 | |
---|---|---|
CentOS | 7.9.2009 | |
Kernel | Linux 3.10.0-1160.25.1.el7.x86_64 | |
etcd | 3.4.13-0 | 使用容器方式部署,預設資料掛載到本地路徑 |
coredns | 1.7.0 | |
kubeadm | v1.19.8 | |
kubectl | v1.19.8 | |
kubelet | v1.19.8 | |
kube-proxy | v1.19.8 | |
flannel | v0.11.0 |
3. 作業系統配置
3.1 設定hosts解析
k8s-master,k8s-slave
)均需執行
-
修改hostname hostname必須只能包含小寫字母、數字、","、"-",且開頭結尾必須是小寫字母或數字
# 在master節點,設定master節點的hostname
hostnamectl set-hostname k8s-master
# 在slave-1節點,設定slave1節點的hostname
hostnamectl set-hostname k8s-slave1
# 在slave-2節點,設定slave2節點的hostname
hostnamectl set-hostname k8s-slave2
-
新增hosts解析
cat >>/etc/hosts<<EOF
172.16.0.142 k8s-master
172.16.0.143 k8s-slave1
172.16.0.144 k8s-slave2
EOF
3.2 調整系統配置
k8s-master,k8s-slave
)需要執行
本章下述操作均以k8s-master為例,其他節點均是相同的操作(ip和hostname的值換成對應機器的真實值)
-
設定安全組開放埠
如果節點間無安全組限制(內網機器間可以任意訪問),可以忽略,否則,至少保證如下埠可通: k8s-master節點:TCP:6443,2379,2380,60080,60081UDP協議埠全部開啟 k8s-slave節點:UDP協議埠全部開啟
-
設定iptables
iptables -P FORWARD ACCEPT
-
關閉swap
swapoff -a
# 防止開機自動掛載 swap 分割槽
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
-
關閉selinux和防火牆
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
systemctl disable firewalld && systemctl stop firewalld
-
修改核心引數
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.max_map_count=262144
EOF
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
-
設定yum源
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum clean all && yum makecache
3.3 安裝docker並配置
操作節點: 所有節點
# 檢視所有的可用版本
yum list docker-ce --showduplicates | sort -r
#安裝舊版本 yum install docker-ce-cli-18.09.9-3.el7 docker-ce-18.09.9-3.el7
yum install docker-ce-20.10.6 -y
# 配置docker加速
mkdir -p /etc/docker
vi /etc/docker/daemon.json
{
"insecure-registries": [
"172.16.0.142:5000"
],
"registry-mirrors" : [
"https://8xpk5wnt.mirror.aliyuncs.com"
]
}
# 啟動docker
systemctl enable docker && systemctl start docker
4. Kubernetes部署
4.1 安裝kubeadm, kubelet及kubectl
操作節點: 所有的master和slave節點(k8s-master,k8s-slave
) 需要執行
yum install -y kubelet-1.19.8 kubeadm-1.19.8 kubectl-1.19.8 --disableexcludes=kubernetes
# 檢視kubeadm 版本
kubeadm version
# 設定kubelet開機啟動
systemctl enable kubelet
4.2 初始化配置檔案
操作節點: 只在master節點(k8s-master
)執行
kubeadm config print init-defaults > kubeadm.yaml
cat kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.16.0.142 # apiserver地址,因為單master,所以配置master的節點內網IP,可以理解為master節點的VIP,代理KPIserver的6443介面
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 修改成阿里映象源
kind: ClusterConfiguration
kubernetesVersion: v1.19.8
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16 # Pod 網段,flannel外掛需要使用這個網段,要確保和docker以及宿主機的網段不能衝突
serviceSubnet: 10.96.0.0/12
scheduler: {}
對於上面的資源清單的文件比較雜,要想完整了解上面的資源物件對應的屬性,可以檢視對應的 godoc 文件,地址: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2。
4.3 提前下載映象
k8s-master
)執行
# 檢視需要使用的映象列表,若無問題,將得到如下列表
kubeadm config images list --config kubeadm.yaml
registry.aliyuncs.com/google_containers/kube-apiserver:v1.16.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.16.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.16.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.16.0
registry.aliyuncs.com/google_containers/pause:3.1
registry.aliyuncs.com/google_containers/etcd:3.3.15-0
registry.aliyuncs.com/google_containers/coredns:1.19.8
# 提前下載映象到本地
kubeadm config images pull --config kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.16.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.3.15-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.19.8
4.4 初始化master節點
操作節點:只在master節點(k8s-master
)執行
kubeadm init --config kubeadm.yaml
若初始化成功後,最後會提示如下資訊:
...
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.0.142:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1c4305f032f4bf534f628c32f5039084f4b103c922ff71b12a5f0f98d1ca9a4f
接下來按照上述提示資訊操作,配置kubectl客戶端的認證
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
⚠️注意::
此時使用 kubectl get nodes檢視節點應該處於notReady狀態,因為還未配置網路外掛
若執行初始化過程中出錯,根據錯誤資訊調整後,執行kubeadm reset後再次執行init操作即可
初始化出錯排查方向:1)etcd服務有沒有啟動;2)apiserver是否正常啟動;3)kubectl是否正常;
5 新增slave節點到叢集中
操作節點:所有的slave節點(k8s-slave
)需要執行 在每臺slave節點,執行如下命令,該命令是在kubeadm init成功後提示資訊中打印出來的,需要替換成實際init後打印出的命令。
kubeadm join 172.16.0.142:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1c4305f032f4bf534f628c32f5039084f4b103c922ff71b12a5f0f98d1ca9a4f
如果忘記新增命令,可以通過如下命令生成:
$ kubeadm token create --print-join-command
6 安裝flannel網路外掛
操作節點:只在master節點(k8s-master
)執行
-
下載flannel的yaml檔案
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
-
修改配置,指定網絡卡名稱,大概在檔案的190行,新增一行配置:
vi kube-flannel.yml
...
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0 # 如果機器存在多網絡卡的話,指定內網網絡卡的名稱,預設不指定的話會找第一塊網絡卡,如果只有一塊網絡卡,這地方不加也可以的。
resources:
requests:
cpu: "100m"
...
-
執行安裝flannel網路外掛
# 先拉取映象,此過程國內速度比較慢
docker pull quay.io/coreos/flannel:v0.14.0-amd64
# 執行flannel安裝
kubectl apply -f kube-flannel.yml
7 Kubernetes各節點配置優化
7.1 配置master節點可排程
操作節點:k8s-master
預設部署成功後,master節點無法排程業務pod,如需設定master節點也可以參與pod的排程,需執行:
kubectl taint node k8s-master node-role.kubernetes.io/master:NoSchedule-
# node/k8s-master untainted
# 配置master節點不可排程
kubectl taint nodes k8s-master node-role.kubernetes.io/master=:NoSchedule
# node/k8s-master tainted
# 如果不執行讓master可排程,則會報錯:
# Warning FailedScheduling 2m13s default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) didn't match node selector.
課程後期會部署系統元件到master節點,因此,此處建議設定k8s-master節點為可排程
7.2設定kubectl自動補全
作用:在輸入kubectl之後,按tab鍵,會自動給出可選項。
操作節點:k8s-master
# 在執行如下命令即可
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
8 驗證叢集
k8s-master
)執行
$ kubectl get nodes # 觀察叢集節點是否全部Ready
建立測試nginx服務
$ kubectl run test-nginx --image=nginx:alpine
檢視pod是否建立成功,並訪問pod ip測試是否可用
kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-nginx-5bd8859b98-5nnnw 1/1 Running 0 9s 10.244.1.2 k8s-slave1 <none> <none>
curl 10.244.1.2
...
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
9 部署Dashboard
# 推薦
# 下載部署yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
# 編輯檔案內容,將Kubernetes-Dashboard部署為NodePort型別
vi recommended.yaml
# 修改Service為NodePort型別,檔案的45行上下
......
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
type: NodePort # 加上type=NodePort變成NodePort型別的服務
......
-
檢視訪問地址,本例為30133埠
# 下載所需映象
# 部署Kubernetes-Dashboard
kubectl apply -f recommended.yaml
kubectl -n kubernetes-dashboard get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.107.120.70 <none> 8000/TCP 16d
kubernetes-dashboard NodePort 10.107.202.113 <none> 443:31590/TCP 16d
-
使用瀏覽器訪問 https://172.16.0.142:31590/,其中172.16.0.142為master節點的外網ip地址,chrome目前由於安全限制,測試訪問不了,使用firefox可以進行訪問。
-
建立ServiceAccount進行訪問
vi dashboard-admin.conf
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: admin
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kubernetes-dashboard
kubectl apply -f dashboard-admin.conf
kubectl -n kubernetes-dashboard get secret |grep admin-token
admin-token-fqdpf kubernetes.io/service-account-token 3 7m17s
# 使用該命令拿到token,然後貼上到
kubectl -n kubernetes-dashboard get secret admin-token-fqdpf -o jsonpath={.data.token}|base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik1rb2xHWHMwbWFPMjJaRzhleGRqaExnVi1BLVNRc2txaEhETmVpRzlDeDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi1mcWRwZiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjYyNWMxNjJlLTQ1ZG...
5. Kubernetes叢集環境清理
# 在全部叢集節點執行
kubeadm reset
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /run/flannel/subnet.env
rm -rf /var/lib/cni/
mv /etc/kubernetes/ /tmp
mv /var/lib/etcd /tmp
mv ~/.kube /tmp
iptables -F
iptables -t nat -F
ipvsadm -C
ip link del kube-ipvs0
ip link del dummy0
人們永遠沒有足夠的時間把它做好,但永遠有足夠的時間重新來過。 可是,因為並不是總有機會重做一遍,你必須做得更好,換句話說, 人們永遠沒有足夠的時間去考慮到底是不是想要它,但永遠有足夠的時間去為之後悔。 ★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★ 淺掘千口井,不如深挖一口井!當知識支撐不了野心時,那就靜下心來學習吧!運維技術交流QQ群:618354452
個人微信公眾號,定期釋出技術文章和運維感悟。歡迎大家關注交流。