搭建K8S高可用叢集
阿新 • • 發佈：2021-06-13
kubeadm是官方社群推出的一個用於快速部署kubernetes叢集的工具。
這個工具能通過兩條指令完成一個kubernetes叢集的部署：
# 建立一個 Master 節點
$ kubeadm init
# 將一個 Node 節點加入到當前叢集中
$ kubeadm join <Master節點的IP和埠 >
1. 安裝要求
在開始之前，部署Kubernetes叢集機器需要滿足以下幾個條件：
- 一臺或多臺機器，作業系統 CentOS7.x-86_x64
- 硬體配置：2GB或更多RAM，2個CPU或更多CPU，硬碟30GB或更多
- 可以訪問外網，需要拉取映象，如果伺服器不能上網，需要提前下載映象並匯入節點
- 禁止swap分割槽
2. 準備環境
| 角色 | IP |
|---|---|
| master1 | 192.168.44.155 |
| master2 | 192.168.44.156 |
| node1 | 192.168.44.157 |
| VIP（虛擬ip） | 192.168.44.158 |
# 關閉防火牆
systemctl stop firewalld
systemctl disable firewalld
# 關閉selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 臨時
# 關閉swap
swapoff -a # 臨時
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
# 根據規劃設定主機名
hostnamectl set-hostname <hostname>
# 在master新增hosts
cat >> /etc/hosts << EOF
192.168.44.158 master.k8s.io k8s-vip
192.168.44.155 master01.k8s.io master1
192.168.44.156 master02.k8s.io master2
192.168.44.157 node01.k8s.io node1
EOF
# 將橋接的IPv4流量傳遞到iptables的鏈
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system # 生效
# 時間同步
yum install ntpdate -y
ntpdate time.windows.com
3. 所有master節點部署keepalived
3.1 安裝相關包和keepalived
yum install -y conntrack-tools libseccomp libtool-ltdl
yum install -y keepalived
3.2配置master節點
master1節點配置
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 250
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.44.158
}
track_script {
check_haproxy
}
}
EOF
master2節點配置
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.44.158
}
track_script {
check_haproxy
}
}
EOF
3.3 啟動和檢查
在兩臺master節點都執行
# 啟動keepalived
$ systemctl start keepalived.service
設定開機啟動
$ systemctl enable keepalived.service
# 檢視啟動狀態
$ systemctl status keepalived.service
啟動後檢視master1的網絡卡資訊
ip a s ens33
4. 部署haproxy
4.1 安裝
yum install -y haproxy
4.2 配置
兩臺master節點的配置均相同，配置中聲明瞭後端代理的兩個master節點伺服器，指定了haproxy執行的埠為16443等，因此16443埠為叢集的入口
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master01.k8s.io 192.168.44.155:6443 check
server master02.k8s.io 192.168.44.156:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
4.3 啟動和檢查
兩臺master都啟動
# 設定開機啟動
$ systemctl enable haproxy
# 開啟haproxy
$ systemctl start haproxy
# 檢視啟動狀態
$ systemctl status haproxy
檢查埠
netstat -lntup|grep haproxy
5. 所有節點安裝Docker/kubeadm/kubelet
Kubernetes預設CRI（容器執行時）為Docker，因此先安裝Docker。
5.1 安裝Docker
$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
$ yum -y install docker-ce-18.06.1.ce-3.el7
$ systemctl enable docker && systemctl start docker
$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
5.2 新增阿里雲YUM軟體源
$ cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
5.3 安裝kubeadm，kubelet和kubectl
由於版本更新頻繁，這裡指定版本號部署：
$ yum install -y kubelet-1.16.3 kubeadm-1.16.3 kubectl-1.16.3
$ systemctl enable kubelet
6. 部署Kubernetes Master
6.1 建立kubeadm配置檔案
在具有vip的master上操作，這裡為master1
$ mkdir /usr/local/kubernetes/manifests -p
$ cd /usr/local/kubernetes/manifests/
$ vi kubeadm-config.yaml
apiServer:
certSANs:
- master1
- master2
- master.k8s.io
- 192.168.44.158
- 192.168.44.155
- 192.168.44.156
- 127.0.0.1
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.3
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.1.0.0/16
scheduler: {}
6.2 在master1節點執行
$ kubeadm init --config kubeadm-config.yaml
按照提示配置環境變數，使用kubectl工具：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes
$ kubectl get pods -n kube-system
按照提示儲存以下內容，一會要使用：
kubeadm join master.k8s.io:16443 --token jv5z7n.3y1zi95p952y9p65 \
--discovery-token-ca-cert-hash sha256:403bca185c2f3a4791685013499e7ce58f9848e2213e27194b75a2e3293d8812 \
--control-plane
檢視叢集狀態
kubectl get cs
kubectl get pods -n kube-system
7.安裝叢集網路
從官方地址獲取到flannel的yaml，在master1上執行
mkdir flannel
cd flannel
wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
安裝flannel網路
kubectl apply -f kube-flannel.yml
檢查
kubectl get pods -n kube-system
8、master2節點加入叢集
8.1 複製金鑰及相關檔案
從master1複製金鑰及相關檔案到master2
# ssh [email protected] mkdir -p /etc/kubernetes/pki/etcd
# scp /etc/kubernetes/admin.conf [email protected]:/etc/kubernetes
# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} [email protected]:/etc/kubernetes/pki
# scp /etc/kubernetes/pki/etcd/ca.* [email protected]:/etc/kubernetes/pki/etcd
8.2 master2加入叢集
執行在master1上init後輸出的join命令,需要帶上引數--control-plane表示把master控制節點加入叢集
kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba --control-plane
檢查狀態
kubectl get node
kubectl get pods --all-namespaces
5. 加入Kubernetes Node
在node1上執行
向叢集新增新節點，執行在kubeadm init輸出的kubeadm join命令：
kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba
叢集網路重新安裝，因為添加了新的node節點
檢查狀態
kubectl get node
kubectl get pods --all-namespaces
7. 測試kubernetes叢集
在Kubernetes叢集中建立一個pod，驗證是否正常執行：
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
訪問地址：http://NodeIP:Port