centos7部署k8s Cluster
我們將部署三個節點的 Kubernetes Cluster
k8s-m1 是 Master，k8s-n1 和 k8s-n2 是 Node。
所有節點的操作系統均為centos1708 minimal ，2G內存，2顆CPU。當然其他 Linux 也是可以的。
1、環境初始化
1.1 給各主機設置主機名稱
hostnamectl set-hostname k8s-m1
hostnamectl set-hostname k8s-n1
hostnamectl set-hostname k8s-n2
1.2 給各主機設置靜態域名解釋：
cat <<EOF > /etc/hosts
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.92.159 k8s-m1
192.168.92.160 k8s-n1
192.168.92.161 k8s-n2
EOF
1.3 k8s-m1上執行ssh免密碼登陸配置
ssh-keygen #一路回車即可
ssh-copy-id k8s-n1
ssh-copy-id k8s-n2
1.4 各主機還需配置：停防火墻、關閉Selinux、關閉Swap、設置內核、K8S的yum源、安裝依賴包、配置ntp
systemctl stop firewalld
systemctl disable firewalld
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
swapoff -a && sysctl -w vm.swappiness=0
sed -i ‘s/.swap./#&/‘ /etc/fstab
設定/etc/sysctl.conf的系統參數。
cat <<EOF > /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim ntpdate libseccomp libtool-ltdl
systemctl enable ntpdate.service
echo ‘/30 * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1‘ > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
(/etc/security/limits.conf 是 Linux 資源使用配置文件，用來限制用戶對系統資源的使用)
echo " soft nofile 65536" >> /etc/security/limits.conf
echo " hard nofile 65536" >> /etc/security/limits.conf
echo " soft nproc 65536" >> /etc/security/limits.conf
echo " hard nproc 65536" >> /etc/security/limits.conf
echo " soft memlock unlimited" >> /etc/security/limits.conf
echo " hard memlock unlimited" >> /etc/security/limits.conf
1.5 配置完後重啟一次
2、所有節點安裝配置docker
2.1 安裝docker（kubeadm目前支持docker最高版本是17.03.x）
wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-17.09.0.ce-1.el7.centos.x86_64.rpm
yum localinstall -y docker-ce-17.09.0.ce-1.el7.centos.x86_64.rpm
2.2 修改配置文件 vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --registry-mirror=https://ms3cfraz.mirror.aliyuncs.com
2.3 啟動docker
systemctl daemon-reload
systemctl enable docker && systemctl start docker
3、安裝 kubelet、kubeadm 和 kubectl
3.1 所有節點安裝kubelet kubeadm kubectl
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet
3.2 所有節點修改kubelet配置文件
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
#修改這一行
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#添加這一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/k8sth/pause-amd64:3.0"
3.3 所有節點修改完配置文件一定要重新加載配置
systemctl daemon-reload
systemctl enable kubelet
3.4 命令補全
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
4、下載K8S相關鏡像
安裝時候最常見的就是time out，因為K8S鏡像在國外，所以我們在前面就說到了提前把他下載下來。
#!/bin/bash
images=(kube-proxy-amd64:v1.10.0 kube-scheduler-amd64:v1.10.0 kube-controller-manager-amd64:v1.10.0 kube-apiserver-amd64:v1.10.0
etcd-amd64:3.1.12 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.8 k8s-dns-kube-dns-amd64:1.14.8
k8s-dns-dnsmasq-nanny-amd64:1.14.8)
for imageName in ${images[@]} ; do
docker pull keveon/$imageName
docker tag keveon/$imageName k8s.gcr.io/$imageName
docker rmi keveon/$imageName
done
上面的shell腳本主要做了3件事，下載各種需要用到的容器鏡像、重新打標記為符合k8s命令規範的版本名稱、清除舊的容器鏡像。
提示：鏡像版本一定要和kubeadm安裝的版本一致，否則會出現time out問題。
5、初始化 Master
5.1 執行上述shell腳本，等待下載完成後，執行kubeadm init
kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16
提示：選項–kubernetes-version=v1.10.0是必須的，否則會因為訪問google網站被墻而無法執行命令。這裏使用v1.10.0版本，剛才前面也說到了下載的容器鏡像版本必須與K8S版本一致否則會出現time out。
--pod-network-cidr 指定 Pod 網絡的範圍。Kubernetes 支持多種網絡方案，而且不同網絡方案對 --pod-network-cidr 有自己的要求，這裏設置為 10.244.0.0/16 是因為我們將使用 flannel 網絡方案，必須設置成這個 CIDR。在後面的實踐中我們會切換到其他網絡方案，比如 Canal。
上面的命令大約需要1分鐘的過程，期間可以觀察下tail -f /var/log/message日誌文件的輸出，掌握該配置過程和進度。上面最後一段的輸出信息保存一份，後續添加工作節點還要用到。
[root@k8s-m1 ~]# kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.09.0-ce. Max validated version: 17.03
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 23.503191 seconds
[uploadconfig]?Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node k8s-m1 as master by adding a label and a taint
[markmaster] Master k8s-m1 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 0ivtjg.t670zkbmni8b8qsw
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.92.159:6443 --token 0ivtjg.t670zkbmni8b8qsw --discovery-token-ca-cert-hash sha256:2b7fd25ced027017b8be24c77dd0472d36c5cdf9dc95ca5fc85f63d7ac4d00c5
初始化過程如下：
① kubeadm 執行初始化前的檢查。
② 生成 token 和證書。
③ 生成 KubeConfig 文件，kubelet 需要這個文件與 Master 通信。
④ 安裝 Master 組件，會從 goolge 的 Registry 下載組件的 Docker 鏡像，這一步可能會花一些時間，主要取決於網絡質量。
⑤ 安裝附加組件 kube-proxy 和 kube-dns。
⑥ Kubernetes Master 初始化成功。
⑦ 提示如何配置 kubectl，後面會實踐。
⑧ 提示如何安裝 Pod 網絡，後面會實踐。
⑨ 提示如何註冊其他節點到 Cluster，後面會實踐。
5.2 初始化失敗後處理辦法
kubeadm reset
#或
rm -rf /etc/kubernetes/.conf
rm -rf /etc/kubernetes/manifests/.yaml
docker ps -a |awk ‘{print $1}‘ |xargs docker rm -f
systemctl stop kubelet
6、配置kubectl認證信息
對於非root用戶
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
對於root用戶
export KUBECONFIG=/etc/kubernetes/admin.conf
也可以直接放到~/.bash_profile
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
7、安裝flannel網絡
mkdir -p /etc/cni/net.d/
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{
“name”: “cbr0”,
“type”: “flannel”,
“delegate”: {
“isDefaultGateway”: true
}
}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
8、讓k8s-n1、k8s-n2加入集群
在k8s-n1和k8s-n2節點上分別執行kubeadm join命令，加入集群：
kubeadm join 192.168.92.159:6443 --token 0ivtjg.t670zkbmni8b8qsw --discovery-token-ca-cert-hash sha256:2b7fd25ced027017b8be24c77dd0472d36c5cdf9dc95ca5fc85f63d7ac4d00c5
提示：細心的童鞋應該會發現，這段命令其實就是前面K8S Matser安裝成功後我讓你們保存的那段命令。
默認情況下，Master節點不參與工作負載，但如果希望安裝出一個All-In-One的k8s環境，則可以執行以下命令，讓Master節點也成為一個Node節點：
kubectl taint nodes --all node-role.kubernetes.io/master-
9、驗證K8S Master是否搭建成功
查看節點狀態
kubectl get nodes
查看pods狀態
kubectl get pods --all-namespaces
查看K8S集群狀態
kubectl get cs
centos7部署k8s Cluster