Kubernetes學習之路（一）之Kubeadm部署K8S集群
阿新 • • 發佈：2018-11-01
chown you eof onf 訪問 perf 處的 iyu inter 一個星期會超過多少閱讀量呢？？發布一篇，看看是否重新在51上寫學習博文，有老鐵支持嘛？？
使用kubeadm部署集群
| 節點名稱 | ip地址 | 部署說明 | Pod 網段 | Service網段 | 系統說明 |
|---|---|---|---|---|---|
| k8s-master | 192.168.56.11 | docker、kubeadm、kubectl、kubelet | 10.244.0.0/16 |
10.96.0.0/12 |
Centos 7.4 |
| k8s-node01 | 192.168.56.12 | docker、kubeadm、kubelet | 10.244.0.0/16 |
10.96.0.0/12 |
Centos 7.4 |
| k8s-node02 | 192.168.56.13 | docker、kubeadm、kubelet | 10.244.0.0/16 |
10.96.0.0/12 |
Centos 7.4 |
1、配置kubernetes源
[root@k8s-master ~]# cd /etc/yum.repos.d/ 配置阿裏雲的源：https://opsx.alibaba.com/mirror [root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #配置dokcer源 [root@k8s-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo #配置kubernetes源 > [kubernetes] > name=Kubernetes > baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ > enabled=1 > gpgcheck=1 > repo_gpgcheck=1 > gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg > EOF [root@k8s-master yum.repos.d]# yum repolist #查看可用源
將源拷貝到node01和node02節點
[root@k8s-master yum.repos.d]# scp kubernetes.repo docker-ce.repo k8s-node1:/etc/yum.repos.d/ kubernetes.repo 100% 276 276.1KB/s 00:00 docker-ce.repo 100% 2640 1.7MB/s 00:00 [root@k8s-master yum.repos.d]# scp kubernetes.repo docker-ce.repo k8s-node2:/etc/yum.repos.d/ kubernetes.repo 100% 276 226.9KB/s 00:00 docker-ce.repo 100% 2640 1.7MB/s 00:00
2、Master節點安裝docker、kubelet、kubeadm、還有命令行工具kubectl
[root@k8s-master yum.repos.d]# yum install -y docker-ce kubelet kubeadm kubectl
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker啟動docker，docker需要到自動到docker倉庫中所依賴的鏡像文件，這些鏡像文件會因為在國外倉庫而下載無法完成，所以最好預先下載鏡像文件，kubeadm也可以支持本地私有倉庫進行獲取鏡像文件。
3、預先下載鏡像
在master節點上使用docker pull拉取鏡像，再通過tag打標簽
docker pull xiyangxixia/k8s-proxy-amd64:v1.11.1
docker tag xiyangxixia/k8s-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker pull xiyangxixia/k8s-scheduler:v1.11.1
docker tag xiyangxixia/k8s-scheduler:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker pull xiyangxixia/k8s-controller-manager:v1.11.1
docker tag xiyangxixia/k8s-controller-manager:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker pull xiyangxixia/k8s-apiserver-amd64:v1.11.1
docker tag xiyangxixia/k8s-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1
docker pull xiyangxixia/k8s-etcd:3.2.18
docker tag xiyangxixia/k8s-etcd:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18
docker pull xiyangxixia/k8s-coredns:1.1.3
docker tag xiyangxixia/k8s-coredns:1.1.3 k8s.gcr.io/coredns:1.1.3
docker pull xiyangxixia/k8s-pause:3.1
docker tag xiyangxixia/k8s-pause:3.1 k8s.gcr.io/pause:3.1
docker pull xiyangxixia/k8s-flannel:v0.10.0-s390x
docker tag xiyangxixia/k8s-flannel:v0.10.0-s390x quay.io/coreos/flannel:v0.10.0-s390x
docker pull xiyangxixia/k8s-flannel:v0.10.0-ppc64le
docker tag xiyangxixia/k8s-flannel:v0.10.0-ppc64le quay.io/coreos/flannel:v0.10.0-ppc64l
docker pull xiyangxixia/k8s-flannel:v0.10.0-arm
docker tag xiyangxixia/k8s-flannel:v0.10.0-arm quay.io/coreos/flannel:v0.10.0-arm
docker pull xiyangxixia/k8s-flannel:v0.10.0-amd64
docker tag xiyangxixia/k8s-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
node節點上拉取的鏡像
docker pull xiyangxixia/k8s-pause:3.1
docker tag xiyangxixia/k8s-pause:3.1 k8s.gcr.io/pause:3.1
docker pull xiyangxixia/k8s-proxy-amd64:v1.11.1
docker tag xiyangxixia/k8s-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker pull xiyangxixia/k8s-flannel:v0.10.0-amd64
docker tag xiyangxixia/k8s-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd644、部署kubernetes的Master節點
[root@k8s-master ~]# vim /etc/sysconfig/kubelet #修改kubelet禁止提示swap警告
KUBELET_EXTRA_ARGS="--fail-swap-on=false" #如果配置了swap不然提示出錯信息
更改kubelet配置，不提示swap警告信息，最好關閉swap
[root@k8s-master ~]# swapoff -a #關閉swap
[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.11.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap #初始化
[init] using Kubernetes version: v1.11.1
[preflight] running pre-flight checks
I0821 18:14:22.223765 18053 kernel_validator.go:81] Validating kernel version
I0821 18:14:22.223894 18053 kernel_validator.go:96] Validating kernel config
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.0-ce. Max validated version: 17.03
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using ‘kubeadm config images pull‘
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.11]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.56.11 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 51.033696 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node k8s-master as master by adding the label "node-role.kubernetes.io/master=‘‘"
[markmaster] Marking the node k8s-master as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master" as an annotation
[bootstraptoken] using token: dx7mko.j2ug1lqjra5bf6p2
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.56.11:6443 --token dx7mko.j2ug1lqjra5bf6p2 --discovery-token-ca-cert-hash sha256:93fe958796db44dcc23764cb8d9b6a2e67bead072e51a3d4d3c2d36b5d1007cf到了這步就可以完成Kubernetes Master的部署，這個過程需要幾分鐘。部署完成後，kubeadm會生成一行指令：
kubeadm join 192.168.56.11:6443 --token dx7mko.j2ug1lqjra5bf6p2 --discovery-token-ca-cert-hash sha256:93fe958796db44dcc23764cb8d9b6a2e67bead072e51a3d4d3c2d36b5d1007cf這個kubeadm join指令，就是用於給這個Master節點添加更多的工作節點（Worker）的命令，後面在部署Worker節點時會用到它，需要記錄下來。
此外，kubeadm還會提示我們第一次使用Kubernetes集群所需要配置的命令：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config? 需要配置這些命令的原因是：Kubernetes集群默認需要加密的方式訪問。所以，這幾條命令，就是將剛剛生成的Kubernetes集群的安全配置文件，保存到當前用戶的.kube目錄下，kubectl默認會使用這個目錄下的授權信息進行訪問Kubernetes集群。如果不那麽做的話，我們每次都需要通過export KUBECONFIG環境變量告訴kubectl這個安全配置文件的位置。
如果是使用root用戶部署，可以使用export進行定義環境變量
[root@k8s-master ~]# export KUBECONFIG=/etc/kubernetes/admin.conf到此，集群的初始化已經完成，可以使用kubectl get cs進行查看集群的健康狀態信息：
[root@k8s-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 43m v1.11.2從上面的結果可以看到，master的組件controller-manager、scheduler、etcd都處於正常狀態。那麽apiserver到哪去了？要知道kubectl是通過apiserver進行通信，從而在etcd中獲取到集群的狀態信息，所以可以獲取到集群的狀態信息，即表示apiserver是處於正常運行的狀態。使用kubectl get node獲取節點信息，可以看到master節點的狀態是NotReady，這是因為還沒有部署好Pod網絡。
5、安裝網絡插件CNI
安裝Pod網絡插件，用於保證Pod之間的相互通信。在每個集群當中只能有一個Pod網絡，在部署flannel之前需要更改的內核參數將橋接的IPv4的流量進行轉發給iptables鏈，這是CNI插件的運行的前提要求。
[root@k8s-master ~]# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
[root@k8s-master ~]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
1
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/c5d10c8/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds created
[root@k8s-master ~]# kubectl get node #再查看master節點的狀態信息，就是已經是Ready狀態了
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3h v1.11.2至此，Kubernetes的Master節點就已經部署完成，在默認情況下，Kubernetes的Master節點是不能運行用戶Pod的，這個屬於Kubernetes的汙點機制。
6、部署Kubernetes的Worker節點
Kubernetes的Worker節點和Master節點幾乎是相同的，它們運行這都是一個kubelet組件。唯一的區別在於kubeadm init的過程中，kubelet啟動後，Master節點還會自動運行kube-api-server、kube-scheduler、kube-controller-manager這三個系統Pod。
所以，相比之下，部署Worker節點只需要2步即可完成：
（1）在所有Worker節點上執行"安裝kubeadm、kubelet和docker"
（2）執行部署Master節點時生產的kubeadm join指令
[root@k8s-node01 ~]# yum install -y docker kubeadm kubelet
[root@k8s-node01 ~]# systemctl enable dokcer kubelet
[root@k8s-node01 ~]# systemctl start docker
這裏需要提前拉取好鏡像，如果docker配置了代理另說，按本次方法，提前拉取node節點所需要的鏡像。
[root@k8s-node01 ~]# kubeadm join 192.168.56.11:6443 --token dx7mko.j2ug1lqjra5bf6p2 --discovery-token-ca-cert-hash sha256:93fe958796db44dcc23764cb8d9b6a2e67bead072e51a3d4d3c2d36b5d1007cf
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 7h v1.11.2
k8s-node01 NotReady <none> 3h v1.11.2加入集群後，查看節點狀態信息，看到node01節點的狀態為NotReady，是因為node01節點上還沒有鏡像或者是還在拉取鏡像。等待拉取完鏡像就會啟動對應的Pod。
[root@k8s-node01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5502c29b43df f0fad859c909 "/opt/bin/flanneld -…" 3 minutes ago Up 3 minutes k8s_kube-flannel_kube-flannel-ds-pgpr7_kube-system_23dc27e3-a5af-11e8-84d2-000c2972dc1f_1
db1cc0a6fec4 d5c25579d0ff "/usr/local/bin/kube…" 3 minutes ago Up 3 minutes k8s_kube-proxy_kube-proxy-vxckf_kube-system_23dc0141-a5af-11e8-84d2-000c2972dc1f_0
bc54ad3399e8 k8s.gcr.io/pause:3.1 "/pause" 9 minutes ago Up 9 minutes k8s_POD_kube-proxy-vxckf_kube-system_23dc0141-a5af-11e8-84d2-000c2972dc1f_0
cbfca066b71d k8s.gcr.io/pause:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-flannel-ds-pgpr7_kube-system_23dc27e3-a5af-11e8-84d2-000c2972dc1f_0
[root@k8s-master ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-78fcdf6894-nmcmz 1/1 Running 0 1d 10.244.0.3 k8s-master
coredns-78fcdf6894-p5pfm 1/1 Running 0 1d 10.244.0.2 k8s-master
etcd-k8s-master 1/1 Running 1 1d 192.168.56.11 k8s-master
kube-apiserver-k8s-master 1/1 Running 8 1d 192.168.56.11 k8s-master
kube-controller-manager-k8s-master 1/1 Running 4 1d 192.168.56.11 k8s-master
kube-flannel-ds-n5c86 1/1 Running 0 1d 192.168.56.11 k8s-master
kube-flannel-ds-pgpr7 1/1 Running 1 1d 192.168.56.12 k8s-node01
kube-proxy-rxlt7 1/1 Running 1 1d 192.168.56.11 k8s-master
kube-proxy-vxckf 1/1 Running 0 1d 192.168.56.12 k8s-node01
kube-scheduler-k8s-master 1/1 Running 2 1d 192.168.56.11 k8s-master
[root@k8s-master ~]# kubectl get node #此時再查看狀態已經變成Ready
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 1d v1.11.2
k8s-node01 Ready <none> 1d v1.11.27、新增加集群的Worker節點
[root@k8s-node02 ~]# yum install -y docker kubeadm kubelet
[root@k8s-node02 ~]# systemctl enable docker kubelet
[root@k8s-node02 ~]# systemctl start docker同樣預先拉取好node節點所需鏡像，在此處犯錯的還有，根據官方說明tonken的默認有效時間為24h，由於時間差，導致這裏的token失效，可以使用kubeadm token list查看token，發現之前初始化的tonken已經失效了。
[root@k8s-master ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
dx7mko.j2ug1lqjra5bf6p2 <invalid> 2018-08-22T18:15:43-04:00 authentication,signing The default bootstrap token generated by ‘kubeadm init‘. system:bootstrappers:kubeadm:default-node-token那麽此處需要重新生成token，生成的方法如下：
[root@k8s-master ~]# kubeadm token create
1vxhuq.qi11t7yq2wj20cpe如果沒有值--discovery-token-ca-cert-hash，可以通過在master節點上運行以下命令鏈來獲取：
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed ‘s/^.* //‘
8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78此時，再運行kube join命令將node02加入到集群當中，此處的--discovery-token-ca-cert-hash依舊可以使用初始化時的證書
[root@k8s-node02 ~]# kubeadm join 192.168.56.11:6443 --token 1vxhuq.qi11t7yq2wj20cpe --discovery-token-ca-cert-hash sha256:93fe958796db44dcc23764cb8d9b6a2e67bead072e51a3d4d3c2d36b5d1007cf
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 1d v1.11.2
k8s-node01 Ready <none> 1d v1.11.2
k8s-node02 Ready <none> 2h v1.11.28、通過Taint/Toleration調整Master執行Pod策略
在前面我們了解到，默認情況下Master節點上是不允許運行用戶Pod的。而Kubernetes采用的是Taint/Toleration機制。
其原理是：一旦某個節點被加上了一個Taint，即被“打上了汙點”，那麽所有Pod就都不能在這個節點上運行，因為Kubernetes的Pod都有"潔癖"。
除非，有個別的Pod聲明自己能容忍這個"汙點"，即聲明Toleration，它才可以在這個節點上運行。
其中，為節點打上"汙點"（Taint）的命令是：
$ kubectl taint nodes node1 foo=bar:NoSchedule此時，該node1節點就會增加一個鍵值對格式的Taint，即foo=bar:NoSchedule。其中值裏面的NoSchedule，意味著這個Taint只會在調度新Pod時產生作用，而不會影響已經在node1上運行的Pod，哪怕它們沒有Toleration。
那麽Pod又該如何聲明Toleration呢？？
只需要在Pod的yaml文件中的spec部署，加入toleration字段即可：
apiVersion: v1
kind: Pod
...
spec:
tolerations:
- key: "foo"
operator: "Equal"
value: "bar"
effect: "NoSchedule"這個Toleration的含義是：這個Pod能容忍所有鍵值對為foo=bar的Taint（operator:"Equal"，"等於"操作）
我們可以通過kubectl describe來檢查一下Master節點上的Taint字段：
[root@k8s-master ~]# kubectl describe node k8s-master
Name: k8s-master
Roles: master
.....
Taints: node-role.kubernetes.io/master:NoSchedule
......可以看到，Master節點默認被加上了node-role.kubernetes.io/master:NoSchedule這樣的一個"汙點"，其中鍵是node-role.kubernetes.io/master，而沒有提供"值"。
此時，需要用"Exists"操作符（operator: "Exists"，"存在"即可）來說明該Pod能夠容忍所有以foo為鍵的Taint，才能讓這個Pod運行在該Master節點上：
apiVersion: v1
kind: Pod
...
spec:
tolerations:
- key: "foo"
operator: "Exists"
effect: "NoSchedule"當然，我們還可以通過刪除這個Taint，來解決汙點阻礙的問題：
$ kubectl taint nodes --all node-role.kubernetes.io/master-我們在"node-role.kubernetes.io/master"這個鍵後面增加了一個橫線"-"，表示移除所有以"node-role.kubernetes.io/master"為鍵的Taint。
到此，使用kubeadm部署k8s集群就暫時告一段落啦！！！！
Kubernetes學習之路（一）之Kubeadm部署K8S集群