k8s叢集搭建
上來裝centos7叢集敲著命令呢，突然網斷了，我以為我哪裡命令搞錯了，弄崩了系統，重灌，之前因為已經zhua裝好一個centos7並且多裝了個vmtools,後面查明
cd /etc/sysconfig/network-scripts/ vim ifcfg-ens33 （ifconfig 獲取）（最基本安裝連這個命令都沒有，我後面都帶介面安裝了）
ONBOOT=no
改成
ONBOOT=yes
service network restart
先講用kubeadm安裝kubernetes叢集
systemctl disable firewalld
systemctl stop firewalld
setenforce 0或者改/etc/sysconfig/selinux 將SELINUX=enforcing 改成SELINUX=disabled
以上操作我暫時沒動
首先是安裝docker
yum install -y docker這個在國內貌似沒問題
[[email protected] ~]$ docker --version
Docker version 1.13.1, build 07f3374/1.13.1
裝kubeadm各路都有，主要是源的問題，我找的的方法是
https://blog.csdn.net/zhuchuangang/article/details/76572157#2
https://blog.csdn.net/zhuchuangang/article/details/76572157
#kubernetes yum源
vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
然後就是
yum install -y kubelet
yum install -y kubeadm
安裝好這兩個，yum install -y kubectl kubernetes-cni時說都已經安裝完成，估計其實是kubeadm都集成了，第一個命令都不一定需要。
systemctl start docker
systemctl enable docker
systemctl enable kubelet
systemctl start kubelet
[[email protected] ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
接下來下載kubernetes的docker映象
由於kubeadmjian將自動下載kubernetes的相關映象，預設gcr.io下載（沒看出來）
修改docker配置/vim /etc/sysconfig/docker，增加registry mirror引數=false那個是我加的
OPTIONS='--selinux-enabled=false --log-driver=journald --signature-verification=false'
OPTIONS='--registry-mirror=http://68e02ab9.m.daocloud.io'
kubeadm init --kubernetes-version=1.6.0
this version of kubeadm only supports deploying clusters with the control plane version >= 1.12.0. Current version: v1.6.0
[[email protected] ~]# kubeadm init --kubernetes-version=1.12.0
[init] Using Kubernetes version: v1.12.0
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[ERROR Swap]: running with swap on is not supported. Please disable swap
[ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.13.1" Control plane version: "1.12.0"
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
systemctl disable firewalld
systemctl stop firewalld
vim /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1重啟
或者
建立/etc/sysctl.d/k8s.conf檔案
sudo vi /etc/sysctl.d/k8s.conf
新增如下內容：
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
sudo sysctl -p /etc/sysctl.d/k8s.conf
https://blog.csdn.net/zzq900503/article/details/81710319
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
ansible all -m shell -a "iptables -P FORWARD ACCEPT"
永久修改：/usr/lib/sysctl.d/00-system.conf
SELINUX=enforcing
swapoff -a
https://blog.csdn.net/CSDN_duomaomao/article/details/75142769?locationNum=8&fps=1
[[email protected] sysctl.d]# kubeadm init --kubernetes-version=1.13.0
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-apiserver ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 108.177.97.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-controller-manager ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-scheduler ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.187.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.0: output: Trying to pull repository k8s.gcr.io/kube-proxy ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Trying to pull repository k8s.gcr.io/pause ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 108.177.125.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Trying to pull repository k8s.gcr.io/etcd ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Trying to pull repository k8s.gcr.io/coredns ...
Get https://k8s.gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
https://blog.csdn.net/jinguangliu/article/details/82792617
kubeadm config images list 根據要求寫名字，不能寫錯，或者根據錯誤寫docker images
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
kubeadm reset
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
執行journalctl -xeu kubelet
Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://192.168.41.137:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.41.137:6443: connect: connection refused
eviction manager: failed to get get summary stats: failed to get node
https://blog.csdn.net/bbwangj/article/details/82024485
按照上面的方法始終不奏效，錯誤又是不停的迴圈滾動，到底也不知道哪裡錯了
後面http://blog.51cto.com/ghbsunny/2162205?source=dra
搜到最後這個錯誤，原來是記憶體小了，改成2G可以了
至於network那個後面kubeadm啟動成功後會有系統解決方式告知
這裡面的問題解決https://blog.csdn.net/u012286287/article/details/79716588
解決方案https://blog.csdn.net/zzq900503/article/details/81710319
kubelet 的cgroup dirver 與 docker的不一樣。docker預設使用cgroupfs,keubelet 預設使用systemd沒遇到，我是一致的
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.41.137:6443 --token 2dva0q.rr3htfw7s4ofmjh4 --discovery-token-ca-cert-hash sha256:9384042d6812ff7631f0c3c5c5b827ccbefdbbac6d9213611c077799e89bfc6e
最後選了這個apply， 版本有點多
mkdir -p ~/k8s/
cd ~/k8s
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
[[email protected] k8s]# kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-86c58d9df4-6m2vv 0/1 ContainerCreating 0 31m <none> master <none> <none>
kube-system coredns-86c58d9df4-xttbs 0/1 ContainerCreating 0 31m <none> master <none> <none>
kube-system etcd-master 1/1 Running 0 30m 192.168.41.137 master <none> <none>
kube-system kube-apiserver-master 1/1 Running 0 30m 192.168.41.137 master <none> <none>
kube-system kube-controller-manager-master 1/1 Running 0 30m 192.168.41.137 master <none> <none>
kube-system kube-flannel-ds-amd64-ftjsk 0/1 CrashLoopBackOff 3 6m4s 192.168.41.137 master <none> <none>
kube-system kube-proxy-bnzzx 1/1 Running 0 31m 192.168.41.137 master <none> <none>
kube-system kube-scheduler-master 1/1 Running 0 30m 192.168.41.137 master <none> <none>
journalctl -u kubelet -f
kubectl describe pod coredns-86c58d9df4-xttbs -n kube-system
kubectl describe pods --all-namespaces
kubectl logs coredns-86c58d9df4-6m2vv -n kube-system
kubectl get pods --all-namespaces
kubectl get pod --all-namespaces -o wide
kubectl get pods -n kube-system -o wide
kubeadm reset
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables
kubeadm init --kubernetes-version=v1.13.0 --pod-network-cidr=10.244.0.0/16
kubeadm join 192.168.41.137:6443 --token ycd1dl.xza4hi7b4prr0387 --discovery-token-ca-cert-hash sha256:15299a96ced577a2a865216b9240511d47e940e961ac461970a55f12e2b564be
docker pull quay.io/coreos/flannel:v0.10.0-amd64
mkdir -p /etc/cni/net.d/
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=172.100.0.0/16
FLANNEL_SUBNET=172.100.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
叢集初始化如果遇到問題，可以使用下面的命令進行清理再重新初始化：
kubeadm reset
[[email protected] k8s]# iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
[runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
mkdir -p ~/k8s/
cd ~/k8s
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
問題解決參考
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS='--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice'
https://blog.csdn.net/qq_34857250/article/details/82562514
open /run/flannel/subnet.env
http://dockone.io/question/1225
參考