Kubernetes 二進位制部署-V1.18.6

阿新 • • 發佈：2020-10-11

介紹

目前生產環境部署 kubernetes 叢集主要有兩種方式：

kubeadm部署

https://www.cnblogs.com/hukey/p/13773927.html

二進位制部署

從 github 下載發行版的二進位制包，手動部署每個元件，組成 Kubernetes 叢集。

Kubeadm 降低了部署門檻，但遮蔽了很多細節，遇到問題很難排查。如果想要更容易可控，推薦使用二進位制部署kubernetes 叢集，雖然手動部署麻煩點，期間可以學習很多工作原理，也利於後期維護。

本文部署版本：【v1.18.6 】所有用到的映象及安裝程式下載地址：

連結：https://pan.baidu.com/s/1au7bfc6O_T_Ja-ALiYzcjg

提取碼：gspr

環境

角色	IP	備註
master	192.168.1.14	kube-apiserver，kube-controller-manager，kube-scheduler，etcd
node1	192.168.1.15	kubelet，kube-proxy，docker， etcd
node2	192.168.1.16	kubelet，kube-proxy，docker ，etcd

環境初始化

所有叢集主機都需要做初始化

停止所有伺服器 firewalld 防火牆

systemctl stop firewalld; systemctl disable firewalld

關閉 Swap

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab

關閉 Selinux

setenforce  0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux

宣告主機名

cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.14	master
192.168.1.15	node1
192.168.1.16	node2

設定主機名、升級核心、安裝 Docker-ce

執行下面 init.sh 指令碼，指令碼完成下面四項任務：

設定伺服器 hostname
安裝 k8s 依賴環境
升級系統核心
安裝 docker-ce

在每臺機器上執行 init.sh 指令碼，示例如下：

Ps：init.sh 指令碼只用於 Centos，支援重複執行。

指令碼內容如下：

#!/usr/bin/env bash
function Check_linux_system(){
    linux_version=`cat /etc/redhat-release`
    if [[ ${linux_version} =~ "CentOS" ]];then
        echo -e "\033[32;32m 系統為 ${linux_version} \033[0m \n"
    else
        echo -e "\033[32;32m 系統不是CentOS,該指令碼只支援CentOS環境\033[0m \n"
        exit 1
    fi
}

function Set_hostname(){
    if [ -n "$HostName" ];then
      grep $HostName /etc/hostname && echo -e "\033[32;32m 主機名已設定，退出設定主機名步驟 \033[0m \n" && return
      case $HostName in
      help)
        echo -e "\033[32;32m bash init.sh 主機名 \033[0m \n"
        exit 1
      ;;
      *)
        hostname $HostName
        echo "$HostName" > /etc/hostname
        echo "`ifconfig eth0 | grep inet | awk '{print $2}'` $HostName" >> /etc/hosts
      ;;
      esac
    else
      echo -e "\033[32;32m 輸入為空，請參照 bash init.sh 主機名 \033[0m \n"
      exit 1
    fi
}

function Install_depend_environment(){
    rpm -qa | grep nfs-utils &> /dev/null && echo -e "\033[32;32m 已完成依賴環境安裝，退出依賴環境安裝步驟 \033[0m \n" && return
    yum install -y nfs-utils curl yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl telnet
    echo -e "\033[32;32m 升級Centos7系統核心到5版本，解決Docker-ce版本相容問題\033[0m \n"
    rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org && \
    rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel repolist && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml.x86_64 && \
    yum remove -y kernel-tools-libs.x86_64 kernel-tools.x86_64 && \
    yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml-tools.x86_64 && \
    grub2-set-default 0
    modprobe br_netfilter
    cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
    sysctl -p /etc/sysctl.d/k8s.conf
    ls /proc/sys/net/bridge
}


function Install_docker(){
    rpm -qa | grep docker && echo -e "\033[32;32m 已安裝docker，退出安裝docker步驟 \033[0m \n" && return
    yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    yum makecache fast
    yum -y install docker-ce-19.03.6 docker-ce-cli-19.03.6
    systemctl enable docker.service
    systemctl start docker.service
    systemctl stop docker.service
    echo '{"registry-mirrors": ["https://4xr1qpsp.mirror.aliyuncs.com"], "log-opts": {"max-size":"500m", "max-file":"3"}}' > /etc/docker/daemon.json
    systemctl daemon-reload
    systemctl start docker
}

HostName=$1
Check_linux_system && \
Set_hostname && \
Install_depend_environment && \
Install_docker

執行指令碼：

# master 機器執行，init.sh 後面接的引數是設定 master 伺服器主機名
chmod +x init.sh && ./init.sh master

# 執行完 init.sh 指令碼，請重啟伺服器
reboot


### node1 ###
chmod +x init.sh && ./init.sh node1
reboot

### node2 ###
chmod +x init.sh && ./init.sh node2
reboot

部署 etcd 叢集

Etcd 是一個分散式鍵值儲存系統，Kubernetes 使用 Etcd 進行資料儲存，所以先準備一個 Etcd 資料庫，為解決Etcd單點故障，應採用叢集方式部署，這裡使用 3臺元件叢集，可容忍1臺機器故障，當然，也可以用 5 臺組建叢集，可容忍2臺機器故障。

節點名稱	IP
etcd01	192.168.1.14
etcd02	192.168.1.15
etcd03	192.168.1.16

Ps：為了節省機器，這裡的 etcd 叢集和 k8s叢集節點複用，也可獨立於k8s叢集之外。

準備 cfssl 證書生成工具

cfssl是一個開源的證書管理工具，使用json檔案生成證書，相比openssl更方便使用。

找任意一臺伺服器操作，這裡用Master節點。

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

生成 etcd 證書

[[email protected] ~]# mkdir -pv /data/tls/{etcd,k8s}
mkdir: created directory ‘/data’
mkdir: created directory ‘/data/tls’
mkdir: created directory ‘/data/tls/etcd’
mkdir: created directory ‘/data/tls/k8s’
[[email protected] ~]#cd /data/tls/etcd/

自籤TLS 證書

# 建立 certificate.sh 指令碼
[[email protected] /data/tls/etcd]#vim certificate.sh
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
              "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------
# 注意：根據自身環境進行配置 IP 地址
#
#

cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.1.14",
      "192.168.1.15",
      "192.168.1.16"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

執行指令碼生成證書：

[[email protected] /data/tls/etcd]#chmod +x certificate.sh && ./certificate.sh
[[email protected] /data/tls/etcd]#ls *.pem
ca-key.pem  ca.pem  server-key.pem  server.pem

建立工作目錄並解壓二進位制包

[[email protected] /data/tls/etcd]#mkdir -pv /opt/etcd/{bin,cfg,ssl}
mkdir: created directory ‘/opt/etcd’
mkdir: created directory ‘/opt/etcd/bin’
mkdir: created directory ‘/opt/etcd/cfg’
mkdir: created directory ‘/opt/etcd/ssl’

[[email protected] ~]#cd /usr/local/src/
[[email protected] /usr/local/src]#tar xf offline-k8s-packages.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages]#cp -a etcd-v3.4.7-linux-amd64/etcd* /opt/etcd/bin/

### 編寫 etcd 配置檔案指令碼 ###
[[email protected] /opt/etcd/cfg]#mkdir -pv /data/etcd/
mkdir: created directory ‘/data/etcd/’
[[email protected] /opt/etcd/cfg]#cd /data/etcd/

[[email protected] /data/etcd]#vim etcd.sh
#!/bin/bash

etcd_NAME=${1:-"etcd01"}
etcd_IP=${2:-"127.0.0.1"}
etcd_CLUSTER=${3:-"etcd01=https://127.0.0.1:2379"}

cat <<EOF >/opt/etcd/cfg/etcd.yml
name: ${etcd_NAME}
data-dir: /var/lib/etcd/default.etcd
listen-peer-urls: https://${etcd_IP}:2380
listen-client-urls: https://${etcd_IP}:2379,https://127.0.0.1:2379

advertise-client-urls: https://${etcd_IP}:2379
initial-advertise-peer-urls: https://${etcd_IP}:2380
initial-cluster: ${etcd_CLUSTER}
initial-cluster-token: etcd-cluster
initial-cluster-state: new

client-transport-security:
  cert-file: /opt/etcd/ssl/server.pem
  key-file: /opt/etcd/ssl/server-key.pem
  client-cert-auth: false
  trusted-ca-file: /opt/etcd/ssl/ca.pem
  auto-tls: false

peer-transport-security:
  cert-file: /opt/etcd/ssl/server.pem
  key-file: /opt/etcd/ssl/server-key.pem
  client-cert-auth: false
  trusted-ca-file: /opt/etcd/ssl/ca.pem
  auto-tls: false

debug: false
logger: zap
log-outputs: [stderr]
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd Server
Documentation=https://github.com/etcd-io/etcd
Conflicts=etcd.service
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
LimitNOFILE=65536
Restart=on-failure
RestartSec=5s
TimeoutStartSec=0
ExecStart=/opt/etcd/bin/etcd --config-file=/opt/etcd/cfg/etcd.yml

[Install]
WantedBy=multi-user.target
EOF

執行 etcd.sh 指令碼：

[[email protected] /data/etcd]#chmod +x etcd.sh
[[email protected] /data/etcd]#./etcd.sh etcd01 192.168.1.14 etcd01=https://192.168.1.14:2380,etcd02=https://192.168.1.15:2380,etcd03=https://192.168.1.16:2380

### 拷貝證書到etcd工作目錄 ###
[[email protected] /data/etcd]#cp -a /data/tls/etcd/*pem /opt/etcd/ssl/
[[email protected] /data/etcd]#ls /opt/etcd/ssl/
ca-key.pem  ca.pem  server-key.pem  server.pem

Ps：如果 etcd 搭建叢集，服務最好能一起啟動，否則會出現報錯資訊

將 etcd 工作目錄拷貝到 node1 node2

[[email protected] ~]#scp -r /opt/etcd/ node1:/opt/
[[email protected] ~]#scp -r /opt/etcd/ node2:/opt/

[[email protected] ~]#scp /usr/lib/systemd/system/etcd.service node1:/usr/lib/systemd/system/
[[email protected] ~]#scp /usr/lib/systemd/system/etcd.service node2:/usr/lib/systemd/system/

切換到 node1 修改配置檔案引數

[[email protected] ~]#cd /opt/etcd/cfg/
[[email protected] /opt/etcd/cfg]#sed -i 's/1.14/1.15/g' etcd.yml
[[email protected] /opt/etcd/cfg]#sed -i 's/name: etcd01/name: etcd02/g' etcd.yml

### 注意這裡要修改正確 ###
[[email protected] /opt/etcd/cfg]#vim etcd.yml
...
initial-cluster: etcd02=https://192.168.1.14:2380,etcd02=https://192.168.1.15:2380,etcd03=https://192.168.1.16:2380
...

切換到 node2 修改配置檔案引數

[[email protected] /opt/etcd/cfg]#sed -i 's/1.14/1.16/g' etcd.yml
[[email protected] /opt/etcd/cfg]#sed -i 's/name: etcd01/name: etcd03/g' etcd.yml

### 注意這裡要修改正確 ###
[[email protected] /opt/etcd/cfg]#vim etcd.yml
...
initial-cluster: etcd02=https://192.168.1.14:2380,etcd02=https://192.168.1.15:2380,etcd03=https://192.168.1.16:2380
...

啟動 etcd 服務並驗證

### 三個節點都要啟動 ###
systemctl enable etcd ; systemctl start etcd

### 新增etcd命令到環境變數
echo "export PATH=/opt/etcd/bin:\$PATH" > /etc/profile.d/etcd.sh
. /etc/profile.d/etcd.sh

### 驗證檢視 ###
etcdctl --write-out=table \
--cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem \
--endpoints=https://192.168.1.14:2379,https://192.168.1.15:2379,https://192.168.1.16:2379 endpoint health
+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://192.168.1.14:2379 |   true | 13.341351ms |       |
| https://192.168.1.15:2379 |   true | 14.125986ms |       |
| https://192.168.1.16:2379 |   true | 18.895577ms |       |
+---------------------------+--------+-------------+-------+

到此，etcd 叢集搭建成功。

部署 Master Node

生成 kube-apiserver 證書

[[email protected] ~]#cd /data/tls/k8s/

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

# 生成證書
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

[[email protected] /data/tls/k8s]#ls *pem
ca-key.pem  ca.pem

簽發kube-apiserver 證書

[[email protected] ~]#cd /data/tls/k8s/
### hosts欄位中IP為所有Master/LB/VIP IP，一個都不能少！為了方便後期擴容可以多寫幾個預留的IP。 ###
cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.1.14",
      "192.168.1.15",
      "192.168.1.16",
      "192.168.1.17",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

### 生成證書 ###

[[email protected] /data/tls/k8s]#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

### 檢視生成後證書 ###
[[email protected] /data/tls/k8s]#ls server*pem
server-key.pem  server.pem

建立k8s工作目錄解壓二進位制包

二進位制包包含在開篇的下載

[[email protected] ~]#mkdir -pv /opt/kubernetes/{bin,cfg,ssl,logs}
[[email protected] ~]#cd /usr/local/src/offline-k8s-packages
[[email protected] /usr/local/src/offline-k8s-packages]#tar xf kubernetes-server-linux-amd64.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages/kubernetes/server/bin]#cp -a kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin/

### 新增環境變數 ###
[[email protected] ~]#echo "export PATH=/opt/kubernetes/bin:\$PATH" > /etc/profile.d/k8s.sh
[[email protected] ~]#. /etc/profile.d/k8s.sh

部署 kube-apiserver

編寫配置檔案及服務啟動指令碼

[[email protected] ~]#mkdir -p /data/k8s
[[email protected] ~]#cd /data/k8s

### 編寫生成 kube-apiserver.conf 指令碼 ###
[[email protected] /data/k8s]#vim apiserver.sh
#!/bin/bash

MASTER_ADDRESS=${1:-"192.168.1.14"}
ETCD_SERVERS=${2:-"http://127.0.0.1:2379"}


cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=${ETCD_SERVERS} \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=${MASTER_ADDRESS} \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

對上面指令碼中的引數做一個註解：

# –logtostderr：啟用日誌
# —v：日誌等級
# –log-dir：日誌目錄
# –etcd-servers：etcd叢集地址
# –bind-address：監聽地址
# –secure-port：https安全埠
# –advertise-address：叢集通告地址
# –allow-privileged：啟用授權
# –service-cluster-ip-range：Service虛擬IP地址段
# –enable-admission-plugins：准入控制模組
# –authorization-mode：認證授權，啟用RBAC授權和節點自管理
# –enable-bootstrap-token-auth：啟用TLS bootstrap機制
# –token-auth-file：bootstrap token檔案
# –service-node-port-range：Service nodeport型別預設分配埠範圍
# –kubelet-client-xxx：apiserver訪問kubelet客戶端證書
# –tls-xxx-file：apiserver https證書
# –etcd-xxxfile：連線Etcd叢集證書
# –audit-log-xxx：審計日誌

拷貝生成的證書

### 拷貝認證檔案到k8s 工作目錄 ###
[[email protected] ~]#cp -a /data/tls/k8s/*pem /opt/kubernetes/ssl/
[[email protected] ~]#ls /opt/kubernetes/ssl/
ca-key.pem  ca.pem  server-key.pem  server.pem

啟用 TLS Bootstrapping 機制

TLS Bootstraping：Master apiserver啟用TLS認證後，Node節點kubelet和kube-proxy要與kube-apiserver進行通訊，必須使用CA簽發的有效證書才可以，當Node節點很多時，這種客戶端證書頒發需要大量工作，同樣也會增加叢集擴充套件複雜度。為了簡化流程，Kubernetes引入了TLS bootstraping機制來自動頒發客戶端證書，kubelet會以一個低許可權使用者自動向apiserver申請證書，kubelet的證書由apiserver動態簽署。所以強烈建議在Node上使用這種方式，目前主要用於kubelet，kube-proxy還是由我們統一頒發一個證書

TLS bootstrapping 工作流程

建立上述檔案中的 token檔案

cat > /opt/kubernetes/cfg/token.csv << EOF
4fb823f13d1d21be5991b9e3a582718c,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

# 格式：token，使用者名稱，UID，使用者組
# 注意：這裡的使用者是不需要通過 useradd 建立的使用者

# token 是隨機碼，也可自行生成：
head -c 16 /dev/urandom | od -An -t x | tr -d ' '

執行 apiserver.sh 指令碼：

[[email protected] /data/k8s]#chmod +x apiserver.sh
[[email protected] /data/k8s]#./apiserver.sh 192.168.1.14 https://192.168.1.14:2379,https://192.168.1.15:2379,https://192.168.1.16:2379

### 啟動並設定開機啟動 ###
[[email protected] ~]#systemctl enable kube-apiserver ; systemctl start kube-apiserver

授權 kubelet-bootstrap 使用者允許請求證書

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

部署 kube-controller-manager

建立配置檔案及啟動檔案

[[email protected] ~]#cd /data/k8s/
[[email protected] /data/k8s]#vim controller-manager.sh
#!/bin/bash
#
cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF


cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF


# –master：通過本地非安全本地埠8080連線apiserver。
# –leader-elect：當該元件啟動多個時，自動選舉（HA）
# –cluster-signing-cert-file/–cluster-signing-key-file：自動為kubelet頒發證書的CA，與apiserver保持一致

執行指令碼：

[[email protected] /data/k8s]#chmod +x controller-manager.sh 
[[email protected] /data/k8s]#./controller-manager.sh

啟動並開機自啟

[[email protected] ~]#systemctl enable kube-controller-manager
[[email protected] ~]#systemctl start kube-controller-manager

部署 kube-scheduler

建立配置檔案及啟動指令碼

[[email protected] ~]#cd /data/k8s/
[[email protected] /data/k8s]#vim scheduler.sh

#!/bin/bash
#
cat > /opt/kubernetes/cfg/kube-scheduler.conf  << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --master=127.0.0.1:8080 --bind-address=127.0.0.1"
EOF


# –master：通過本地非安全本地埠8080連線apiserver。
# –leader-elect：當該元件啟動多個時，自動選舉（HA）


cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF


### 執行指令碼 ###
[[email protected] /data/k8s]#chmod +x scheduler.sh
[[email protected] /data/k8s]#./scheduler.sh

啟動服務並開機自啟

[[email protected] /data/k8s]#systemctl enable kube-scheduler
[[email protected] /data/k8s]#systemctl start kube-scheduler

設定 kuberctl 自動補全功能

[[email protected] ~]#yum install bash-completion -y
[[email protected] ~]#kubectl completion bash > /etc/bash_completion.d/kubectl

重啟會話即可實現 kubectl 自動補全功能

檢視叢集

# 所有元件都已經啟動成功，通過kubectl工具檢視當前叢集元件狀態：
[[email protected] ~]#kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-1               Healthy   {"health":"true"}
etcd-0               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}

部署 Work node

下面切換到 node1 （192.168.1.15）節點操作

建立工作目錄並拷貝二進位制檔案

[[email protected] ~]#mkdir -pv /opt/kubernetes/{bin,cfg,ssl,logs}
[[email protected] ~]#cd /usr/local/src/
[[email protected] /usr/local/src]#tar xf offline-k8s-packages.tar.gz
[[email protected] /usr/local/src]#cd offline-k8s-packages/
[[email protected] /usr/local/src/offline-k8s-packages]#tar xf kubernetes-server-linux-amd64.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages/kubernetes/server/bin]#cp -a kubelet kube-proxy /opt/kubernetes/bin/
[[email protected] ~]#echo "export PATH=/opt/kubernetes/bin:\$PATH" > /etc/profile.d/k8s.sh
[[email protected] ~]#. /etc/profile.d/k8s.sh

部署kubelet

匯入k8s所需的docker映象檔案

[[email protected] ~]#cd /usr/local/src/offline-k8s-packages
[[email protected] /usr/local/src/offline-k8s-packages]#tar xf k8s-imagesV1.18.6.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages]#docker load < k8s-imagesV1.18.6.tar
[[email protected] /usr/local/src/offline-k8s-packages]#docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
quay.io/coreos/flannel   v0.13.0-rc2         79dd6d6368e2        3 weeks ago         57.2MB
k8s.gcr.io/pause         3.2                 80d28bedfe5d        7 months ago        683kB

建立配置檔案

[[email protected] ~]#mkdir -pv /data/k8s
[[email protected] ~]#cd /data/k8s
[[email protected] ~]#vim kubelet.sh

#!/bin/bash
#

HostName=${1:-"node1"}
cat > /opt/kubernetes/cfg/kubelet.conf <<EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=${HostName} \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=k8s.gcr.io/pause:3.2"
EOF

# –hostname-override：顯示名稱，叢集中唯一
# –network-plugin：啟用CNI
# –kubeconfig：空路徑，會自動生成，後面用於連線apiserver
# –bootstrap-kubeconfig：首次啟動向apiserver申請證書
# –config：配置引數檔案
# –cert-dir：kubelet證書生成目錄
# –pod-infra-container-image：管理Pod網路容器的映象


### 啟動指令碼 ###
cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF


### 執行指令碼 ###
[[email protected] /data/k8s]#chmod +x kubelet.sh
[[email protected] /data/k8s]#./kubelet.sh node1

配置引數檔案

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

生成 bootstrap.kubeconfig 檔案

注意：生成 bootstrap.kubeconfig 是在 master 節點執行的，完成後拷貝到 node 節點

[[email protected] ~]#cd /data/k8s/

KUBE_APISERVER="https://192.168.1.14:6443" # apiserver IP:PORT
TOKEN="4fb823f13d1d21be5991b9e3a582718c" # 與master節點 token.csv裡保持一致

# 生成 kubelet bootstrap kubeconfig 配置檔案
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
  
  
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
  
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
  
  
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

# 拷貝到配置檔案路徑
[[email protected] /data/k8s]#scp bootstrap.kubeconfig node1:/opt/kubernetes/cfg/

拷貝認證檔案到 node 節點

從 master 節點拷貝到 node

[[email protected] ~]#scp /opt/kubernetes/ssl/ca.pem node1:/opt/kubernetes/ssl/

啟動 kubelet

[[email protected] ~]#systemctl enable kubelet; systemctl start kubelet

批准 kubelet 證書並加入叢集

master 節點操作

# 檢視kubelet證書請求
[[email protected] ~]#kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-D_9vbQyKez8UqyNqYkVXrQ0fvpLyZ2LqSPqNhu65ZaQ   4m8s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 批准申請
[[email protected] ~]#kubectl certificate approve node-csr-D_9vbQyKez8UqyNqYkVXrQ0fvpLyZ2LqSPqNhu65ZaQ

# 檢視節點
[[email protected] ~]#kubectl get nodes
NAME    STATUS     ROLES    AGE   VERSION
node1   NotReady   <none>   14s   v1.18.6

# 由於網路外掛還沒有部署，節點會沒有準備就緒 NotReady

部署 kube-proxy

建立配置檔案

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

建立引數檔案

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: node1
clusterCIDR: 10.0.0.0/24
EOF

生成 kubeconfig 檔案

在 master 節點執行

[[email protected] ~]#cd /data/tls/k8s/

cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

# 生成證書
[[email protected] /data/tls/k8s]#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

[[email protected] /data/tls/k8s]#ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

生成檔案：

KUBE_APISERVER="https://192.168.1.14:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
  
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
  
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig


# 拷貝配置檔案到指定路徑
[[email protected] /data/tls/k8s]#scp kube-proxy.kubeconfig node1:/opt/kubernetes/cfg/

建立kube-proxy 啟動指令碼

回到 node1 （192.168.1.15）操作

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

啟動並設定開機啟動

[[email protected] ~]#systemctl enable kube-proxy; systemctl start kube-proxy

部署 CNI 網路

注： master 節點是未安裝 kubelet 和 kube-proxy

準備二進位制檔案

[[email protected] ~]#mkdir -pv /opt/cni/bin
[[email protected] ~]#cd /usr/local/src/offline-k8s-packages/
[[email protected] /usr/local/src/offline-k8s-packages]#tar xf cni-plugins-linux-amd64-v0.8.7.tgz -C /opt/cni/bin/

匯入映象

[[email protected] /usr/local/src/offline-k8s-packages]#tar xf k8s-imagesV1.18.6.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages]#docker load < k8s-imagesV1.18.6.tar
[[email protected] /usr/local/src/offline-k8s-packages]#docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
quay.io/coreos/flannel         v0.13.0-rc2         79dd6d6368e2        3 weeks ago         57.2MB
coredns/coredns                1.7.0               bfe3a36ebd25        3 months ago        45.2MB
kubernetesui/metrics-scraper   v1.0.4              86262685d9ab        6 months ago        36.9MB
k8s.gcr.io/pause               3.2                 80d28bedfe5d        7 months ago        683kB

初始化 flannel

初始化操作是在 master 節點執行

[[email protected] /usr/local/src/offline-k8s-packages]#kubectl apply -f coredns.yaml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

檢視驗證

### 檢視 flannel pod ###
[[email protected] /usr/local/src/offline-k8s-packages]#kubectl get pods -n kube-system
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-sz7x7   1/1     Running   0          5s

### 檢視節點狀態 ###
[[email protected] /usr/local/src/offline-k8s-packages]#kubectl get nodes
NAME    STATUS   ROLES    AGE     VERSION
node1   Ready    <none>   4m46s   v1.18.6

Work node 節點已經從 NotReady 更新為 Ready

部署 CoreDNS

初始化 CoreDNS

[[email protected] /usr/local/src/offline-k8s-packages]#kubectl apply -f coredns.yaml

檢視

[[email protected] ~]#kubectl get pods,service -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
pod/coredns-85b4878f78-lrvpc   1/1     Running   0          68s
pod/kube-flannel-ds-sz7x7      1/1     Running   0          9m3s

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP,9153/TCP   68s

驗證

[[email protected] ~]#kubectl run nginx-deploy --image=nginx:alpine
pod/nginx-deploy created
[[email protected] ~]#kubectl expose pod nginx-deploy --name=nginx --port=80 --target-port=80 --protocol=TCP
service/nginx exposed

### 建立 pod，嘗試通過 nginx 服務名訪問 ###
[[email protected] ~]#kubectl run -it --rm client --image=busybox
If you don't see a command prompt, try pressing enter.
/ # wget -O - -q nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

新增 Work node 部署

新增新的node2 到叢集

拷貝已經部署好的 node1 相關檔案到新節點

注：預設已經做過初始化，安裝了 docker-ce

拷貝檔案到新節點

[[email protected] ~]#scp -r /opt/cni/ node2:/opt/
[[email protected] ~]#scp -r /opt/kubernetes/ node2:/opt/
[[email protected] ~]#scp /usr/lib/systemd/system/kube* node2:/usr/lib/systemd/system/

匯入映象、修改配置檔案並啟動服務

[[email protected] /usr/local/src/offline-k8s-packages]#tar xf k8s-imagesV1.18.6.tar.gz
[[email protected] /usr/local/src/offline-k8s-packages]#docker load < k8s-imagesV1.18.6.tar
[[email protected] /usr/local/src/offline-k8s-packages]#docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
quay.io/coreos/flannel         v0.13.0-rc2         79dd6d6368e2        3 weeks ago         57.2MB
coredns/coredns                1.7.0               bfe3a36ebd25        3 months ago        45.2MB
kubernetesui/metrics-scraper   v1.0.4              86262685d9ab        6 months ago        36.9MB
k8s.gcr.io/pause               3.2                 80d28bedfe5d        7 months ago        683kB

# 這幾個檔案是證書申請審批後自動生成的，每個Node不同，必須刪除重新生成。
[[email protected] ~]#rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig
[[email protected] ~]#cd /opt/kubernetes/cfg/
[[email protected] ~]#rm -rf /opt/kubernetes/ssl/kubelet*

# 修改主機名，這裡兩個檔案配置必須一致，否則會報錯
[[email protected] /opt/kubernetes/cfg]#sed -i 's/node1/node2/g' kubelet.conf
[[email protected] /opt/kubernetes/cfg]#sed -i 's/node1/node2/g' kube-proxy-config.yml

# 啟動服務
[[email protected] ~]#systemctl enable kubelet kube-proxy
[[email protected] ~]#systemctl start kubelet kube-proxy

在mster 上批准node kubelet 證書申請

[[email protected] ~]#kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-8Y-o-S-Ge3MR4Ico7cb93xpf1GAbwlwUreiH0tpqJR0   38m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
node-csr-brk2tg384kUGQcDcAwXNzOtz0bhS5qYKSAoESIwV6qU   85s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending
[[email protected] ~]#kubectl certificate approve node-csr-brk2tg384kUGQcDcAwXNzOtz0bhS5qYKSAoESIwV6qU
certificatesigningrequest.certificates.k8s.io/node-csr-brk2tg384kUGQcDcAwXNzOtz0bhS5qYKSAoESIwV6qU approved

# 等待一會時間
[[email protected] ~]#kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    <none>   38m   v1.18.6
node2   Ready    <none>   11s   v1.18.6

# 檢視
[[email protected] ~]#kubectl get pods,svc -n kube-system -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE    NOMINATED NODE   READINESS GATES
pod/coredns-85b4878f78-lrvpc   1/1     Running   0          26m   10.244.0.2     node1   <none>           <none>
pod/kube-flannel-ds-q4w4g      1/1     Running   0          73s   192.168.1.16   node2   <none>           <none>
pod/kube-flannel-ds-sz7x7      1/1     Running   0          34m   192.168.1.15   node1   <none>           <none>

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
service/kube-dns   ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP,9153/TCP   26m   k8s-app=kube-dns

部署 Dashboard

配置檔案及映象在下載包裡直接使用。

修改yaml配置檔案使其埠暴露外部訪問

# 預設Dashboard只能叢集內部訪問，修改Service為NodePort型別，暴露到外部：

[[email protected] /usr/local/src/offline-k8s-packages/dashboard]#vim recommended.yaml
...
  ports:
    - port: 443
      targetPort: 8443 
      nodePort: 30001 # 修改這裡
  type: NodePort #修改這裡
...

[[email protected] /usr/local/src/offline-k8s-packages/dashboard]#kubectl apply -f recommended.yaml

[[email protected] ~]#kubectl get pods,svc -n kubernetes-dashboard -o wide
NAME                                             READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
pod/dashboard-metrics-scraper-6b4884c9d5-kg98t   1/1     Running   0          42s   10.244.1.3   node2   <none>           <none>
pod/kubernetes-dashboard-7d8574ffd9-dj6cx        1/1     Running   0          42s   10.244.1.2   node2   <none>           <none>

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE   SELECTOR
service/dashboard-metrics-scraper   ClusterIP   10.0.0.177   <none>        8000/TCP        42s   k8s-app=dashboard-metrics-scraper
service/kubernetes-dashboard        NodePort    10.0.0.3     <none>        443:30001/TCP   42s   k8s-app=kubernetes-dashboard

建立 service account 並繫結預設 cluster-admin 管理員叢集角色

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

# 接下來訪問https://node ip:30001
# 然後將上面過濾出來的token複製上面即可訪問dashboard