02-安裝etcd叢集.md

下載etcd/etcdctl 二進位制檔案、建立證書目錄

建立etcd證書請求 etcd-csr.json.j2

  首先判斷下是否etcd 證書已經存在，如果已經存在就跳過證書生成步驟

{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "{{ inventory_hostname }}"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "HangZhou"
 
,
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

  etcd使用對等證書，hosts 欄位必須指定授權使用該證書的 etcd 節點 IP

建立證書和私鑰

cd /etc/etcd/ssl && {{ bin_dir }}/cfssl gencert \
        -ca={{ ca_dir }}/ca.pem \
        -ca-key={{ ca_dir }}/ca-key.pem \
        -config={{ ca_dir }}/ca-config.json \
        -profile=kubernetes etcd-csr.json | 
 
{{ bin_dir }}/cfssljson -bare etcd

建立etcd 服務檔案 etcd.service.j2

  先建立工作目錄 /var/lib/etcd/

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart={{ bin_dir }}/etcd \
  --name={{ NODE_NAME
 
 }} \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file={{ ca_dir }}/ca.pem \
  --peer-trusted-ca-file={{ ca_dir }}/ca.pem \
  --initial-advertise-peer-urls=https://{{ inventory_hostname }}:2380 \
  --listen-peer-urls=https://{{ inventory_hostname }}:2380 \
  --listen-client-urls=https://{{ inventory_hostname }}:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://{{ inventory_hostname }}:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster={{ ETCD_NODES }} \
  --initial-cluster-state=new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

  完整引數列表請使用 etcd –help 查詢

啟動etcd服務

systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

驗證etcd叢集狀態

  systemctl status etcd 檢視服務狀態
  journalctl -u etcd 檢視執行日誌
  在任一 etcd 叢集節點上執行如下命令

# 根據hosts中配置設定shell變數 $NODE_IPS，這裡是三臺
export NODE_IPS="192.168.1.1 192.168.1.2 192.168.1.3"
$ for ip in ${NODE_IPS}; do
  ETCDCTL_API=3 etcdctl \
  --endpoints=https://${ip}:2379  \
  --cacert=/etc/kubernetes/ssl/ca.pem \
  --cert=/etc/etcd/ssl/etcd.pem \
  --key=/etc/etcd/ssl/etcd-key.pem \
  endpoint health; done

  結果：

https://192.168.1.1:2379 is healthy: successfully committed proposal: took = 2.210885ms
https://192.168.1.2:2379 is healthy: successfully committed proposal: took = 2.784043ms
https://192.168.1.3:2379 is healthy: successfully committed proposal: took = 3.275709ms

03-安裝docker服務.md

建立docker的systemd unit檔案

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
Environment="PATH={{ bin_dir }}:/bin:/sbin:/usr/bin:/usr/sbin"
ExecStart={{ bin_dir }}/dockerd
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

  dockerd 執行時會呼叫其它 docker 命令，如 docker-proxy，所以需要將 docker 命令所在的目錄加到 PATH 環境變數中；
  docker 從 1.13 版本開始，將iptables 的filter 表的FORWARD 鏈的預設策略設定為DROP，從而導致 ping 其它 Node 上的 Pod IP 失敗，因此必須在 filter 表的FORWARD 鏈增加一條預設允許規則 iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
  執行dockerd –help 檢視所有可配置引數，確保預設開啟 –iptables 和 –ip-masq 選項

配置國內映象加速

  從國內下載docker官方倉庫映象非常緩慢，所以對於k8s叢集來說配置映象加速非常重要，配置 /etc/docker/daemon.json

{
  "registry-mirrors": ["https://registry.docker-cn.com"],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    }
}

  由於K8S的官方映象存放在gcr.io倉庫，因此這個映象加速對K8S的官方映象沒有效果；好在Docker Hub上有很多K8S映象的轉存，而Docker Hub上的映象可以加速。

清理 iptables

iptables -F && iptables -X \
        && iptables -F -t nat && iptables -X -t nat \
        && iptables -F -t raw && iptables -X -t raw \
        && iptables -F -t mangle && iptables -X -t mangle

  calico 網路支援 network-policy，使用的calico-kube-controllers 會使用到iptables 所有的四個表 filter nat raw mangle，所以一併清理。

啟動 docker

安裝docker查詢映象 tag的小工具

$ docker-tag library/ubuntu
"14.04"
"16.04"
"17.04"
"latest"
"trusty"
"trusty-20171117"
"xenial"
"xenial-20171114"
"zesty"
"zesty-20171114"
$ docker-tag mirrorgooglecontainers/kubernetes-dashboard-amd64
"v0.1.0"
"v1.0.0"
"v1.0.0-beta1"
"v1.0.1"
"v1.1.0-beta1"
"v1.1.0-beta2"
"v1.1.0-beta3"
"v1.7.0"
"v1.7.1"
"v1.8.0"

  需要先apt安裝輕量JSON處理程式 jq
  然後下載指令碼即可使用
  指令碼很簡單，就一行命令如下

#!/bin/bash
curl -s -S "https://registry.hub.docker.com/v2/repositories/[email protected]/tags/" | jq '."results"[]["name"]' |sort

  對於 CentOS7 安裝 jq 稍微費力一點，需要啟用 EPEL 源

wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum install jq

驗證
  執行ansible-playbook 03.docker.yml 成功後可以驗證

systemctl status docker     # 服務狀態
journalctl -u docker        # 執行日誌
docker version
docker info

  iptables-save|grep FORWARD 檢視 iptables filter表 FORWARD鏈，最後要有一個 -A FORWARD -j ACCEPT 保底允許規則。

iptables-save|grep FORWARD
:FORWARD ACCEPT [0:0]
:FORWARD DROP [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j ACCEPT