etcd集群搭建
阿新 • • 發佈：2018-07-06
cal eos 不能 ecs mit network 秘鑰 數字 lin
主機
kubermaster1 192.168.4.11
kubermaster2 192.168.4.12
kubermaster3 192.163.4.13系統
[root@kubermaster1 etcd-v3.2.11-linux-amd64]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core) TLS密鑰和證書
這裏部署的etcd集群使用TLS證書對證書通信進行加密，並開啟基於CA根證書簽名的雙向數字證書認證。
安裝go語言組件
登錄 https://golang.org/dl/ 找到最新版的go並下載
cd /usr/local/src wget http://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz tar -xvf go1.9.2.linux-amd64.tar.gz -C /usr/local
配置go環境
cat >> /etc/profile << EOF
#go的安裝路徑
export GOROOT=/usr/local/go
#go安裝的工具路徑
export GOPATH=/apps/local/go
export PATH=$GOROOT/bin:$PATH
EOF
source /etc/profileGOPATH和GOROOT不能相同
配置生效
[root@kubermaster2 bin]# go version
go version go1.9.2 linux/amd64安裝cfssl
將會用使用cfssl生成所需要的私鑰和證書
go get -u github.com/cloudflare/cfssl/cmd/...
會在$GOPATH/bin下安裝cfssl, cfssjosn, mkbundle等工具。
創建CA證書和私鑰，準備為etcd和其它組件辦法證書和簽名
創建ca-config.json
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "aspire": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } }
ca-config.json中可以定義多個profile，分別設置不同的expiry和usages等參數。如上面的ca-config.json中定義了名稱為aspire的profile，這個profile的expiry 87600h為10年，useages中：
- signing表示此CA證書可以用於簽名其他證書，ca.pem中的CA=TRUE
- server auth表示TLS Server Authentication
- client auth表示TLS Client Authentication
創建CA證書簽名請求配置ca-csr.json
{
"CN": "aspire",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "aspire",
"OU": "cloudnative"
}
]
}生成CA證書和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem生成etcd證書和私鑰
創建etcd證書簽名請求配置etcd-csr.json
{
"CN": "aspire.etcd",
"hosts": [
"127.0.0.1",
"192.168.4.11",
"192.168.4.12",
"192.168.4.13",
"kubermaster1",
"kubermaster2",
"kubermaster3"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "aspire.etcd",
"OU": "Operation and maintenance center"
}
]
}該"hosts"是可以使用該證書域名列表。‘CN’，kube-apiserver從證書中提取該字段作為請求的用戶名 (User Name)；瀏覽器使用該字段驗證網站是否合法；
該"names"值實際上是名稱對象的列表。每個名稱對象應至少包含一個“C”，“L”，“O”，“OU”或“ST”值（或這些的任意組合）。這些值是：
- “C”：國家
- “L”：地區或城市（如城市或城鎮名稱）
- “O”：組織 Organization，kube-apiserver從證書中提取該字段作為請求用戶所屬的組 (Group)；
- “OU”：組織單位，如負責擁有密鑰的部門; 它也可以用於“做生意”（DBS）的名稱
- “ST”：州或省
下面生成etcd的證書和私鑰：
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=aspire etcd-csr.json | cfssljson -bare etcd對生成的證書可以使用cfssl或openssl查看
$ cfssl-certinfo -cert etcd.pem
{
"subject": {
...
"cloudnative",
"aspire"
]
},
"serial_number": "555738010691550377350124675225187029254417657480",
"sans": [
"kubermaster1",
"kubermaster2",
"kubermaster3",
"127.0.0.1",
"192.168.4.11",
"192.168.4.12",
"192.168.4.13"
],
"not_before": "2017-12-18T06:57:00Z",
"not_after": "2027-12-16T06:57:00Z",
"sigalg": "SHA256WithRSA",
"authority_key_id": "DB:5D:58:25:31:D5:2A:D8:DB:C1:EF:C4:68:B4:B0:13:FA:6B:42:C3",
"subject_key_id": "6D:9B:6E:6A:F8:40:4D:4C:03:A4:0F:05:58:E1:9A:72:2E:8E:AB:58",
"pem": "-----BEGIN CERTIFICATE-----\nMIIETjCCAzagAwIBAgIUYVgnfkNJEfm75Tye3fynwTrvrogwDQYJKoZIhvcNAQEL\nBQAwaTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDA...
"
}將生成的CA證書ca.pem, etcd秘鑰etcd-key.pem, etcd證書etcd.pem拷貝到各節點的/etc/etcd/ssl目錄中
安裝etcd
下載安裝包
訪問github https://github.com/coreos/etcd/releases 找到最新安裝包並下載
cd /usr/local/src
wget https://github.com/coreos/etcd/releases/download/v3.2.11/etcd-v3.2.11-linux-amd64.tar.gz解壓縮etcd-v3.2.11-linux-amd64.tar.gz，將其中的etcd和etcdctl兩個可執行文件復制到各節點的/usr/bin目錄。
在各節點創建etcd的數據目錄：
mkdir -p /var/lib/etcd在每個節點上創建etcd的systemd unit文件/usr/lib/systemd/system/etcd.service，註意替換ETCD_NAME和INTERNAL_IP變量的值：
export ETCD_NAME=kubermaster3
export INTERNAL_IP=192.168.4.13
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name ${ETCD_NAME} --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --peer-cert-file=/etc/etcd/ssl/etcd.pem --peer-key-file=/etc/etcd/ssl/etcd-key.pem --trusted-ca-file=/etc/etcd/ssl/ca.pem --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 --listen-peer-urls https://${INTERNAL_IP}:2380 --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 --advertise-client-urls https://${INTERNAL_IP}:2379 --initial-cluster-token etcd-cluster-1 --initial-cluster node1=https://192.168.4.11:2380,node2=https://192.168.4.12:2380,node3=https://192.168.4.13:2380 --initial-cluster-state new --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF上面在啟動參數中指定了etcd的工作目錄和數據目錄是/var/lib/etcd
- --cert-file和--key-file分別指定etcd的公鑰證書和私鑰
- --peer-cert-file和--peer-key-file分別指定了etcd的Peers通信的公鑰證書和私鑰。
- --trusted-ca-file指定了客戶端的CA證書
- --peer-trusted-ca-file指定了Peers的CA證書
- --initial-cluster-state new表示這是新初始化集群，--name指定的參數值必須在--initial-cluster中
啟動etcd
在各節點啟動etcd
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd檢查集群是否健康
etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 cluster-health
2017-04-24 19:53:40.545148 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-24 19:53:40.546127 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 4f2f99d70000fc19 is healthy: got healthy result from https://192.168.61.12:2379
member 99a756f799eb4163 is healthy: got healthy result from https://192.168.61.11:2379
member a9aff19397de2e4e is healthy: got healthy result from https://192.168.61.13:2379
cluster is healthyetcd集群搭建