k8s -- etcd叢集部署
阿新 • • 發佈：2021-02-10
部署etcd叢集 叢集規劃
主機名 角色 ip
HDss-1-12.host.com ectc lead 192.168.1.12
HDss-1-21.host.com ectc follow 192.168.1.21
HDss-1-22.host.com ectc follow 192.168.1.22部署master節點服務
1. 在主配置伺服器上生成證書 編輯 ca-config.json檔案
{ "signing": { "default": { "expiry": "175200h" }, "profiles": { "server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }
2.在主配置伺服器上生成證書 編輯 etcd-peer-csr.json檔案
{ "signing": { "default": { "expiry": "175200h" }, "profiles": { "server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }
3.將編輯好的json檔案上傳到主控服務端
[[email protected] certs]# ll total 28 -rw-r--r-- 1 root root 836 Feb 9 16:52 ca-config.json -rw-r--r-- 1 root root 993 Feb 9 15:30 ca.csr -rw-r--r-- 1 root root 346 Feb 9 15:29 ca-csr.json -rw------- 1 root root 1675 Feb 9 15:30 ca-key.pem -rw-r--r-- 1 root root 1346 Feb 9 15:30 ca.pem -rw-r--r-- 1 root root 375 Feb 9 16:52 etcd-peer-csr.json
4.生成承載式證書如下：
[[email protected] certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
2021/02/09 16:53:42 [INFO] generate received request
2021/02/09 16:53:42 [INFO] received CSR
2021/02/09 16:53:42 [INFO] generating key: rsa-2048
2021/02/09 16:53:43 [INFO] encoded CSR
2021/02/09 16:53:43 [INFO] signed certificate with serial number 376584182207976051180015643037273875708040992548
2021/02/09 16:53:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[[email protected] certs]# ll
total 40
-rw-r--r-- 1 root root 836 Feb 9 16:52 ca-config.json
-rw-r--r-- 1 root root 993 Feb 9 15:30 ca.csr
-rw-r--r-- 1 root root 346 Feb 9 15:29 ca-csr.json
-rw------- 1 root root 1675 Feb 9 15:30 ca-key.pem
-rw-r--r-- 1 root root 1346 Feb 9 15:30 ca.pem
-rw-r--r-- 1 root root 1062 Feb 9 16:53 etcd-peer.csr
-rw-r--r-- 1 root root 375 Feb 9 16:53 etcd-peer-csr.json
-rw------- 1 root root 1675 Feb 9 16:53 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Feb 9 16:53 etcd-peer.pem
[[email protected] certs]#5. 開始部署etcd叢集在12 21 22三臺伺服器上，這裡部署文件以hdss-1-12.host.com主機為例，另外兩臺安裝部署方法類似
建立使用者
[[email protected] ~]# useradd -s /sbin/nologin -M etcd
[[email protected] ~]# id etcd
uid=1001(etcd) gid=1001(etcd) groups=1001(etcd)6.下載軟體並解壓(下載etcd軟體，建議用不超3.3的版本)
[[email protected] ~]# mkdir /opt/src
[[email protected] ~]# cd /opt/src/
[[email protected] src]# ls
etcd-v3.1.20-linux-amd64.tar.gz
[[email protected] src]# tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt/7.建立軟連線，方便後續維護
[[email protected] src]# cd /opt
[[email protected] opt]# mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20
[[email protected] opt]# ln -s /opt/etcd-v3.1.20 /opt/etcd
[[email protected] opt]# ll
total 0
lrwxrwxrwx 1 root root 17 Feb 9 17:11 etcd -> /opt/etcd-v3.1.20
drwxr-xr-x 3 478493 89939 123 Oct 11 2018 etcd-v3.1.20
drwxr-xr-x. 2 root root 6 Oct 31 2018 rh
drwxr-xr-x 2 root root 45 Feb 9 17:09 src
[[email protected] opt]# 8. 建立目錄，拷貝證書、私鑰
[[email protected] ~]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server9.將運維主機上生成的ca.pem etc-peer-key.pem etc-peer.pem 拷貝到/opt/etcd/certs目錄中，私鑰檔案許可權為600
[[email protected] etcd]# cd certs/
[[email protected] certs]# pwd
/opt/etcd/certs
[[email protected] certs]# ls
[[email protected] certs]# scp hdss-1-200:/opt/certs/ca.pem .
[[email protected] certs]# scp hdss-1-200:/opt/certs/etcd-peer.pem .
[[email protected] certs]# scp hdss-1-200:/opt/certs/etcd-peer-key.pem .
[[email protected] certs]# ll
total 12
-rw-r--r-- 1 root root 1346 Feb 9 17:19 ca.pem
-rw------- 1 root root 1675 Feb 9 17:22 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Feb 9 17:20 etcd-peer.pem10.更改屬主屬組
[[email protected] certs]# chown -R etcd.etcd /opt/etcd/certs /data/etcd /data/logs/etcd-server
[[email protected] certs]# ll
total 12
-rw-r--r-- 1 etcd etcd 1346 Feb 9 17:19 ca.pem
-rw------- 1 etcd etcd 1675 Feb 9 17:22 etcd-peer-key.pem
-rw-r--r-- 1 etcd etcd 1428 Feb 9 17:20 etcd-peer.pem
[[email protected] certs]# 11.建立etcd服務啟動指令碼IP地址改成本機IP etcd-server-startup.sh 並上傳
#!/bin/sh
./etcd --name etcd-server-1-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.12:2380 \
--listen-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.12:2380 \
--advertise-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout12 賦權並修改屬主和屬組
[[email protected] etcd]# chmod +x /opt/etcd/etcd-server-startup.sh
[[email protected] etcd]# chown -R etcd.etcd /opt/etcd-v3.1.20/ /data/etcd /data/logs/etcd-server
[[email protected] etcd]# 13.為了使使etcd後端執行，需要安裝supervisor (管理後臺程式的軟體)
[[email protected] etcd]# yum install supervisor -y14.啟動supervisor
[[email protected] etcd]# systemctl start supervisord
[[email protected] etcd]# systemctl enable supervisord
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.15.更改supervisord的配置檔案 etcd-server.ini
[email protected] supervisord.d]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)16.建立後端啟動etcd
[[email protected] supervisord.d]# supervisorctl update
etcd-server-1-12: added process group
[[email protected] supervisord.d]# supervisorctl status
etcd-server-1-12 STARTING
[[email protected] supervisord.d]# netstat -luntp|grep etcd
tcp 0 0 192.168.1.12:2379 0.0.0.0:* LISTEN 60692/./etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 60692/./etcd
tcp 0 0 192.168.1.12:2380 0.0.0.0:* LISTEN 60692/./etcd 17.另外兩臺部署流程和上面一樣，需要修改倆配置檔案
(1) 21上的配置檔案etcd-server-startup.sh
[[email protected] etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-1-21 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.21:2380 \
--listen-client-urls https://192.168.1.21:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.21:2380 \
--advertise-client-urls https://192.168.1.21:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout （2）21 etcd-server.ini 配置檔案
[[email protected] etcd]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-21]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)(3) 22上的配置檔案etcd-server-startup.sh
[[email protected] etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-1-22 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://192.168.1.22:2380 \
--listen-client-urls https://192.168.1.22:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.1.22:2380 \
--advertise-client-urls https://192.168.1.22:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-1-12=https://192.168.1.12:2380,etcd-server-1-21=https://192.168.1.21:2380,etcd-server-1-22=https://192.168.1.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout （4）22 etcd-server.ini 配置檔案
[[email protected] etcd]# vi /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-22]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)18.如果有報錯可以檢視日誌
[[email protected] supervisord.d]# tail -fn 200 /data/logs/etcd-server/etcd.stdout.log檢視叢集狀態是否正常
[[email protected] etcd]# ./etcdctl cluster-health
[[email protected] etcd]# ./etcdctl member list
