zookeeper和etcd有狀態服務部署
阿新 • • 發佈：2019-01-08
zookeeper和etcd有狀態服務部署實踐
4k 次閱讀 · 讀完需要 78 分鐘 0一. 概述
kubernetes通過statefulset為zookeeper、etcd等這類有狀態的應用程式提供完善支援，statefulset具備以下特性：
- 為pod提供穩定的唯一的網路標識
- 穩定值持久化儲存：通過pv/pvc來實現
- 啟動和停止pod保證有序：優雅的部署和伸縮性
本文闡述瞭如何在k8s叢集上部署zookeeper和etcd有狀態服務，並結合ceph實現資料持久化。
二. 總結
- 使用k8s的statefulset、storageclass、pv、pvc和ceph的rbd，能夠很好的支援zookeeper、etcd這樣的有狀態服務部署到kubernetes叢集上。
- k8s不會主動刪除已經建立的pv、pvc物件，防止出現誤刪。
如果使用者確定刪除pv、pvc物件，同時還需要手動刪除ceph段的rbd映象。
- 遇到的坑
storageclass中引用的ceph客戶端使用者，必須要有mon rw，rbd rwx許可權。如果沒有mon write許可權，會導致釋放rbd鎖失敗，無法將rbd映象掛載到其他的k8s worker節點。
- zookeeper使用探針檢查zookeeper節點的健康狀態，如果節點不健康，k8s將刪除pod，並自動重建該pod，達到自動重啟zookeeper節點的目的。
因zookeeper 3.4版本的叢集配置，是通過靜態載入檔案zoo.cfg來實現的，所以當zookeeper節點pod ip變動後，需要重啟zookeeper叢集中的所有節點。
- etcd部署方式有待優化
本次試驗中使用靜態方式部署etcd叢集，如果etcd節點變遷時，需要執行etcdctl member remove/add等命令手動配置etcd叢集，嚴重限制了etcd叢集自動故障恢復、擴容縮容的能力。因此，需要考慮對部署方式優化，改為使用DNS或者etcd descovery的動態方式部署etcd，才能讓etcd更好的執行在k8s上。
三. zookeeper叢集部署
1. 下載映象
docker pull gcr.mirrors.ustc.edu.cn/google_containers/kubernetes-zookeeper:1.0-3.4.10
docker tag gcr.mirrors.ustc.edu.cn/google_containers/kubernetes-zookeeper:1.0-3.4.10 172.16.18.100:5000/gcr.io/google_containers/kubernetes-zookeeper:1.0-3.4.10
docker push 172.16.18.100:5000/gcr.io/google_containers/kubernetes-zookeeper:1.0-3.4.102. 定義ceph secret
cat << EOF | kubectl create -f -
apiVersion: v1
data:
key: QVFBYy9ndGFRUno4QlJBQXMxTjR3WnlqN29PK3VrMzI1a05aZ3c9PQo=
kind: Secret
metadata:
creationTimestamp: 2017-11-20T10:29:05Z
name: ceph-secret
namespace: default
resourceVersion: "2954730"
selfLink: /api/v1/namespaces/default/secrets/ceph-secret
uid: a288ff74-cddd-11e7-81cc-000c29f99475
type: kubernetes.io/rbd
EOF3. 定義storageclass rbd儲存
cat << EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph
parameters:
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: default
fsType: ext4
imageFormat: "2"
imagefeatures: layering
monitors: 172.16.13.223
pool: k8s
userId: admin
userSecretName: ceph-secret
provisioner: kubernetes.io/rbd
reclaimPolicy: Delete
EOF4. 建立zookeeper叢集
使用rbd儲存zookeeper節點資料
cat << EOF | kubectl create -f -
---
apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
spec:
ports:
- port: 2888
name: server
- port: 3888
name: leader-election
clusterIP: None
selector:
app: zk
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
spec:
ports:
- port: 2181
name: client
selector:
app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
selector:
matchLabels:
app: zk
maxUnavailable: 1
---
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: StatefulSet
metadata:
name: zk
spec:
selector:
matchLabels:
app: zk
serviceName: zk-hs
replicas: 3
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: zk
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: "172.16.18.100:5000/gcr.io/google_containers/kubernetes-zookeeper:1.0-3.4.10"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=512M \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
securityContext:
runAsUser: 1000
fsGroup: 1000
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.beta.kubernetes.io/storage-class: ceph
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
EOF檢視建立結果
[[email protected]172 zookeeper]# kubectl get no
NAME STATUS ROLES AGE VERSION
172.16.20.10 Ready <none> 50m v1.8.2
172.16.20.11 Ready <none> 2h v1.8.2
172.16.20.12 Ready <none> 1h v1.8.2
[ro[email protected]172 zookeeper]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE
zk-0 1/1 Running 0 8m 192.168.5.162 172.16.20.10
zk-1 1/1 Running 0 1h 192.168.2.146 172.16.20.11
[roo[email protected] zookeeper]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv/pvc-226cb8f0-d322-11e7-9581-000c29f99475 1Gi RWO Delete Bound default/datadir-zk-0 ceph 1h
pv/pvc-22703ece-d322-11e7-9581-000c29f99475 1Gi RWO Delete Bound default/datadir-zk-1 ceph 1h
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc/datadir-zk-0 Bound pvc-226cb8f0-d322-11e7-9581-000c29f99475 1Gi RWO ceph 1h
pvc/datadir-zk-1 Bound pvc-22703ece-d322-11e7-9581-000c29f99475 1Gi RWO ceph 1hzk-0 pod的rbd的鎖資訊為
[[email protected] ceph]# rbd lock list kubernetes-dynamic-pvc-227b45e5-d322-11e7-90ab-000c29f99475 -p k8s --user admin
There is 1 exclusive lock on this image.
Locker ID Address
client.24146 kubelet_lock_magic_172.16.20.10 172.16.20.10:0/1606152350 5. 測試pod遷移
嘗試將172.16.20.10節點設定為汙點，讓zk-0 pod自動遷移到172.16.20.12
kubectl cordon 172.16.20.10
[[email protected] zookeeper]# kubectl get no
NAME STATUS ROLES AGE VERSION
172.16.20.10 Ready,SchedulingDisabled <none> 58m v1.8.2
172.16.20.11 Ready <none> 2h v1.8.2
172.16.20.12 Ready <none> 1h v1.8.2
kubectl delete po zk-0觀察zk-0的遷移過程
[[email protected]172 zookeeper]# kubectl get po -owide -w
NAME READY STATUS RESTARTS AGE IP NODE
zk-0 1/1 Running 0 14m 192.168.5.162 172.16.20.10
zk-1 1/1 Running 0 1h 192.168.2.146 172.16.20.11
zk-0 1/1 Terminating 0 16m 192.168.5.162 172.16.20.10
zk-0 0/1 Terminating 0 16m <none> 172.16.20.10
zk-0 0/1 Terminating 0 16m <none> 172.16.20.10
zk-0 0/1 Terminating 0 16m <none> 172.16.20.10
zk-0 0/1 Terminating 0 16m <none> 172.16.20.10
zk-0 0/1 Terminating 0 16m <none> 172.16.20.10
zk-0 0/1 Pending 0 0s <none> <none>
zk-0 0/1 Pending 0 0s <none> 172.16.20.12
zk-0 0/1 ContainerCreating 0 0s <none> 172.16.20.12
zk-0 0/1 Running 0 3s 192.168.3.4 172.16.20.12此時zk-0正常遷移到172.16.20.12
再檢視rbd的鎖定資訊
[[email protected] ceph]# rbd lock list kubernetes-dynamic-pvc-227b45e5-d322-11e7-90ab-000c29f99475 -p k8s --user admin
There is 1 exclusive lock on this image.
Locker ID Address
client.24146 kubelet_lock_magic_172.16.20.10 172.16.20.10:0/1606152350
[[email protected] ceph]# rbd lock list kubernetes-dynamic-pvc-227b45e5-d322-11e7-90ab-000c29f99475 -p k8s --user admin
There is 1 exclusive lock on this image.
Locker ID Address
client.24154 kubelet_lock_magic_172.16.20.12 172.16.20.12:0/3715989358 之前在另外一個ceph叢集測試這個zk pod遷移的時候，總是報錯無法釋放lock，經分析應該是使用的ceph賬號沒有相應的許可權，所以導致釋放lock失敗。記錄的報錯資訊如下：
Nov 27 10:45:55 172 kubelet: W1127 10:45:55.551768 11556 rbd_util.go:471] rbd: no watchers on kubernetes-dynamic-pvc-f35a411e-d317-11e7-90ab-000c29f99475
Nov 27 10:45:55 172 kubelet: I1127 10:45:55.694126 11556 rbd_util.go:181] remove orphaned locker kubelet_lock_magic_172.16.20.12 from client client.171490: err exit status 13, output: 2017-11-27 10:45:55.570483 7fbdbe922d40 -1 did not load config file, using default settings.
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.600816 7fbdbe922d40 -1 Errors while parsing config file!
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.600824 7fbdbe922d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.600825 7fbdbe922d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.600825 7fbdbe922d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.602492 7fbdbe922d40 -1 Errors while parsing config file!
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.602494 7fbdbe922d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.602495 7fbdbe922d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
Nov 27 10:45:55 172 kubelet: 2017-11-27 10:45:55.602496 7fbdbe922d40 -1 parse_file: cannot open ceph.conf: (2) No