kubernetes中部署Heketi和GlusterFS(二)
阿新 • • 發佈：2018-07-09
sta 自動創建 bdc ber ESS ddr ubunt cto cat kubernetes中部署Heketi和GlusterFS(二)
在上一節中，Heketi的部署方式還不能用於生產環境，因為Heketi Pod的數據並沒有持久化，容易導致heketi的數據丟失，Heketi的數據保存在/var/lib/heketi/heketi.db文件中，因此需要把此目錄掛載到GlusterFS分布式存儲中。
在上一節中，Heketi的部署方式還不能用於生產環境，因為Heketi Pod的數據並沒有持久化，容易導致heketi的數據丟失，Heketi的數據保存在/var/lib/heketi/heketi.db文件中，因此需要把此目錄掛載到GlusterFS分布式存儲中。
按照上一節的步驟，執行heketi-cli topology load --json=topology-sample.json
$ echo $HEKETI_CLI_SERVER http://10.254.49.43:8080 $ heketi-cli topology load --json=topology-sample.json Creating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516 Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b Adding device /dev/sdb ... OK Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a Adding device /dev/sdb ... OK Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b Adding device /dev/sdb ... OK
緊接著執行：將生成heketi-storage.json文件。
$ heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json執行完後，查看Pod deploy-heketi日誌信息，看看做了哪些事：
#只截取了部分日誌，基本操作就是進入到各個glusterfs Pod創建brick目錄及創建一個副本為3的Replicate volume, volume名為heketidbstorage [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6 Result: [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4 Result: [kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick [kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick Result: [cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3 [kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick Result: volume create: heketidbstorage: success: please start the volume to access data
進入任意GlusterFS Pod查看卷信息：
$ kubectl exec glusterfs-25cm8 -it bash [root@ubuntu16 /]# gluster volume info Volume Name: heketidbstorage Type: Replicate Volume ID: c8da2a4a-3066-4708-a59d-201d22decd92 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick Brick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick Brick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick Options Reconfigured: transport.address-family: inet nfs.disable: on [root@ubuntu16 /]#
分析下heketi-storage.json：
#將創建如下資源信息：
Endpoints：
name：heketi-storage-endpoints
10.30.1.16:1 10.30.1.15:1 10.30.1.17:1
Service：
name: heketi-storage-endpoints
job：
name: heketi-storage-copy-job
images: heketi/heketi:dev
聲明了 volume：heketi-storage
"volumes": [
{
"name": "heketi-storage",
"glusterfs": {
"endpoints": "heketi-storage-endpoints",
"path": "heketidbstorage"
}
},
掛載到 /heketi：
volumeMounts": [
{
"name": "heketi-storage",
"mountPath": "/heketi"
},
啟動時執行命令：cp /db/heketi.db /heketi
#由此可知，此job的作用就是復制heketi中的數據文件到 /heketi，而/heketi目錄掛載在了卷heketi-storage中，而heketi-storage volume是前面執行"heketi-cli setup-openshift-heketi-storage"時創建好了的創建之：
$ kubectl create -f heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created當Job執行完後就可以刪除它了：
$ kubectl get job
NAME DESIRED SUCCESSFUL AGE
heketi-storage-copy-job 1 1 1m等到job完成後，刪除bootstrap Heketi實例相關的組件：
#把之前由heketi-bootstrap.json創建的資源刪除
$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
deployment "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-69bfbd4bbd-q8tsk" deleted
service "deploy-heketi" deleted
secret "heketi-storage-secret" deleted之前創建的名為deploy-heketi的pod,service已經刪除了：
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
glusterfs-25cm8 1/1 Running 1 1h
glusterfs-8qrpt 1/1 Running 1 1h
glusterfs-c4859 1/1 Running 1 1h
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heketi-storage-endpoints ClusterIP 10.254.191.233 <none> 1/TCP 4m最後，使用heketi-deployment.json文件重新部署heketi
$ cat heketi-deployment.json
{
"kind": "List",
"apiVersion": "v1",
"items": [
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "heketi-db-backup",
"labels": {
"glusterfs": "heketi-db",
"heketi": "db"
}
},
"data": {
},
"type": "Opaque"
},
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-service",
"deploy-heketi": "support"
},
"annotations": {
"description": "Exposes Heketi Service"
}
},
"spec": {
"selector": {
"name": "heketi"
},
"ports": [
{
"name": "heketi",
"port": 8080,
"targetPort": 8080
}
]
}
},
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-deployment"
},
"annotations": {
"description": "Defines how to deploy Heketi"
}
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"name": "heketi",
"labels": {
"name": "heketi",
"glusterfs": "heketi-pod"
}
},
"spec": {
"serviceAccountName": "heketi-service-account",
"containers": [
{
"image": "heketi/heketi:dev",
"imagePullPolicy": "Always",
"name": "heketi",
"env": [
{
"name": "HEKETI_EXECUTOR",
"value": "kubernetes"
},
{
"name": "HEKETI_DB_PATH",
"value": "/var/lib/heketi/heketi.db"
},
{
"name": "HEKETI_FSTAB",
"value": "/var/lib/heketi/fstab"
},
{
"name": "HEKETI_SNAPSHOT_LIMIT",
"value": "14"
},
{
"name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
"value": "y"
}
],
"ports": [
{
"containerPort": 8080
}
],
"volumeMounts": [
{
"mountPath": "/backupdb",
"name": "heketi-db-secret"
},
{
"name": "db",
"mountPath": "/var/lib/heketi"
},
{
"name": "config",
"mountPath": "/etc/heketi"
}
],
"readinessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 3,
"httpGet": {
"path": "/hello",
"port": 8080
}
},
"livenessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 30,
"httpGet": {
"path": "/hello",
"port": 8080
}
}
}
],
"volumes": [
{
"name": "db",
"glusterfs": {
"endpoints": "heketi-storage-endpoints",
"path": "heketidbstorage"
}
},
{
"name": "heketi-db-secret",
"secret": {
"secretName": "heketi-db-backup"
}
},
{
"name": "config",
"secret": {
"secretName": "heketi-config-secret"
}
}
]
}
}
}
}
]
}
heketi-deployment.json 文件 創建了如下資源：
Service:
name: heketi
port: 8080
Deployment：
name: heketi
replicas: 1
image: heketi/heketi:dev
volumeMounts:
name: db mountPath: /var/lib/heketi
volumes:
endpoints: heketi-storage-endpoints #由heketi-storage.json文件創建
path：heketidbstorage #這是gluster volume名，此volume是由"heketi-cli setup-openshift-heketi-storage"自動創建。
# 將heketi容器內的/var/lib/heketi 目錄掛載到了GlusterFS volume “heketidbstorage”中。部署之：
$ kubectl create -f heketi-deployment.json
secret "heketi-db-backup" created
service "heketi" created
deployment "heketi" created
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
heketi 1 1 1 1 45s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heketi ClusterIP 10.254.239.189 <none> 8080/TCP 51s
heketi-storage-endpoints ClusterIP 10.254.191.233 <none> 1/TCP 31m驗證heketi是否在用在用gluster volume：
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
glusterfs-25cm8 1/1 Running 1 1h
glusterfs-8qrpt 1/1 Running 1 1h
glusterfs-c4859 1/1 Running 1 1h
heketi-7898db85dd-nb6kn 1/1 Running 0 1m
$ kubectl exec heketi-7898db85dd-nb6kn -it bash
[root@heketi-7898db85dd-nb6kn /]# mount |grep heketi
10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)至此，heketi db已正確配置了GlusterFS卷。
kubernetes中部署Heketi和GlusterFS(二)