k8s附加組件之存儲-glusterfs
阿新 • • 發佈：2018-01-19
現在 個數 oot session rwx efault 答案 .org 我們 k8s附加組件之存儲-glusterfs
2018/1/16
部署 glusterfs 集群
- 初始化 glusterfs 集群
- 創建 glusterfs 卷，用於保存數據
不做特別說明的，都在 67 上操作
集群節點
- 10.10.9.67
- 10.10.9.68
- 10.10.9.69
初始化 glusterfs 集群
~]# yum install centos-release-gluster310 -y ~]# yum install glusterfs-server -y ~]# yum install xfsprogs -y ##### 準備磁盤： ~]# umount /data1 ~]# mkfs.xfs -i size=512 -f /dev/vdc1 ~]# mount /dev/vdc1 /data1 ~]# vim /etc/fstab ### glusterfs /dev/vdc1 /data1 xfs defaults 0 0 ##### 啟動服務： ~]# systemctl start glusterd && systemctl enable glusterd ##### 建立集群： ~]# gluster peer probe 10.10.9.68 ~]# gluster peer probe 10.10.9.69 ~]# gluster peer status ##### 在所有節點操作： ~]# mkdir /data1/glusterfs_data/gv0 -p && echo ‘### glusterfs data for k8s-dev only!‘ >/data1/glusterfs_data/README.md
創建 glusterfs 卷，用於保存數據
~]# gluster volume create gv0 replica 3 transport tcp 10.10.9.67:/data1/glusterfs_data/gv0 10.10.9.68:/data1/glusterfs_data/gv0 10.10.9.69:/data1/glusterfs_data/gv0 ~]# gluster volume start gv0 ~]# gluster volume info gv0 ##### 在客戶端上配置 glusterfs 環境並掛載測試 ~]# yum install centos-release-gluster310 -y ~]# yum install glusterfs glusterfs-fuse -y ~]# mkdir /mnt/test ~]# mount -t glusterfs -o backup-volfile-servers=10.10.9.68:10.10.9.69 10.10.9.67:/gv0 /mnt/test ~]# df -h |grep test 10.10.9.67:/gv0 1.0T 33M 1.0T 1% /mnt/test ~]# umount /mnt/test
準備 k8s yaml 來使用 glusterfs 集群
- 創建一個數據卷
- 通過定義 endpoints 和 service 來提供訪問外部服務（glusterfs集群）的方法
- 通過定義 pv 和 pvc 來使用存儲卷
- 測試，在 deployment 中通過關聯 pvc 來掛載卷
- 測試，多副本讀寫同一個日誌文件的場景
創建一個數據卷
[root@tvm-00 glusterfs]# ls 10.endpoints 20.pv 30.pvc bin deploy_test [root@tvm-00 glusterfs]# cat bin/create-gv1-default.sh #!/bin/bash # gluster volume create gv1-default replica 3 transport tcp 10.10.9.67:/data1/glusterfs_data/gv1-default 10.10.9.68:/data1/glusterfs_data/gv1-default 10.10.9.69:/data1/glusterfs_data/gv1-default gluster volume start gv1-default gluster volume info gv1-default
通過定義 endpoints 和 service 來提供訪問外部服務（glusterfs集群）的方法
[root@tvm-00 glusterfs]# cat 10.endpoints/glusterfs-r3/default.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: glusterfs-r3
namespace: default
subsets:
- addresses:
- ip: 10.10.9.67
- ip: 10.10.9.68
- ip: 10.10.9.69
ports:
- port: 49152
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: glusterfs-r3
namespace: default
spec:
ports:
- port: 49152
protocol: TCP
targetPort: 49152
sessionAffinity: None
type: ClusterIP
[root@tvm-00 glusterfs]# kubectl apply -f 10.endpoints/glusterfs-r3/default.yaml
##### 查看狀態：
[root@tvm-00 glusterfs]# kubectl get ep glusterfs-r3
NAME ENDPOINTS AGE
glusterfs-r3 10.10.9.67:49152,10.10.9.68:49152,10.10.9.69:49152 4h
通過定義 pv 和 pvc 來使用存儲卷
##### 註意：pv 是不區分 namespace 的資源，但又和不同 namespace 下的資源交互，此處有點奇怪，不知為何如此設計
[root@tvm-00 glusterfs]# cat 20.pv/glusterfs-r3/gv1-default.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: glusterfs-r3-gv1-default
labels:
type: glusterfs
spec:
storageClassName: gv1-default
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs-r3"
path: "gv1-default"
readOnly: false
[root@tvm-00 glusterfs]# cat 30.pvc/glusterfs-r3/gv1-default.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: glusterfs-r3-gv1-default
namespace: default
spec:
storageClassName: gv1-default
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
[root@tvm-00 glusterfs]# kubectl apply -f 20.pv/glusterfs-r3/gv1-default.yaml
[root@tvm-00 glusterfs]# kubectl apply -f 30.pvc/glusterfs-r3/gv1-default.yaml
##### 查看狀態：
[root@tvm-00 glusterfs]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv/glusterfs-r3-gv1-default 100Gi RWX Retain Bound default/glusterfs-r3-gv1-default gv1-default 46m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc/glusterfs-r3-gv1-default Bound glusterfs-r3-gv1-default 100Gi RWX gv1-default 44m
測試，在 deployment 中通過關聯 pvc 來掛載卷
[root@tvm-00 glusterfs]# cat deploy_test/gv1-default-t1.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: gv1-default-t1
namespace: default
labels:
app.name: gv1-default-t1
spec:
replicas: 1
selector:
matchLabels:
app.name: gv1-default-t1
template:
metadata:
labels:
app.name: gv1-default-t1
spec:
containers:
- name: nginx-test
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: glusterfsvol
mountPath: "/mnt/glusterfsvol"
volumes:
- name: glusterfsvol
persistentVolumeClaim:
claimName: glusterfs-r3-gv1-default
[root@tvm-00 glusterfs]# kubectl apply -f deploy_test/gv1-default-t1.yaml --record
##### 查看狀態：
[root@tvm-00 glusterfs]# kubectl get deploy,po
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/gv1-default-t1 1 1 1 1 33s
NAME READY STATUS RESTARTS AGE
po/gv1-default-t1-848455b8b6-fldkb 1/1 Running 0 33s
##### 驗證掛載的數據盤：
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- mount | grep gluster
10.10.9.67:gv1-default on /mnt/glusterfsvol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- df -h | grep glusterfsvol
10.10.9.67:gv1-default 1.0T 33M 1.0T 1% /mnt/glusterfsvol
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- ls -l /mnt/glusterfsvol
total 0
##### 讀寫測試：
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- sh -c ‘echo " [$(date)] writeFrom: $(hostname)" >>/mnt/glusterfsvol/README.md‘
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- cat /mnt/glusterfsvol/README.md
[Tue Jan 16 09:50:01 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-fldkb
[Tue Jan 16 09:50:40 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-fldkb
##### 符合預期
測試，多副本讀寫同一個日誌文件的場景
##### 之前測試的都是 1 個 pod 副本的場景，現在，我們擴容到 3 個副本，然後再看看讀寫狀況：
replicas: 1
->
replicas: 3
[root@tvm-00 glusterfs]# kubectl scale --replicas=3 -f deploy_test/gv1-default-t1.yaml
##### 查看狀態：
[root@tvm-00 glusterfs]# kubectl get deploy,po
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/gv1-default-t1 3 3 3 3 20m
NAME READY STATUS RESTARTS AGE
po/gv1-default-t1-848455b8b6-fldkb 1/1 Running 0 20m
po/gv1-default-t1-848455b8b6-gzd4s 1/1 Running 0 1m
po/gv1-default-t1-848455b8b6-t6bsk 1/1 Running 0 1m
##### 分別在 3 個 pod 中往同一個文件寫入一條數據
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- sh -c ‘echo " [$(date)] writeFrom: $(hostname)" >>/mnt/glusterfsvol/README.md‘
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-gzd4s -- sh -c ‘echo " [$(date)] writeFrom: $(hostname)" >>/mnt/glusterfsvol/README.md‘
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-t6bsk -- sh -c ‘echo " [$(date)] writeFrom: $(hostname)" >>/mnt/glusterfsvol/README.md‘
##### 驗證：
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-fldkb -- cat /mnt/glusterfsvol/README.md
[Tue Jan 16 09:50:01 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-fldkb
[Tue Jan 16 09:50:40 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-fldkb
[Tue Jan 16 09:55:53 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-fldkb
[Tue Jan 16 09:55:59 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-gzd4s
[Tue Jan 16 09:56:04 UTC 2018] writeFrom: gv1-default-t1-848455b8b6-t6bsk
##### 符合預期，但有一個細節請註意，，如何合理的記錄日誌以便於區分某條記錄是來自哪一個 pod 實例？
##### 答案是：寫入日誌時，業務層應提取類似 hostname 那樣的標誌符來寫入日誌中
[root@tvm-00 glusterfs]# kubectl exec gv1-default-t1-848455b8b6-t6bsk -- hostname
gv1-default-t1-848455b8b6-t6bsk
如何快速的為指定 namespace 部署一個 volume
##### 目標：
##### namespace=ns-test1
##### 準備：
cp -a bin/create-gv1-default.sh bin/create-gv1-ns-test1.sh
sed -i ‘s/default/ns-test1/g‘ bin/create-gv1-ns-test1.sh
sh bin/create-gv1-ns-test1.sh
cp -a 10.endpoints/glusterfs-r3/default.yaml 10.endpoints/glusterfs-r3/ns-test1.yaml
sed -i ‘s/default/ns-test1/g‘ 10.endpoints/glusterfs-r3/ns-test1.yaml
cp -a 20.pv/glusterfs-r3/gv1-default.yaml 20.pv/glusterfs-r3/gv1-ns-test1.yaml
sed -i ‘s/default/ns-test1/g‘ 20.pv/glusterfs-r3/gv1-ns-test1.yaml
cp -a 30.pvc/glusterfs-r3/gv1-default.yaml 30.pvc/glusterfs-r3/gv1-ns-test1.yaml
sed -i ‘s/default/ns-test1/g‘ 30.pvc/glusterfs-r3/gv1-ns-test1.yaml
cp -a deploy_test/gv1-default-t1.yaml deploy_test/gv1-ns-test1-t1.yaml
sed -i ‘s/default/ns-test1/g‘ deploy_test/gv1-ns-test1-t1.yaml
##### 部署：
kubectl apply -f 10.endpoints/glusterfs-r3/ns-test1.yaml
kubectl apply -f 20.pv/glusterfs-r3/gv1-ns-test1.yaml
kubectl apply -f 30.pvc/glusterfs-r3/gv1-ns-test1.yaml
kubectl apply -f deploy_test/gv1-ns-test1-t1.yaml
##### 驗證：
[root@tvm-00 glusterfs]# kubectl -n ns-test1 get ep,svc |grep ‘glusterfs-r3‘
ep/glusterfs-r3 10.10.9.67:49152,10.10.9.68:49152,10.10.9.69:49152 2m
svc/glusterfs-r3 ClusterIP 10.107.81.35 <none> 49152/TCP 2m
[root@tvm-00 glusterfs]# kubectl -n ns-test1 get pv,pvc |grep ‘ns-test1‘
pv/glusterfs-r3-gv1-ns-test1 100Gi RWX Retain Bound ns-test1/glusterfs-r3-gv1-ns-test1 gv1-ns-test1 2m
pvc/glusterfs-r3-gv1-ns-test1 Bound glusterfs-r3-gv1-ns-test1 100Gi RWX gv1-ns-test1 2m
[root@tvm-00 glusterfs]# kubectl -n ns-test1 get deploy,po |grep ‘ns-test1‘
deploy/gv1-ns-test1-t1 1 1 1 1 3m
po/gv1-ns-test1-t1-7f986bfbf8-v5zmz 1/1 Running 0 3m
[root@tvm-00 glusterfs]# kubectl -n ns-test1 exec gv1-ns-test1-t1-7f986bfbf8-v5zmz -- df -h |grep gluster
10.10.9.67:gv1-ns-test1 1.0T 36M 1.0T 1% /mnt/glusterfsvol
ZYXW、參考
- glusterfs
- CentOS Storage
- k8s-glusterfs
- 使用glusterfs做持久化存儲
k8s附加組件之存儲-glusterfs