k8s中部署有狀態應用等需要持久化數據的應用，必不可少得用存儲，k8s支持很多中存儲方案，我司目前使用的存儲有glusterfs(分為容器化和裸機方式)、nfs供應用選用，本次就簡單實戰下glusterfs配合k8s做數據存儲。

介紹

GlusterFS系統是一個可擴展的網絡文件系統，相比其他分布式文件系統，GlusterFS具有高擴展性、高可用性、高性能、可橫向擴展等特點，並且其沒有元數據服務器的設計，讓整個服務沒有單點故障的隱患。

規劃

• 192.168.209.130 master-01
• 192.168.209.131 master-02
• 192.168.209.132 master-03

以上三臺作為主控並作為GlusterFS集群節點。

添加hosts

[[email protected] gluster]# cat /etc/hosts
192.168.209.130 master-01
192.168.209.131 master-02
192.168.209.132 master-03

安裝GlusterFS

GlusterFS默認配置文件在/etc/glusterfs/glusterd.vol
工作目錄默認在/var/lib/glusterfsd

[[email protected] ~]#yum install -y centos-release-gluster glusterfs glusterfs-server glusterfs-fuse
[[email protected] ~]#systemctl start glusterd
[[email protected] ~]# systemctl status  glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled; vendor preset: disabled)
   Active: active (running) since 日 2019-04-28 13:20:08 CST; 2min 57s ago
     Docs: man:glusterd(8)
 

集群配置

[[email protected] ~]# gluster peer probe master-01
peer probe: success. Probe on localhost not needed   
[[email protected] ~]# gluster peer probe master-02
peer probe: success. 
[[email protected] ~]# gluster peer probe master-03
peer probe: success. 

創建卷並啟動

[[email protected] ~]# gluster volume create gv0 replica 3 master-01:/opt/data/gv0 master-02:/opt/data/gv0 master-03:/opt/data/gv0 force
volume create: gv0: success: please start the volume to access data
[[email protected] ~]# gluster volume  start gv0
volume start: gv0: success
 

• 默認不支持使用/分區作為存儲路徑，我這沒有數據盤，所以使用force強制創建了。
• /opt/data/gv0目錄得先在三臺機器上都創建。

查看節點和卷信息

[[email protected] ~]# gluster peer status
Number of Peers: 2
Hostname: master-02
Uuid: 6f5a823c-4044-4dd9-95bc-129018b2eec0
State: Peer in Cluster (Connected)
Hostname: master-03
Uuid: dfeb6d81-c90b-4628-82f2-e379ed0ceded
State: Peer in Cluster (Connected)
[[email protected] ~]# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 1891c9f8-8acb-48c2-94d9-29e2a625f9a0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: master-01:/opt/data/gv0
Brick2: master-02:/opt/data/gv0
Brick3: master-03:/opt/data/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

查看端口監聽

[[email protected] ~]#  netstat -tunlp|grep glusterd
tcp        0      0 0.0.0.0:24007           0.0.0.0:*               LISTEN      19533/glusterd

測試掛載

想要正常的在kubernetes集群中使用或者掛載glusterfs，集群中的對應節點都需要安裝 glusterfs-fuse

[[email protected] harbor]# yum install -y  centos-release-gluster  glusterfs-fuse
[[email protected] ~]# mount -t glusterfs master-01:/gv0 /mnt
[[email protected] ~]# df -h
...
master-01:/gv0            38G   13G   25G   34% /mnt

常用命令

#刪除卷
gluster volume stop gv0
gluster volume delete gv0
#將機器移出集群
gluster peer detach master-01
#只允許172.28.0.0的網絡訪問glusterfs
gluster volume set gv0 auth.allow 172.28.26.*
gluster volume set gv0 auth.allow 192.168.222.1,192.168.*.*
#加入新的機器並添加到卷裏(由於副本數設置為2,至少要添加2（4、6、8..）臺機器)
gluster peer probe master-02
gluster peer probe master-03
#新加卷
gluster volume add-brick gv0 repl 2 master-02:/data/gluster master-03:/data/gluster force
#刪除卷
gluster volume remove-brick gv0 repl 2 master-02:/opt/data/gv0 master-03:/opt/data/gv0 start
gluster volume remove-brick gv0 repl 2 master-02:/opt/data/gv0 master-03:/opt/data/gv0 status
gluster volume remove-brick gv0 repl 2 master-02:/opt/data/gv0 master-03:/opt/data/gv0 commit
註意：擴展或收縮卷時，也要按照卷的類型，加入或減少的brick個數必須滿足相應的要求。
#當對卷進行了擴展或收縮後，需要對卷的數據進行重新均衡。
gluster volume rebalance mamm-volume start|stop|status
###########################################################
遷移卷---主要完成數據在卷之間的在線遷移
#啟動遷移過程
gluster volume replace-brick gv0 master-02:/opt/data/gv0 master-02:/opt/test start force
#查看遷移狀態
gluster volume replace-brick gv0 master-02:/opt/data/gv0 master-02:/opt/test status
#遷移完成後提交完成
gluster volume replace-brick gv0 master-02:/opt/data/gv0 master-02:/opt/test commit
#機器出現故障,執行強制提交
gluster volume replace-brick gv0 master-02:/opt/data/gv0 master-02:/opt/test commit force
###########################################################
觸發副本自愈
gluster volume heal mamm-volume #只修復有問題的文件
gluster volume heal mamm-volume full #修復所有文件
gluster volume heal mamm-volume info #查看自愈詳情
#####################################################
data-self-heal, metadata-self-heal and entry-self-heal
啟用或禁用文件內容、文件元數據和目錄項的自我修復功能，默認情況下三個全部是“on”。
#將其中的一個設置為off的範例：
gluster volume set gv0 entry-self-heal off

到這一步GlusterFS就部署完成了，接下來就在k8s使用它。

創建Endpoints

# ep 編排
[[email protected] gluster]# cat gluster-ep.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
- addresses:
  - ip: 192.168.209.130
  - ip: 192.168.209.131
  - ip: 192.168.209.132
  ports:
  - port: 1990
    protocol: TCP
---
kind: Service
apiVersion: v1
metadata:
  name: glusterfs-cluster
spec:
  ports:
  - port: 1990
# 開始創建 
[[email protected] gluster]# kubectl  apply -f gluster-ep.yaml 
endpoints/glusterfs-cluster created
service/glusterfs-cluster created
# 查看狀態結果
[[email protected] gluster]# kubectl  get ep,svc|grep glusterfs
endpoints/glusterfs-cluster   192.168.209.130:1990,192.168.209.131:1990,192.168.209.132:1990   2m4s
service/glusterfs-cluster   ClusterIP   10.254.120.33   <none>        1990/TCP       2m4s

創建Pv/Pvc

# 編排
[[email protected] gluster]# cat gluster-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolume   # pv
metadata:
  name: test-pv
  namespace: default
  labels:
    alicloud-pvname: test-pv
spec:     # 定義pv屬性
  capacity:         # 容量
    storage: 2Gi   # 存儲容量
  accessModes:    # 訪問模式,支持ReadWriteOnce、ReadOnlyMany和ReadWriteMany
    - ReadWriteMany  
  glusterfs:
    endpoints: 'glusterfs-cluster'
    path: 'gv0'   
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle  
---
kind: PersistentVolumeClaim  # pvc
apiVersion: v1
metadata:
  name: test-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  selector:
    matchLabels:
      alicloud-pvname: test-pv
# 開始創建
[[email protected] gluster]# kubectl  apply -f gluster-pvc.yaml 
persistentvolume/dt-pv created
persistentvolumeclaim/dt-pvc created
# 查看結果狀態
[[email protected] gluster]# kubectl  get pv,pvc
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
persistentvolume/test-pv   2Gi        RWX            Recycle          Bound    default/test-pvc                           50s
NAME                           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/test-pvc   Bound    test-pv    2Gi        RWX                           50s

創建pod

# 編排
[[email protected] gluster]# cat ng-deploy.yaml 
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    run: nginx01
  name: nginx01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-pod1  
  template:
    metadata:
      labels:
        app: nginx-pod1
    spec:
      containers:
      - name: nginx11
        image: nginx
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /usr/share/nginx/html   # 掛載到容器的路徑
          name: glu
      restartPolicy: Always
      volumes: 
      - name: glu   定義卷
        persistentVolumeClaim:
          claimName: test-pvc    使用pvc
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx1
  name: nginx1
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30003
  selector:
    app: nginx-pod1
  type: NodePort

# 創建
[[email protected] gluster]# kubectl  apply -f ng-deploy.yaml 
deployment.apps/nginx01 created
service/nginx1 created

查看狀態

[[email protected] gluster]# kubectl  get pod,svc|grep nginx
pod/nginx01-7df547bf8c-cxvht       1/1     Running       0          10m
service/nginx1              NodePort    10.254.55.187   <none>        80:30003/TCP   79m

[[email protected] gluster]# kubectl  describe po nginx01-7df547bf8c-cxvht
Name:               nginx01-7df547bf8c-cxvht
......
Volumes:
  glu:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  test-pvc
    ReadOnly:   false
  default-token-tb5bg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tb5bg

測試

在glusterfs存儲卷中創建index.html,並寫入內容

[[email protected] gluster]# vi  /opt/data/gv0/index.html
[[email protected] gluster]# kubectl  exec -ti nginx01-7df547bf8c-cxvht bash
[email protected]:/# ls /usr/share/nginx/html/
index.html
[email protected]:/# cat  /usr/share/nginx/html/index.html 
<h1>hello glusterfs</h1>

瀏覽器訪問測試

現在就算pod 掛了等情況，數據也不會丟的，需要註意的是，上GlusterFS 一定得有人技術兜底，不然出了問題很麻煩的。

Kubernetes使用GlusterFS實現數據持久化