06 雲原生 RBD 塊儲存（轉載）
阿新 • • 發佈：2022-12-01
目錄
雲原生
校驗
校驗
容器
雲原生 RBD 塊儲存
RBD 塊儲存概述
什麼是塊儲存：A block is a sequence of bytes (often 512). Block-based storage interfaces are a mature and common way to store data on media including HDDs, SSDs, CDs, floppy disks, and even tape.
EBSCBSFC塊
- Thin-provisioned （受分配，使用多少分配多少，慢慢擴大）
- Images up to 16 exabytes （單個映象最大16EB）
- Configurable striping（可配置切片）
- In-memory caching （記憶體快取）
- Snapshots（支援快照）
- Copy-on-write cloning（快照克隆）
- Kernel driver support（核心支援）
- KVM/libvirt support（kvm/librirt支援）
- Back-end for cloud solutions（後端支援雲解決方案）
- Incremental backup（增量備份）
- Disaster recovery (multisite asynchronous replication)（災難恢復）
RBD 與容器對接
三種方式
-
volume: 卷的儲存方式，支援多種驅動，FC，EBS，Ceph等 -
PV/PVC:Persistent Volume和Persistent Volume Claim -
StorageClass: 包含靜態+動態兩種- 管理員定義好
provioner - 終端使用者通過
PVC關聯
- 管理員定義好
Ceph 對接回顧
Ceph 和 kubernetes 的對接過程涉及到 pool 的建立， Ceph 認證資訊，配置檔案， CSI
StorageClass 建立等一系列過程。配置過程有一定的難度，如果對 Ceph 不熟悉的同學，對接可能有一定難度，而 Rook 則將這些配置過程簡化，以雲原生的方式實現對接，其預設已經繼承好相關驅動，直接通過 kubernetes 建立 storageclass 即可。
CSI 驅動資訊
- 包含
rbd和cephfs的驅動，csi-cephfsplugin和csi-rbdplugin - 驅動由
provisioner和plugin組成
RBD塊儲存類
RBD 相關的儲存驅動和 provisioner 安裝 rook 時候已經建立好，因此接下來只需要直接對接即可， Rook 提供了兩種對接的方式：
FlexVolumeCSI
其中 Flex 的方式比較老，預設驅動未安裝，需要安裝才可以對接，逐步淘汰，不建議使用，推薦使用 CSI 的對接方式
[root@m1 ceph]# cat csi/rbd/storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-ceph # namespace:cluster
pool: replicapool
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete
# 執行資源清單
[root@m1 ceph]# kubectl apply -f csi/rbd/storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
校驗 pool 安裝情況
- 配置
ceph客戶端認證
[root@m1 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-77bf5b9b7d-9pq6m -- cat /etc/ceph/ceph.conf | tee /etc/ceph/ceph.conf
[global]
mon_host = 10.68.231.222:6789,10.68.163.216:6789,10.68.61.127:6789
[client.admin]
keyring = /etc/ceph/keyring
[root@m1 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-77bf5b9b7d-9pq6m -- cat /etc/ceph/keyring | tee /etc/ceph/keyring
[client.admin]
key = AQDm435jUMuCKhAAXXPqMX08cLyjs/EOvchkzA==
[root@m1 ceph]# ceph -s
cluster:
id: 17a413b5-f140-441a-8b35-feec8ae29521
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 2h)
mgr: a(active, since 6h)
osd: 5 osds: 5 up (since 6h), 5 in (since 6h)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 5.0 GiB used, 245 GiB / 250 GiB avail
pgs: 1 active+clean
- 校驗
pool
[root@m1 ceph]# ceph osd lspools
1 device_health_metrics
2 replicapool
[root@m1 ceph]# ceph osd pool get replicapool size
size: 3
校驗 StorageClass
[root@m1 ceph]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 2m56s
容器 PVC 呼叫儲存
建立好 storageclass 之後，我們就可以通過 PVC 向 storageclass 申請容量空間了， PVC 會自動和 storageclass 完成儲存容量的建立過程，包括自動建立 PV ， PV 與後端儲存自動完成 RBD 塊儲存的建立，整個過程不需要我們關心，均通過 storageclass 和驅動自動完成，我們只需要關注使用即可，如下是一個 wordpress 部落格應用連線 MySQL 資料庫的一個雲原生應用的範例
wordpress 資源清單
[root@m1 ceph]# cd ../
[root@m1 kubernetes]# cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
tier: frontend
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.6.1-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
value: changeme
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
MySQL 資源清單
[root@m1 kubernetes]# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
tier: mysql
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
建立 wordpress 和 mysql
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-7b989dbf57-cvqtk 1/1 Running 0 6m25s
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 6m20s
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 29s
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 34s
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 31s
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 35s
PVC 呼叫邏輯
PVC 會完成一系列的出對接過程，包含有什麼動作。 PVC —> storageclass 申請容量 —> 建立 PV ——> 向 Ceph 申請 RBD 塊，完成和 Ceph 對接
檢視PVC建立情況
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 8m14s
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 8m19s
- 檢視詳情
[root@m1 kubernetes]# kubectl get pvc mysql-pv-claim -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
......
name: mysql-pv-claim
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: rook-ceph-block
volumeMode: Filesystem
volumeName: pvc-2bcd6af2-31e3-40b8-add9-476b167399be
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
PVC 會自動建立 PV
-
PV資訊
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 10m
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 10m
- 詳細的
PV資訊
[root@m1 kubernetes]# kubectl get pv pvc-2bcd6af2-31e3-40b8-add9-476b167399be -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
......
name: pvc-2bcd6af2-31e3-40b8-add9-476b167399be
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: mysql-pv-claim
namespace: default
resourceVersion: "502998"
uid: 2bcd6af2-31e3-40b8-add9-476b167399be
csi:
controllerExpandSecretRef:
name: rook-csi-rbd-provisioner
namespace: rook-ceph
driver: rook-ceph.rbd.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
name: rook-csi-rbd-node
namespace: rook-ceph
volumeAttributes:
clusterID: rook-ceph
imageFeatures: layering
imageFormat: "2"
imageName: csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4 # 關聯 rbd 資訊
journalPool: replicapool
pool: replicapool
radosNamespace: ""
storage.kubernetes.io/csiProvisionerIdentity: 1669260272979-8081-rook-ceph.rbd.csi.ceph.com
volumeHandle: 0001-0009-rook-ceph-0000000000000002-10342079-6beb-11ed-b5c4-6efb82c232c4
persistentVolumeReclaimPolicy: Delete
storageClassName: rook-ceph-block
volumeMode: Filesystem
status:
phase: Bound
PV 會完成和 Ceph 的對接，自動建立 RBD 塊儲存空間，期間由 plugin 驅動完成建立
-
rbd資訊
[root@m1 kubernetes]# rbd -p replicapool ls
csi-vol-0d43155e-6beb-11ed-b5c4-6efb82c232c4
csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4
-
mysql使用rbd塊資訊
[root@m1 kubernetes]# rbd -p replicapool info csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4
rbd image 'csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4':
size 20 GiB in 5120 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 12fae9f8d2613
block_name_prefix: rbd_data.12fae9f8d2613
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Nov 24 19:28:05 2022
access_timestamp: Thu Nov 24 19:28:05 2022
modify_timestamp: Thu Nov 24 19:28:05 2022
Wordpress 功能驗證
wordpress 和 mysql 實際儲存空間存放在 Ceph RBD 塊中，通過 nodePort 的登入 MySQL ，驗證 wordpress 的安裝情況，首先需要核對 pods 部署是否正常，確保均 Running
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
wordpress-7b989dbf57-cvqtk 1/1 Running 0 17m
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 17m
wordpress 通過 service 暴露服務，預設型別為 LoadBalancer ， LoadBalancer 依賴於 NodePort ，因此可以直接通過 NodePort 的形式訪問，通過任意一個節點的 NodePort 即可訪問到 wordpress 的應用。
[root@m1 kubernetes]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 2d3h
wordpress LoadBalancer 10.68.235.228 <pending> 80:36063/TCP 19m
wordpress-mysql ClusterIP None <none> 3306/TCP 19m
- 檢視
rbd對映情況
[root@m1 kubernetes]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
wordpress-7b989dbf57-cvqtk 1/1 Running 0 41m 172.20.4.27 192.168.100.136 <none> <none>
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 41m 172.20.4.28 192.168.100.136 <none> <none>
[root@m1 kubernetes]# ssh n3
Last login: Thu Nov 24 20:07:31 2022 from m1
[root@n3 ~]# rbd showmapped
2022-11-24 20:09:07.150816 7f5daf0d6d80 -1 did not load config file, using default settings.
id pool image snap device
0 replicapool csi-vol-0d43155e-6beb-11ed-b5c4-6efb82c232c4 - /dev/rbd0
1 replicapool csi-vol-10342079-6beb-11ed-b5c4-6efb82c232c4 - /dev/rbd1
[root@n3 ~]# exit
儲存持久化模板
PVC 使用模式適用於單個 pods 容器，如多個 pods 都需要有各自的儲存如何實現，需要藉助於 StatefulSet 的 volumeClaimTemplates 功能，實現每個 pods 均有各自的儲存。
[root@m1 kubernetes]# cat << EOF > sts-test.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "rook-ceph-block"
resources:
requests:
storage: 10Gi
EOF
- 部署服務檢視
pv pvc情況
[root@m1 kubernetes]# kubectl apply -f sts-test.yaml
statefulset.apps/web created
[root@m1 kubernetes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 103s
web-1 1/1 Running 0 65s
web-2 1/1 Running 0 51s
wordpress-7b989dbf57-cvqtk 1/1 Running 0 14h
wordpress-mysql-6965fc8cc8-mt2p9 1/1 Running 0 14h
[root@m1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO rook-ceph-block 14h
wp-pv-claim Bound pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO rook-ceph-block 14h
www-web-0 Bound pvc-9d32f5ff-174a-461f-9f3c-e0f0f5e60cc8 10Gi RWO rook-ceph-block 83s
www-web-1 Bound pvc-af0e2916-ec93-4f55-871f-0d008c0394e0 10Gi RWO rook-ceph-block 45s
www-web-2 Bound pvc-174fb859-91ad-49e2-8e44-d7ee64645e7e 10Gi RWO rook-ceph-block 31s
[root@m1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-174fb859-91ad-49e2-8e44-d7ee64645e7e 10Gi RWO Delete Bound default/www-web-2 rook-ceph-block 32s
pvc-2bcd6af2-31e3-40b8-add9-476b167399be 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 14h
pvc-2fadab42-abd6-49e0-b53f-b587f3b8f95d 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 14h
pvc-9d32f5ff-174a-461f-9f3c-e0f0f5e60cc8 10Gi RWO Delete Bound default/www-web-0 rook-ceph-block 84s
pvc-af0e2916-ec93-4f55-871f-0d008c0394e0 10Gi RWO Delete Bound default/www-web-1 rook-ceph-block 46s