7. Ceph 高階篇 - RBD塊裝置回收站、快照、克隆

阿新 • • 發佈：2022-03-26

文章轉載自：https://mp.weixin.qq.com/s?__biz=MzI1MDgwNzQ1MQ==&mid=2247485298&idx=1&sn=b83fda07370632136935edfa2fc2ac4f&chksm=e9fdd286de8a5b9086595cffd5775495d8a75629934400e842221e57e4d514462ba846711ec8&scene=178&cur_album_id=1600845417376776197#rd

RDB 回收站

官網：https://docs.ceph.com/en/latest/rbd/rados-rbd-cmds/

資料可以先儲存在回收站裡面，配置一個策略週期，當你不需要的時候，再去刪除，當你需要時，可從回收站裡面恢復出來，這就是簡單的回收站實現機制；

建立映象

[root@ceph-node01 ~]# rbd create ceph-demo/ceph-trash.img --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd info ceph-demo/ceph-trash.img
rbd image 'ceph-trash.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 392c7a4143a46
  block_name_prefix: rbd_data.392c7a4143a46
  format: 2
  features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
  op_features:
  flags:
  create_timestamp: Fri Oct 16 23:11:09 2020
  access_timestamp: Fri Oct 16 23:11:09 2020
  modify_timestamp: Fri Oct 16 23:11:09 2020
[root@ceph-node01 ~]#

未設定回機制前刪除

[root@ceph-node01 ~]# rbd remove -p ceph-demo ceph-trash.img
Removing image: 100% complete...done.
[root@ceph-node01 ~]#

驗證直接刪除

[root@ceph-node01 ~]# rbd -p ceph-demo ls
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd -p ceph-demo trash ls
[root@ceph-node01 ~]#

直接刪除的話，直接就沒有了，在對應的pool池和回收站裡面都沒有；

驗證回收站刪除

[root@ceph-node01 ~]# rbd create ceph-demo/ceph-trash.img --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd trash move ceph-demo/ceph-trash.img --expires-at 20201020
[root@ceph-node01 ~]# rbd -p ceph-demo ls
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd -p ceph-demo trash ls
3931bebcf0579 ceph-trash.img
[root@ceph-node01 ~]#

注意在回收站的時間，是根據你上面設定的回收週期而定的，到期後，會自動清理；

找回刪除塊裝置

[root@ceph-node01 ~]# rbd trash restore -p ceph-demo 3931bebcf0579
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]#

這樣就可以防止誤刪除；

RBD 快照

官網：https://docs.ceph.com/en/latest/rbd/rbd-snapshot/

A snapshot is a read-only logical copy of an image at a particular point in time: a checkpoint. One of the advanced features of Ceph block devices is that you can create snapshots of images to retain（保留、持有、保有） point-in-time state history. Ceph also supports snapshot layering（快照分層）, which allows you to clone images (e.g., a VM image) quickly and easily. Ceph block device snapshots are managed using the rbd command and multiple higher level interfaces, including QEMU, libvirt, OpenStack and CloudStack.

Important

To use RBD snapshots, you must have a running Ceph cluster.

Note

Because RBD does not know about any filesystem within an image (volume), snapshots are not crash-consistent unless they are coordinated within the mounting (attaching) operating system. We therefore recommend that you pause or stop I/O before taking a snapshot. If the volume contains a filesystem, it must be in an internally consistent state before taking a snapshot. Snapshots taken at inconsistent points may need a fsck pass before subsequent mounting. To stop I/O you can use fsfreeze command. See fsfreeze(8) man page for more details. For virtual machines, qemu-guest-agent can be used to automatically freeze file systems when creating a snapshot.

如果在做快照時映像仍在進行 I/O 操作，快照可能就獲取不到該映像準確的或最新的資料，並且該快照可能不得不被克隆到一個新的可掛載的映像中。所以，我們建議在做快照前先停止 I/O 操作。如果映像內包含檔案系統，在做快照前請確保檔案系統處於一致的狀態。要停止 I/O 操作可以使用 fsfreeze 命令。詳情可參考 fsfreeze(8) 手冊頁。對於虛擬機器，qemu-guest-agent 被用來在做快照時自動凍結檔案系統。

其實快照就是一種備份機制；

建立一塊裝置

[root@ceph-node01 ~]# rbd create ceph-demo/rbd-test.img --image-feature layering --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img
[root@ceph-node01 ~]#

檢視塊設定資訊

[root@ceph-node01 ~]# rbd info -p ceph-demo rbd-test.img
rbd image 'rbd-test.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 3936fc8add13c
  block_name_prefix: rbd_data.3936fc8add13c
  format: 2
  features: layering
  op_features:
  flags:
  create_timestamp: Fri Oct 16 23:52:31 2020
  access_timestamp: Fri Oct 16 23:52:31 2020
  modify_timestamp: Fri Oct 16 23:52:31 2020
[root@ceph-node01 ~]#

掛載

[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test.img
/dev/rbd1
[root@ceph-node01 ~]# mkfs.ext4 /dev/rbd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: 完成
檔案系統標籤=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
第一個資料塊=0
Maximum filesystem blocks=2151677952
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
  32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: 完成
正在寫入inode表: 完成
Creating journal (32768 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@ceph-node01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 50G 0 disk
├─vda1 252:1 0 500M 0 part /boot
└─vda2 252:2 0 49.5G 0 part
  ├─centos-root 253:0 0 44.5G 0 lvm /
  └─centos-swap 253:1 0 5G 0 lvm [SWAP]
vdb 252:16 0 100G 0 disk
└─ceph--48dc8bff--6f69--41e1--9bfa--b188edc8f419-osd--block--909ff4ad--d05b--476f--b7e3--56d9e77004b4
                                                                 253:2 0 100G 0 lvm
vdc 252:32 0 100G 0 disk
└─ceph--6f6493a0--ecaa--45de--8dcb--04f5b6f8e957-osd--block--ded9a04f--71b5--4088--87c3--0e71604c7d75
                                                                 253:3 0 100G 0 lvm
rbd0 251:0 0 20G 0 disk /mnt/rbd-demo
rbd1 251:16 0 10G 0 disk
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# mount /dev/rbd1 /media/
[root@ceph-node01 ~]# cd /media/
[root@ceph-node01 media]# ls
lost+found
[root@ceph-node01 media]# echo `date` > file.log
[root@ceph-node01 media]# ls
file.log lost+found
[root@ceph-node01 media]# sync
[root@ceph-node01 media]#

使用sync進行刷盤，把當前的狀態儲存下來，下面製作快照使用。

製作快照

[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201011
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
     4 snap_20201011 10 GiB Sat Oct 17 00:02:00 2020
[root@ceph-node01 media]#

這裡可以看到快照的名稱及製作快照的時間；

快照恢復

# 1. 模擬檔案刪除
[root@ceph-node01 media]# rm -rf file.log
[root@ceph-node01 media]#

# 2. 檢視要恢復映象的快照名稱
[root@ceph-node01 ~]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
     4 snap_20201011 10 GiB Sat Oct 17 00:02:00 2020

# 3. 快照恢復
[root@ceph-node01 ~]# rbd snap rollback ceph-demo/rbd-test.img@snap_20201011
Rolling back to snapshot: 100% complete...done.

# 4. 注意這裡需要重新掛載
[root@ceph-node01 ~]# umount /media/
[root@ceph-node01 ~]# mount /dev/rbd1 /media/
[root@ceph-node01 ~]# cd /media/

# 5. 檢視恢復檔案資訊
[root@ceph-node01 media]# ls
file.log lost+found
[root@ceph-node01 media]# cat file.log
2020年 10月 16日 星期五 23:58:43 EDT
[root@ceph-node01 media]#

快照的一個目的，就是對映象做備份，比如，當你做一些高風險的操作時，把映象當時的狀態先保留下來，當資料異常時，可以把資料進行一次回滾；

快照刪除

# 1. 刪除單個快照
[root@ceph-node01 media]# rbd snap remove ceph-demo/rbd-test.img@snap_20201011
Removing snap: 100% complete...done.
[root@ceph-node01 media]#

# 2. 驗證刪除所有快照
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201011
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201012
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201013
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
     6 snap_20201011 10 GiB Sat Oct 17 00:17:58 2020
     7 snap_20201012 10 GiB Sat Oct 17 00:18:00 2020
     8 snap_20201013 10 GiB Sat Oct 17 00:18:04 2020
[root@ceph-node01 media]#
[root@ceph-node01 media]# rbd snap purge ceph-demo/rbd-test.img
Removing all snapshots: 100% complete...done.
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
[root@ceph-node01 media]#

映象克隆機制

Ceph supports the ability to create many copy-on-write (COW) clones of a block device snapshot. Snapshot layering enables Ceph block device clients to create images very quickly. For example, you might create a block device image with a Linux VM written to it; then, snapshot the image, protect the snapshot, and create as many copy-on-write clones as you like. A snapshot is read-only, so cloning a snapshot simplifies semantics–making（克隆快照的語義） it possible to create clones rapidly.

The terms “parent” and “child” refer to a Ceph block device snapshot (parent), and the corresponding image cloned from the snapshot (child). These terms are important for the command line usage below.

這裡的術語“父”和“子”指的是一個 Ceph 塊裝置快照（父），和從此快照克隆出來的對應映像（子）。這些術語對下列的命令列用法來說很重要。

Each cloned image (child) stores a reference to its parent image, which enables the cloned image to open the parent snapshot and read it.

每一個克隆出來的映像（子）都儲存著對父映像的引用，這使得克隆出來的映像可以開啟父映像並讀取它。

A COW clone of a snapshot behaves exactly like any other Ceph block device image. You can read to, write from, clone, and resize cloned images. There are no special restrictions with cloned images. However, the copy-on-write clone of a snapshot depends on the snapshot, so you MUST protect the snapshot before you clone it. The following diagram depicts the process.

一個快照的 COW 克隆的行為更像其它任何 Ceph 塊裝置映像的一樣。克隆出的映像沒有特別的限制，你可以讀出、寫入、克隆、調整克隆映像的大小。然而快照的寫時複製克隆引用了快照，所以你克隆快照前必須保護它。下圖描述了此過程。

Ceph only supports cloning of RBD format 2 images (i.e., created with rbd create --image-format 2). The kernel client supports cloned images beginning with the 3.10 release.

Ceph 僅支援克隆 format 2 的映像（即用 rbd create --image-format 2 建立的）。核心客戶端從 3.10 版開始支援克隆的映像。

GETTING STARTED WITH LAYERING

Ceph block device layering is a simple process. You must have an image. You must create a snapshot of the image. You must protect the snapshot. Once you have performed these steps, you can begin cloning the snapshot.

Ceph 塊裝置的分層是個簡單的過程。你必須有個映像、必須為它建立快照、並且必須保護快照，執行過這些步驟後，你才能克隆快照。

The cloned image has a reference （引用）to the parent snapshot, and includes the pool ID, image ID and snapshot ID. The inclusion （包含）of the pool ID means that you may clone snapshots from one pool to images in another pool.

1.Image Template（映象模板）: A common use case for block device layering is to create a master image and a snapshot that serves as a template for clones. For example, a user may create an image for a Linux distribution（Linux 發行版的映像） (e.g., Ubuntu 12.04), and create a snapshot for it. Periodically（定期的）, the user may update the image and create a new snapshot (e.g., sudo apt-get update, sudo apt-get upgrade, sudo apt-get dist-upgrade followed by rbd snap create). As the image matures（隨著映像的成熟）, the user can clone any one of the snapshots.

2.Extended Template（擴充套件模板）: A more advanced use case includes extending a template image that provides more information than a base image. For example, a user may clone an image (e.g., a VM template) and install other software (e.g., a database, a content management system, an analytics system, etc.) and then snapshot the extended image, which itself may be updated just like the base image.

3.Template Pool（模板池）: One way to use block device layering is to create a pool that contains master images that act as templates, and snapshots of those templates. You may then extend read-only privileges to users so that they may clone the snapshots without the ability to write or execute within the pool.

4.Image Migration/Recovery（映象遷移和恢復）: One way to use block device layering is to migrate or recover data from one pool into another pool.

映象克隆用例

# 1. 檢視 rbd 塊裝置
[root@ceph-node01 ~]# rbd ls -p ceph-demo rbd-test.img
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img

# 2. 檢視快照
[root@ceph-node01 ~]# rbd snap ls -p ceph-demo rbd-test.img

# 3. 建立快照
[root@ceph-node01 ~]# rbd snap create ceph-demo/rbd-test.img@template
[root@ceph-node01 ~]# rbd snap ls -p ceph-demo rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
    12 template 10 GiB Sat Oct 17 03:23:22 2020

# 4. 把快照保護起來，以免誤刪
[root@ceph-node01 ~]# rbd snap protect ceph-demo/rbd-test.img@template

# 5. 刪除測試，發現根據刪除不了
[root@ceph-node01 ~]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 2020-10-17 03:23:59.647 7fb6e3844c80 -1 librbd::Operations: snapshot is protected
0% complete...failed.
rbd: snapshot 'template' is protected from removal.

# 5. 基於模板進行 clone 映象
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm1-clone.img
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm2-clone.img
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm3-clone.img
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img
vm1-clone.img
vm2-clone.img
vm3-clone.img

# 6. 檢視映象資訊，可以看到繼承資訊；
[root@ceph-node01 ~]# rbd -p ceph-demo info vm3-clone.img
rbd image 'vm3-clone.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 394a48fcc1688
  block_name_prefix: rbd_data.394a48fcc1688
  format: 2
  features: layering
  op_features:
  flags:
  create_timestamp: Sat Oct 17 03:24:59 2020
  access_timestamp: Sat Oct 17 03:24:59 2020
  modify_timestamp: Sat Oct 17 03:24:59 2020
  parent: ceph-demo/rbd-test.img@template
  overlap: 10 GiB
[root@ceph-node01 ~]#

掛載使用

[root@ceph-node01 ~]# rbd device map ceph-demo/vm1-clone.img
/dev/rbd2
[root@ceph-node01 ~]# rbd device map -p ceph-demo vm2-clone.img
/dev/rbd4
[root@ceph-node01 ~]# mkdir abc
[root@ceph-node01 ~]# mount /dev/rbd4 /root/abc
[root@ceph-node01 ~]# cd /root/abc/
[root@ceph-node01 abc]# ls
file.log lost+found
[root@ceph-node01 abc]#

父子映象剝離

Cloned images retain（保留） a reference to the parent snapshot. When you remove the reference from the child clone to the parent snapshot, you effectively “flatten” the image by copying the information from the snapshot to the clone. The time it takes to flatten a clone increases with the size of the snapshot. To delete a snapshot, you must flatten the child images first.

克隆出來的映像仍保留了對父快照的引用，要從子克隆刪除這些到父快照的引用，你可以把快照的資訊複製給子克隆，也就是“拍平”它，拍平克隆映像的時間隨快照尺寸增大而增加，要刪除快照，必須先拍平子映像。

Since a flattened image contains all the information from the snapshot, a flattened image will take up more storage space than a layered clone。

因為拍平的映像包含了快照的所有資訊，所以拍平的映像佔用的儲存空間會比分層克隆要大。

檢視父映象 clone 了多少個子映象

[root@ceph-node01 abc]# rbd children ceph-demo/rbd-test.img@template
ceph-demo/vm1-clone.img
ceph-demo/vm2-clone.img
ceph-demo/vm3-clone.img
[root@ceph-node01 abc]#

解除父子關係

[root@ceph-node01 abc]# rbd flatten ceph-demo/vm1-clone.img
Image flatten: 100% complete...done.
[root@ceph-node01 abc]# rbd flatten ceph-demo/vm2-clone.img
Image flatten: 100% complete...done.
[root@ceph-node01 abc]# rbd info ceph-demo/vm1-clone.img
rbd image 'vm1-clone.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 3948c22f5e5f
  block_name_prefix: rbd_data.3948c22f5e5f
  format: 2
  features: layering
  op_features:
  flags:
  create_timestamp: Sat Oct 17 03:24:52 2020
  access_timestamp: Sat Oct 17 03:24:52 2020
  modify_timestamp: Sat Oct 17 03:24:52 2020
[root@ceph-node01 abc]#

刪除父映象

# 1. 刪除映象模板，無法刪除，原因是還有一個clone物件
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 0% complete...failed.
rbd: snapshot 'template'2020-10-17 03:51:31.135 7f618c8d8c80 -1 librbd::Operations: snapshot is protected
 is protected from removal.

# 2. 解除父子關係
[root@ceph-node01 abc]# rbd flatten ceph-demo/vm3-clone.img
Image flatten: 100% complete...done.
# 3. 檢視父子映象
[root@ceph-node01 abc]# rbd children ceph-demo/rbd-test.img@template

# 4. 刪除父映象
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 2020-10-17 03:52:21.409 7f3141b96c80 -1 librbd::Operations: snapshot is protected0% complete...failed.

rbd: snapshot 'template' is protected from removal.

# 5. 取消保護機制
[root@ceph-node01 abc]# rbd snap unprotect ceph-demo/rbd-test.img@template

# 6. 刪除原來的映象模板（父映象）
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 100% complete...done.
[root@ceph-node01 abc]#

原來的父子映象關係解除後，原來的子映象與父映象就沒有任何關係了，映象依然可以掛載對映使用。

[root@ceph-node01 abc]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
[root@ceph-node01 abc]#

RBD 備份與恢復

當叢集功能不可用時，哪麼塊裝置還能正常使用嗎，很顯然是不可以的，但在不可抗因素下，很有可能造成整個叢集掛掉，哪有什麼辦法呢，我們可以使用離線的備份與恢復機制來實現，可以把映象匯入到磁帶庫或者別外的Ceph 叢集；

# 1. 建立一個快照
[root@ceph-node01 abc]# rbd snap create ceph-demo/rbd-test.img@snap-demo

# 2. 檢視快照
[root@ceph-node01 abc]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
    14 snap-demo 10 GiB Sat Oct 17 04:09:23 2020
[root@ceph-node01 abc]#

備份（匯出）

[root@ceph-node01 abc]# rbd export ceph-demo/rbd-test.img@snap-demo /root/rbd-test.img
Exporting image: 100% complete...done.
[root@ceph-node01 abc]# ls /root/rbd-test.img -lrth
-rw-r--r-- 1 root root 10G 10月 17 04:13 /root/rbd-test.img
[root@ceph-node01 abc]#

恢復（匯入）

[root@ceph-node01 abc]# rbd import /root/rbd-test.img ceph-demo/rbd-test-new.img
Importing image: 100% complete...done.
[root@ceph-node01 abc]#

使用

# 1. 匯入後，發現有些feature 功能加了進來，我們需要關掉
[root@ceph-node01 abc]# rbd device map ceph-demo/rbd-test-new.img
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address

# 2. 檢視info
[root@ceph-node01 abc]# rbd info ceph-demo/rbd-test-new.img
rbd image 'rbd-test-new.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 39594d499ff4f
  block_name_prefix: rbd_data.39594d499ff4f
  format: 2
  features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
  op_features:
  flags:
  create_timestamp: Sat Oct 17 04:18:02 2020
  access_timestamp: Sat Oct 17 04:18:02 2020
  modify_timestamp: Sat Oct 17 04:18:02 2020

# 3. 關掉 feature
[root@ceph-node01 abc]# rbd feature disable ceph-demo/rbd-test-new.img exclusive-lock object-map fast-diff deep-flatten
[root@ceph-node01 abc]# rbd device map ceph-demo/rbd-test-new.img
/dev/rbd5
[root@ceph-node01 abc]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
5 ceph-demo rbd-test-new.img - /dev/rbd5

# 4. 掛載檢視
[root@ceph-node01 abc]# mount /dev/rbd5 /data/abc
[root@ceph-node01 abc]# cd /data/abc/
[root@ceph-node01 abc]# cat file.log
2020年 10月 16日 星期五 23:58:43 EDT
[root@ceph-node01 abc]#

增量備份演示

# 1. 建立塊裝置
[root@ceph-node01 ~]# rbd create ceph-demo/rbd-test-k8s.img --image-feature layering --size 10G

# 2. 對映塊裝置使用
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
/dev/rbd5

# 3. 格式化
[root@ceph-node01 ~]# mkfs.ext4 /dev/rbd5
mke2fs 1.42.9 (28-Dec-2013)
。。。
Creating journal (32768 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

# 4. 掛載
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
lost+found
[root@ceph-node01 abe]# echo `date` >> a
[root@ceph-node01 abe]# echo `date` >> aa
[root@ceph-node01 abe]# sync
[root@ceph-node01 abe]#

# 5. 建立快照
[root@ceph-node01 abe]# rbd snap create ceph-demo/rbd-test-k8s.img@v1

# 6. 備份
[root@ceph-node01 abe]# rbd export ceph-demo/rbd-test-k8s.img@v1 /root/rbd-test-k8s.img
Exporting image: 100% complete...done.

# 7. 增量寫入
[root@ceph-node01 abe]# ls
a aa lost+found
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>aa
[root@ceph-node01 abe]# echo `date` >>aa
[root@ceph-node01 abe]# echo `date` >>bb
[root@ceph-node01 abe]# sync

# 8. 再次建立快照
[root@ceph-node01 abe]# rbd snap create ceph-demo/rbd-test-k8s.img@v2

# 9. 匯出備份
[root@ceph-node01 abe]# rbd export-diff ceph-demo/rbd-test-k8s.img@v2 /root/rbd-test-k8s_v2.img
Exporting image: 100% complete...done.
[root@ceph-node01 abe]#

# 10. 刪除所有快照
[root@ceph-node01 ~]# rbd snap purge ceph-demo/rbd-test-k8s.img
Removing all snapshots: 100% complete...done.
[root@ceph-node01 ~]#

# 11. 取消裝置對映
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-k8s.img

# 12. 刪除映象
[root@ceph-node01 ~]# rbd rm ceph-demo/rbd-test-k8s.img
Removing image: 100% complete...done.
[root@ceph-node01 ~]#

恢復

# 1. 備份恢復原始映象
[root@ceph-node01 ~]# rbd import /root/rbd-test-k8s.img ceph-demo/rbd-test-k8s.img
Importing image: 100% complete...done.
[root@ceph-node01 ~]#

# 2. 對映原始映象
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@ceph-node01 ~]# rbd info ceph-demo/rbd-test-k8s.img
rbd image 'rbd-test-k8s.img':
  size 10 GiB in 2560 objects
  order 22 (4 MiB objects)
  snapshot_count: 0
  id: 3988541adbc3b
  block_name_prefix: rbd_data.3988541adbc3b
  format: 2
  features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
  op_features:
  flags:
  create_timestamp: Sat Oct 17 05:43:24 2020
  access_timestamp: Sat Oct 17 05:43:24 2020
  modify_timestamp: Sat Oct 17 05:43:24 2020
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# rbd feature disable ceph-demo/rbd-test-k8s.img exclusive-lock object-map fast-diff deep-flatten
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
/dev/rbd5

# 3. 掛載檢視
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
a aa lost+found
[root@ceph-node01 abe]# cat a
2020年 10月 17日 星期六 05:24:57 EDT
[root@ceph-node01 abe]# cat aa
2020年 10月 17日 星期六 05:25:00 EDT

# 4. 匯入增量備份
[root@ceph-node01 abe]# rbd import-diff /root/rbd-test-k8s_v2.img ceph-demo/rbd-test-k8s.img
Importing image diff: 100% complete...done.
[root@ceph-node01 abe]#

# 5. 重新掛載並驗證
[root@ceph-node01 ~]# umount /data/abe/
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
a aa bb lost+found
[root@ceph-node01 abe]# cat a
2020年 10月 17日 星期六 05:24:57 EDT
2020年 10月 17日 星期六 05:27:04 EDT
2020年 10月 17日 星期六 05:27:05 EDT
2020年 10月 17日 星期六 05:27:05 EDT
2020年 10月 17日 星期六 05:27:06 EDT
[root@ceph-node01 abe]# cat aa
2020年 10月 17日 星期六 05:25:00 EDT
2020年 10月 17日 星期六 05:27:12 EDT
2020年 10月 17日 星期六 05:27:13 EDT
[root@ceph-node01 abe]# cat bb
2020年 10月 17日 星期六 05:27:17 EDT
[root@ceph-node01 abe]#

清理主機對映

[root@ceph-node01 ~]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
5 ceph-demo rbd-test-new.img - /dev/rbd5
6 ceph-demo rbd-test-new2.img - /dev/rbd6
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-new2.img
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-new.img

總結

建立儲存池

# 1. 建立儲存池
[root@ceph-node01 ~]# ceph osd pool create kube 64 64
pool 'kube' created

# 2. 檢視
[root@ceph-node01 ~]# ceph osd pool ls
。。。。
kube

# 3. 設定儲存池型別為rbd
[root@ceph-node01 ~]# ceph osd pool application enable kube rbd
enabled application 'rbd' on pool 'kube'

# 4. 初始化
[root@ceph-node01 ~]# rbd pool init kube
[root@ceph-node01 ~]#

建立rbd塊裝置

[root@ceph-node01 ~]# rbd create -p kube k8s01 --size 2G
[root@ceph-node01 ~]# rbd create --pool kube --image k8s02 --size 2G
[root@ceph-node01 ~]# rbd create kube/k8s03 --size 2G

檢視rbd資訊

[root@ceph-node01 ~]# rbd -p kube ls
[root@ceph-node01 ~]# rbd -p kube ls -l
[root@ceph-node01 ~]# rbd info kube/k8s03
[root@ceph-node01 ~]# rbd -p kube ls -l --format json --pretty-format

rbd 特性

layering：磁碟分層技術；exclusive-lock：排它鎖，單獨讀寫操作；object-map：物件點陣圖功能；fast-diff：快照對比使用的；deep-flatten：多層快照之間分離時使用的一種特性；

[root@ceph-node01 ~]# rbd feature disable kube/k8s03 object-map fast-diff deep-flatten

客戶端配置(注意yum源)

[root@ceph-node01 ~]# yum -y install ceph-common

客戶端需要/etc/ceph/ceph.conf和金鑰環檔案檔案；

[root@ceph-node01 ceph-deploy]# ceph --user kube -s

客戶端使用rbd map，然後可以格式化並且mount掛載即可；

[root@ceph-node01 ceph-deploy]# rbd map kube/k8s01
/dev/rbd6
[root@ceph-node01 ceph-deploy]# rbd --user kube map kube/k8s02
/dev/rbd7
[root@ceph-node01 ceph-deploy]# lsblk
 ...
rbd6 251:96 0 2G 0 disk
rbd7 251:112 0 2G 0 disk
[root@ceph-node01 ceph-deploy]#

檢視對映、刪除對映資訊

# 1. 檢視對映
[root@ceph-node01 ceph-deploy]# rbd showmapped
id pool namespace image snap device
....
6 kube k8s01 - /dev/rbd6
7 kube k8s02 - /dev/rbd7

# 2. 在叢集上面檢視哪些被掛載了，注意LOCK
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 2 GiB 2
k8s03 2 GiB 2
[root@ceph-node01 ceph-deploy]#

# 3. 刪除對映關係
[root@ceph-node01 ceph-deploy]# rbd --user kube unmap kube/k8s02
[root@ceph-node01 ceph-deploy]# rbd showmapped
id pool namespace image snap device
。。。
6 kube k8s01 - /dev/rbd6
[root@ceph-node01 ceph-deploy]#

調整映象空間容量

[root@ceph-node01 ceph-deploy]# rbd resize -s 5G kube/k8s02
Resizing image: 100% complete...done.
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
k8s03 2 GiB 2
[root@ceph-node01 ceph-deploy]#

刪除映象

[root@ceph-node01 ceph-deploy]# rbd rm kube/k8s03
Removing image: 100% complete...done.
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]#

刪除到回收站並恢復

[root@ceph-node01 ceph-deploy]# rbd trash move kube/k8s01
[root@ceph-node01 ceph-deploy]# rbd trash list kube
3edbd828dee7c k8s01
[root@ceph-node01 ceph-deploy]# rbd -p kube ls -l
NAME SIZE PARENT FMT PROT LOCK
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]# rbd trash restore -p kube 3edbd828dee7c
[root@ceph-node01 ceph-deploy]# rbd -p kube ls -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]#

快照總結

Ceph 服務端建立快照：rbd snap create kube/k8s01@k8s01snap01

Ceph 服務端檢視快照：rbd snap list kube/k8s01

要想恢復快照，需要先把磁碟掛載刪除掉，繫結著的映象是不允許進行回滾的，並且需要把對映也需要刪除掉；

Ceph 客戶端：先 umount， 然後rbd unmap /dev/rbd0；

Ceph 服務端回滾快照：rbd snap rollback kube/k8s01@k8s01snap01

Ceph 客戶端再mount，檢視資料，可以從快照中恢復；

Ceph 服務端刪除快照：rbd snap rm kube/k8s01@k8s01snap01

Ceph 服務端限制快照數量：rbd snap limit set kube/k8s01 --limit 10

Ceph 服務端清除快照限制：rbd snap limit clear kube/k8s01

克隆（多重快照）總結

對原始的磁碟映象資料做第一次快照，然後對快照置於保護模式下，此時即使你修改了原始磁碟映象中的資料，你也無法改變第一次快照的磁碟資料，因為他被設定為保護模式下了，接下來，我們基於第一次的快照，再進行一次快照，二次快照，這也就是克隆技術；

映象克隆的話，可以跨儲存池進行；

Ceph 服務端建立一個映象：略

Ceph 服務端建立一個快照：rbd snap create kube/k8s01@clone01

Ceph 服務端檢視快照：rbd snap ls kube/k8s01

Ceph 服務端設定快照為保護模式： rbd snap protect kube/k8s01@clone01

Ceph 服務端基於保持的快照進行克隆映象：rbd clone kube/k8s01@clone01 kube/cloneimg01（注意可以跨儲存池）

Ceph 服務端檢視映象：rbd -p kube ls -l

Ceph 客戶端直接對映：rbd --user kube map kube/cloneimg01

Ceph 客戶端直接掛載：略；

Ceph 服務端檢視快照克隆了多少映象：rbd children kube/k8s01@clone01

Ceph 服務端刪除克隆的映象：rbd rm kube/cloneimg02

如果此時，我們想把原資料kube/k8s01刪除，此時應該如何做？刪除後，剛才克隆的兩個映象就無根了，因為兩個克隆映象的快照無根了，這個時候，就需要做展平；此時只需要指定把誰flatten（展平）操作即可，它的意思就是把引用別人的，都複製過來，不再依賴他人；

Ceph 服務端進行flatten操作：rbd flatten kube/cloneimg01，此時它已經不需要依賴任何人了；

Ceph 服務端可以把快照取消保護：rbd snap unprotect kube/k8s01@clone01，此時快照刪除也沒有關係，因為其它人也不依賴你了；

Ceph 服務端刪除快照：rbd snap rm  kube/k8s01@clone01

Ceph 服務端可以匯入外部image：rbd import cenots.img kube/centos7