一、背景說明

1、問題描述

建立4T大小krbd（未寫入任何資料），刪除塊耗時將近二十多分鐘

2、問題排查

2.1、排查思路

預設rbd特性：layering, exclusive-lock, object-map, fast-diff, deep-flatten

[root@node117 ~]# rbd info rbd/lun02
rbd image 'lun02':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.b5cd36b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags: 
    create_timestamp: Tue Dec 22 10:31:06 2020
 

krbd特性：layering, exclusive-lock
當前系統核心版本過低，不支援deep-flatten、fast-diff、object-map特性，執行rbd map操作時，報錯資訊RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable lun02 object-map fast-diff deep-flatten".，故krbd需要關閉這三個特性

[root@node117 ~]# uname -a
Linux node117 4.14.113-1.el7.x86_64 #1 SMP Wed Sep 9 17:22:41 CST 2020 x86_64 x86_64 x86_64 GNU/Linux


[root@node117 ~]# rbd info rbd/lun03
rbd image 'lun03':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.b759e6b8b4567
    format: 2
    features: layering, exclusive-lock
    flags: 
    create_timestamp: Tue Dec 22 10:31:22 2020
 

4T大小的預設rbd可以在1s內完成刪除，依次關閉deep-flatten、fast-diff、object-map比對測試，發現在關閉object-map特性後，刪除耗時增加二十多分鐘

-關閉deep-flatten特性，刪除耗時0.616s
[root@node117 ~]# rbd feature disable rbd/lun003 deep-flatten
[root@node117 ~]# rbd info rbd/lun003
rbd image 'lun003':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.4006d6b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff
    flags: 
    create_timestamp: Mon Dec 21 18:20:15 2020
[root@node117 ~]# time rbd rm rbd/lun003
Removing image: 100% complete...done.


real    0m0.616s
user    0m0.393s
sys    0m0.013s


-關閉deep-flatten、fast-diff特性，刪除耗時0.659s
[root@node117 ~]# rbd feature disable rbd/lun004 deep-flatten,fast-diff
[root@node117 ~]# rbd info rbd/lun004
rbd image 'lun004':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.416d66b8b4567
    format: 2
    features: layering, exclusive-lock, object-map
    flags: 
    create_timestamp: Mon Dec 21 18:26:41 2020
[root@node117 ~]# time rbd rm rbd/lun004
Removing image: 100% complete...done.


real    0m0.659s
user    0m0.398s
sys    0m0.017s


-關閉deep-flatten、fast-diff、object-map特性，刪除耗時21m36.884s
[root@node117 ~]# rbd feature disable rbd/lun005 deep-flatten,fast-diff,object-map
[root@node117 ~]# rbd info rbd/lun005
rbd image 'lun005':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.41f676b8b4567
    format: 2
    features: layering, exclusive-lock
    flags: 
    create_timestamp: Mon Dec 21 18:31:12 2020
[root@node117 ~]# time rbd rm rbd/lun005
Removing image: 100% complete...done.


real    21m36.884s
user    1m20.757s
sys    0m48.640s
 

2.2、根因分析

塊物件分配為thin-provisioning（自動精簡配置），建立塊時，並不會佔真實的容量，只有當塊寫入資料時，才會根據寫入資料量分配對應的物件數（預設4MB一個物件）

• 當開啟object-map特性時，會記錄塊所有物件的一個位圖，用以標記物件是否真的存在，未寫入資料的塊不包含任何物件，直接回收塊，故刪除速度很快
• 當關閉object-map特性時，ceph無法得知該塊是否真實寫入資料（4M一個物件，4T塊有1048576 個物件），在執行刪除操作時，仍會向rados發起請求，一個個物件進行刪除（即便該塊不存在任何物件資料），故刪除速度很慢

[root@node117 ~]# rbd create rbd/pp --size 40000MB
[root@node117 ~]# rbd info rbd/pp
rbd image 'pp':
    size 39.1GiB in 10000 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.be1136b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags: 
    create_timestamp: Tue Dec 22 11:22:49 2020
[root@node117 ~]# rados -p rbd ls | grep rbd_data.be1136b8b4567 | wc -l
0

二、解決方法

根據一中的根因分析，需要升級核心版本以支援object-map, fast-diff, deep-flatten特性，示例升級核心版本至5.10.2

1、核心升級

• 下載5.10.2核心版本包
  kernel-ml-devel-5.10.2-1.el7.elrepo.x86_64.rpm
  kernel-ml-5.10.2-1.el7.elrepo.x86_64.rpm
• 安裝5.10.2核心版本包

[root@node147 kernel5.10.2]# rpm -ivh kernel-ml-5.10.2-1.el7.elrepo.x86_64.rpm  kernel-ml-devel-5.10.2-1.el7.elrepo.x86_64.rpm

• 設定預設啟動項

[root@node147 kernel5.10.2]# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
0 : CentOS Linux (5.10.2-1.el7.elrepo.x86_64) 7 (Core)
1 : CentOS Linux (4.14.113-1.el7.x86_64) 7 (Core)
2 : CentOS Linux (4.4.248-1.el7.elrepo.x86_64) 7 (Core)
3 : CentOS Linux (3.10.0-957.el7.x86_64) 7 (Core)
4 : CentOS Linux (0-rescue-ea5fd2d87aff411a84e213a00b214bf2) 7 (Core)
[root@node147 kernel5.10.2]# grub2-set-default "CentOS Linux (5.10.2-1.el7.elrepo.x86_64) 7 (Core)"

2、測試驗證

• 未升級核心版本前，rbd對映失敗，提示核心不支援object-map fast-diff deep-flatten特性

[root@node147 ~]# rbd create rbd/lun001 --size 4096G
[root@node147 ~]# rbd info rbd/lun001
rbd image 'lun001':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.40dcf76b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags: 
    create_timestamp: Tue Dec 22 07:51:02 2020
[root@node147 ~]# rbd map rbd/lun001
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable lun001 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@node147 ~]# uname -a
Linux node147 4.14.113-1.el7.x86_64 #1 SMP Wed Sep 9 17:22:41 CST 2020 x86_64 x86_64 x86_64 GNU/Linux

• 升級核心版本後，rbd對映成功，且可以很快刪除完rbd

[root@node147 ~]# rbd info rbd/lun001
rbd image 'lun001':
    size 4TiB in 1048576 objects
    order 22 (4MiB objects)
    block_name_prefix: rbd_data.40dcf76b8b4567
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    flags: 
    create_timestamp: Tue Dec 22 07:51:02 2020
[root@node147 ~]# rbd map rbd/lun001
/dev/rbd0
[root@node147 ~]# uname -a
Linux node147 5.10.2-1.el7.elrepo.x86_64 #1 SMP Sun Dec 20 09:53:23 EST 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@node147 ~]# time rbd rm rbd/lun001
Removing image: 100% complete...done.

real    0m12.197s
user    0m0.599s
sys    0m0.028s

三、擴充套件補充

1、rbd特性說明

rbd預設啟用layering, exclusive-lock, object-map, fast-diff, deep-flatten特性

[root@node117 ~]# ceph --show-config | grep rbd_default_feature
rbd_default_features = 61