0 簡介

ceph-objectstore-tool工具是Ceph提供的一個low level（可以操作object和pg）的工具，可以用來檢視、修改、刪除ObjectStore上的object、pg以及dump OSD journal中的資料，請謹慎使用這個工具，它有可能會造成資料的永久丟失，可能這也是很少有文章介紹它的一個原因吧。

1 使用

ceph-objectstore-tool目前沒有manpage，它的help文件寫的也不是很友好。

[[email protected] ~]# ceph-objectstore-tool -h
Must provide --data-path

Allowed options:
  --help                      produce help message
  --type arg                  Arg is one of [filestore (default), memstore]
  --data-path arg             path to object store, mandatory
  --journal-path arg          path to journal, mandatory for filestore type
  --pgid arg                  PG id, mandatory for info, log, remove, export,
                              rm-past-intervals, mark-complete
  --op arg                    Arg is one of [info, log, remove, mkfs, fsck,
                              fuse, export, import, list, fix-lost, list-pgs,
                              rm-past-intervals, dump-journal, dump-super,
                              meta-list, get-osdmap, set-osdmap,
                              get-inc-osdmap, set-inc-osdmap, mark-complete]
  --epoch arg                 epoch# for get-osdmap and get-inc-osdmap, the
                              current epoch in use if not specified
  --file arg                  path of file to export, import, get-osdmap,
                              set-osdmap, get-inc-osdmap or set-inc-osdmap
  --mountpoint arg            fuse mountpoint
  --format arg (=json-pretty) Output format which may be json, json-pretty,
                              xml, xml-pretty
  --debug                     Enable diagnostic output to stderr
  --force                     Ignore some types of errors and proceed with
                              operation - USE WITH CAUTION: CORRUPTION POSSIBLE
                              NOW OR IN THE FUTURE
  --skip-journal-replay       Disable journal replay
  --skip-mount-omap           Disable mounting of omap
  --head                      Find head/snapdir when searching for objects by
                              name
  --dry-run                   Don't modify the objectstore

Positional syntax:
ceph-objectstore-tool ... <object> (get|set)-bytes [file]
ceph-objectstore-tool ... <object> set-(attr|omap) <key> [file]
ceph-objectstore-tool ... <object> (get|rm)-(attr|omap) <key>
ceph-objectstore-tool ... <object> get-omaphdr
ceph-objectstore-tool ... <object> set-omaphdr [file]
ceph-objectstore-tool ... <object> list-attrs
ceph-objectstore-tool ... <object> list-omap
ceph-objectstore-tool ... <object> remove
ceph-objectstore-tool ... <object> dump
ceph-objectstore-tool ... <object> set-size
ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid>

<object> can be a JSON object description as displayed by --op list.
<object> can be an object name which will be looked up in all the OSD's PGs.
<object> can be the empty string ('') which with a provided pgid specifies the pgmeta object

The optional [file] argument will read stdin or write stdout if not specified or if '-' specified.
 

3 使用事例

下面介紹一下，ceph-objectstore-tool的常用用法

1）list osd上有哪些pg，比如如果要list osd.2種儲存哪些pg，可以執行以下命令：

<span style="font-size:10px;">[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path 
/var/lib/ceph/osd/ceph-2/journal --type filestore --op list-pgs</span>

2）除了能list pg之外，還能list某個pg的pg_info資訊

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path 
/var/lib/ceph/osd/ceph-2/journal --type filestore  --pgid 0.0 --op info</span></span>
 

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --type filestore  --pgid 0.38 --op list
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
["0.38",{"oid":"rbd_id.test","key":"","snapid":-2,"hash":2586801272,"max":0,"pool":0,"namespace":"","max":0}]
["0.38",{"oid":"rbd_data.6e742ae8944a.0000000000000000","key":"","snapid":-2,"hash":3591771128,"max":0,"pool":0,"namespace":"","max":0}]

3) dump某個pg的pg_log資訊

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --type filestore  --pgid 0.36 --op log
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
{
    "pg_log_t": {
        "head": "400'27",
        "tail": "0'0",
        "log": [
            {
                "op": "modify  ",
                "object": "0:6c8c8ef5:::rb.0.5e9a.2ae8944a.0000000000db:head",
                "version": "376'1",
                "prior_version": "0'0",
                "reqid": "client.24221.1:135326",
                "extra_reqids": [],
                "mtime": "2016-07-21 22:19:29.377000",
                "mod_desc": {
                    "object_mod_desc": {
                        "can_local_rollback": false,
                        "rollback_info_completed": true,
                        "ops": []
                    }
                }
            },
...
        ]
    },
    "pg_missing_t": {
        "missing": []
    },
    "map": {
        "divergent_priors": []
    }

4) remove某個pg

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --type filestore  --pgid 0.36 --op remove

5) export/import 一個pg

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --type filestore  --pgid 0.36 --op export --file 1.txt
[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --type filestore  --op import --file 1.txt

6) 該命令還可以dump出OSD的super_block資訊

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal --op dump-super
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
{
    "cluster_fsid": "7b4eb697-fd53-4bd8-9b7a-ca1b37110623",
    "osd_fsid": "3549e257-373a-44a0-a7d9-99068e9854dc",
    "whoami": 2,
    "current_epoch": 442,
    "oldest_map": 1,
    "newest_map": 442,
    "weight": 0.000000,
    "compat": {
        "compat": {},
        "ro_compat": {},
        "incompat": {
            "feature_1": "initial feature set(~v.18)",
            "feature_2": "pginfo object",
            "feature_3": "object locator",
            "feature_4": "last_epoch_clean",
            "feature_5": "categories",
            "feature_6": "hobjectpool",
            "feature_7": "biginfo",
            "feature_8": "leveldbinfo",
            "feature_9": "leveldblog",
            "feature_10": "snapmapper",
            "feature_12": "transaction hints",
            "feature_13": "pg meta object"
        }
    },
    "clean_thru": 442,
    "last_epoch_mounted": 439
}

7）dump一個object所有的元資料資訊

[[email protected] 0.13_head]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --journal-path /var/lib/ceph/osd/ceph-2/journal rbd_data.6e742ae8944a.0000000000000013 dump
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
{
    "id": {
        "oid": "rbd_data.6e742ae8944a.0000000000000013",
        "key": "",
        "snapid": -2,
        "hash": 4028149687,
        "max": 0,
        "pool": 0,
        "namespace": "",
        "max": 0
    },
    "info": {
        "oid": {
            "oid": "rbd_data.6e742ae8944a.0000000000000013",
            "key": "",
            "snapid": -2,
            "hash": 4028149687,
            "max": 0,
            "pool": 0,
            "namespace": ""
        },
        "version": "442'1060",
        "prior_version": "442'1059",
        "last_reqid": "osd.1.0:59339",
        "user_version": 1059,
        "size": 4194304,
        "mtime": "2016-07-22 14:58:47.086000",
        "local_mtime": "2016-07-22 14:58:47.306385",
        "lost": 0,
        "flags": 52,
        "snaps": [],
        "truncate_seq": 0,
        "truncate_size": 0,
        "data_digest": 3408368108,
        "omap_digest": 4294967295,
        "watchers": {}
    },
    "stat": {
        "size": 4194304,
        "blksize": 4096,
        "blocks": 8192,
        "nlink": 1
    },
    "SnapSet": {
        "snap_context": {
            "seq": 0,
            "snaps": []
        },
        "head_exists": 1,
        "clones": []
    }
}

8)  當然，它可以使用get-osdmap匯出叢集中leveldb資料庫中的資料，比如OSDMap，而且可以dump出各個epoch的map，這個大家可以在下面自己試驗一下。