2021-08-31 第二天: ceph的進階使用(二)
阿新 • • 發佈：2021-09-03
一：物件儲存 RadosGW 使用
1.1RadosGW 物件儲存簡介
RadosGW 是物件儲存(OSS,Object Storage Service)的一種實現方式，RADOS 閘道器也稱為 Ceph 物件閘道器、RADOSGW、RGW，是一種服務，使客戶端能夠利用標準物件儲存 API 來訪問 Ceph 叢集，它支援AWS S3和Swift API，rgw運行於librados之上，在ceph 0.8版本之後使用Civetweb 的 web 伺服器來響應 api 請求，可以使用 nginx 或或者 apache 替代，客戶端基於 http/https 協議通過 RESTful API 與 rgw 通訊，而 rgw 則使用 librados 與 ceph 叢集通訊，rgw 客戶端通 過 s3 或者 swift api 使用 rgw 使用者進行身份驗證，然後 rgw 閘道器代表使用者利用 cephx 與 ceph 儲存進行身份驗證。
S3 由 Amazon 於 2006 年推出，全稱為 Simple Storage Service,S3 定義了物件儲存，是物件存 儲事實上的標準，從某種意義上說，S3 就是物件儲存，物件儲存就是 S3,它物件儲存市場的 霸主，後續的物件儲存都是對 S3 的模仿。
1.2物件儲存特點
通過物件儲存將資料儲存為物件，每個物件除了包含資料，還包含資料自身的元資料。 物件通過 Object ID 來檢索，無法通過普通檔案系統的方式通過檔案路徑及檔名稱操作來 直接訪問物件，只能通過 API 來訪問，或者第三方客戶端（實際上也是對 API 的封裝）。 物件儲存中的物件不整理到目錄樹中，而是儲存在扁平的名稱空間中，Amazon S3 將這個扁平名稱空間稱為 bucket，而 swift 則將其稱為容器。 無論是 bucket 還是容器，都不能巢狀。 bucket 需要被授權才能訪問到，一個帳戶可以對多個 bucket 授權，而許可權可以不同。 方便橫向擴充套件、快速檢索資料 不支援客戶端掛載,且需要客戶端在訪問的時候指定檔名稱。 不是很適用於檔案過於頻繁修改及刪除的場景。
ceph 使用 bucket)作為儲存桶(儲存空間)，實現物件資料的儲存和多使用者隔離，資料儲存在 bucket 中，使用者的許可權也是針對 bucket 進行授權，可以設定使用者對不同的 bucket 擁有不通 的許可權，以實現許可權管理
1.3部署 RadosGW 服務
將 ceph-mgr1、ceph-mgr2 伺服器部署為高可用的 radosGW 服務1.31 安裝 radosgw 服務並初始化
#mgr節點 test@ceph-mgr1:~$ sudo apt install radosgw test@ceph-mgr2:~$ sudo apt install radosgw#deploy節點 #在 ceph deploy 伺服器將 ceph-mgr1 ceph-mgr2初始化為 radosGW 服務 test@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr1 test@ceph-deploy:~/ceph-cluster$ ceph-deploy rgw create ceph-mgr2
1.32驗證 radosgw 服務狀態
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 635d9577-7341-4085-90ff-cb584029a1ea health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 2h) mgr: ceph-mgr2(active, since 20h), standbys: ceph-mgr1 mds: 2/2 daemons up, 2 standby osd: 12 osds: 12 up (since 2h), 12 in (since 2d) rgw: 2 daemons active (2 hosts, 1 zones) #2個執行 data: volumes: 1/1 healthy pools: 10 pools, 329 pgs objects: 372 objects, 314 MiB usage: 1.8 GiB used, 238 GiB / 240 GiB avail pgs: 329 active+clean
1.33驗證 radosgw 服務程序
#mgr節點 test@ceph-mgr1:~$ ps -ef|grep radosgw ceph 608 1 0 06:43 ? 00:00:27 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
1.34訪問 radosgw 服務
#deploy節點 test@ceph-deploy:~/ceph-cluster$ curl http://10.0.0.104:7480 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> test@ceph-deploy:~/ceph-cluster$ curl http://10.0.0.105:7480 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
1.4radosgw 服務配置
1.41radosgw 高可用架構
1.42自定義埠
radosgw 伺服器（ceph-mgr1、ceph-mgr2）的配置檔案要和deploy伺服器的一致，可以ceph-deploy 伺服器修改然後統一推送，或者單獨修改每個 radosgw 伺服器的配置為同一配置#deploy節點 test@ceph-deploy:~/ceph-cluster$ cat ceph.conf [global] fsid = 635d9577-7341-4085-90ff-cb584029a1ea public_network = 10.0.0.0/24 cluster_network = 192.168.133.0/24 mon_initial_members = ceph-mon1 mon_host = 10.0.0.101 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx mon clock drift allowed = 2 mon clock drift warn backoff = 30 [mds.ceph-mgr2] #mds_standby_for_fscid = mycephfs mds_standby_for_name = ceph-mgr1 mds_standby_replay = true [mds.ceph-mon3] mds_standby_for_name = ceph-mon2 mds_standby_replay = true [client.rgw.ceph-mgr1] rgw_host = ceph-mgr1 rgw_frontends = civetweb port=9900 [client.rgw.ceph-mgr2] rgw_host = ceph-mgr2 rgw_frontends = civetweb port=9900 #進行推送 test@ceph-deploy:~/ceph-cluster$ scp ceph.conf [email protected]:/etc/ceph/ test@ceph-deploy:~/ceph-cluster$ scp ceph.conf [email protected]:/etc/ceph/ #mgr節點 #進行重啟服務 test@ceph-mgr1:/etc/ceph$ sudo systemctl restart [email protected]mgr1.service test@ceph-mgr2:~$ sudo systemctl restart [email protected]
1.43啟用 SSL
生成簽名證書並配置 radosgw 啟用 SSL1.43.1自簽名證書
#mgr2節點 test@ceph-mgr2:~$ cd /etc/ceph/ test@ceph-mgr2:/etc/ceph$ sudo mkdir certs test@ceph-mgr2:/etc/ceph$ cd certs/ test@ceph-mgr2:/etc/ceph/certs$ sudo openssl genrsa -out civetweb.key 2048 test@ceph-mgr2:/etc/ceph/certs$ sudo openssl req -new -x509 -key civetweb.key -out civetweb.crt -subj "/CN=rgw.magedu.net" root@ceph-mgr2:/etc/ceph/certs# cat civetweb.key civetweb.crt > civetweb.pem root@ceph-mgr2:/etc/ceph/certs# ls civetweb.crt civetweb.key civetweb.pem
1.43.2SSL 配置
#mgr節點 root@ceph-mgr2:/etc/ceph# cat ceph.conf [global] fsid = 635d9577-7341-4085-90ff-cb584029a1ea public_network = 10.0.0.0/24 cluster_network = 192.168.133.0/24 mon_initial_members = ceph-mon1 mon_host = 10.0.0.101 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx mon clock drift allowed = 2 mon clock drift warn backoff = 30 [mds.ceph-mgr2] #mds_standby_for_fscid = mycephfs mds_standby_for_name = ceph-mgr1 mds_standby_replay = true [mds.ceph-mon3] mds_standby_for_name = ceph-mon2 mds_standby_replay = true [client.rgw.ceph-mgr1] rgw_host = ceph-mgr1 rgw_frontends = civetweb port=9900 [client.rgw.ceph-mgr2] rgw_host = ceph-mgr2 rgw_frontends = civetweb port=9900 [client.rgw.ceph-mgr2] rgw_host = ceph-mgr2 rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem"
#重啟服務
root@ceph-mgr2:/etc/ceph# systemctl restart [email protected]
1.43.3驗證 9443 埠
#mgr節點 root@ceph-mgr2:/etc/ceph# ss -tln State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 128 0.0.0.0:9443 0.0.0.0:* LISTEN 0 128 0.0.0.0:9900 0.0.0.0:* LISTEN 0 128 10.0.0.105:6800 0.0.0.0:* LISTEN 0 128 10.0.0.105:6801 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:43447 0.0.0.0:* LISTEN 0 128 [::1]:6010 [::]:* LISTEN 0 128 [::]:22 [::]:*
1.43.4驗證訪問
1.43.5優化配置
#mgr節點 #建立日誌目錄 test@ceph-mgr2:~$ sudo mkdir /var/log/radosgw test@ceph-mgr2:~$ sudo chown -R ceph:ceph /var/log/radosgw #修改配置 test@ceph-mgr2:~$ cat /etc/ceph/ceph.conf [client.rgw.ceph-mgr2] rgw_host = ceph-mgr2 rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log request_timeout_ms=30000 num_threads=200" #重啟服務 test@ceph-mgr2:~$ sudo systemctl restart [email protected]mgr2.service #訪問測試 test@ceph-mgr2:~$ curl -k https://10.0.0.105:9443 test@ceph-mgr2:~$ curl -k https://10.0.0.105:9443 #驗證日誌 test@ceph-mgr2:~$ tail /var/log/radosgw/civetweb.access.log 10.0.0.105 - - [31/Aug/2021:14:44:47 +0800] "GET / HTTP/1.1" 200 414 - curl/7.58.0 10.0.0.105 - - [31/Aug/2021:14:44:48 +0800] "GET / HTTP/1.1" 200 414 - curl/7.58.0 10.0.0.105 - - [31/Aug/2021:14:44:50 +0800] "GET / HTTP/1.1" 200 414 - curl/7.58.0
注:mgr1做一樣的操作
1.5 測試資料的讀寫
1.51 建立RGW賬戶
#deploy節點
test@ceph-deploy:~/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="test user" { "user_id": "user1", "display_name": "test user", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "user1", "access_key": "6LO8046SQ3DVGVKS84LX", "secret_key": "iiVFHXC6qc4iTnKVcKDVJaOLeIpl39EbQ2OwueRV" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }
1.5.2 安裝s3cmd客戶端
S3cmd是一個免費的命令列工具客戶端，用於在Amazon S3和其他使用S3協議的雲端儲存服務提供商（如京東雲OSS）上傳，檢索和管理資料。它適合熟悉命令列程式的高階使用者。它也是批處理指令碼和S3自動備份的理想選擇，由cron等觸發。
#deploy節點 test@ceph-deploy:~/ceph-cluster$ sudo apt-cache madison s3cmd s3cmd | 2.0.1-2 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages s3cmd | 2.0.1-2 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages test@ceph-deploy:~/ceph-cluster$ sudo apt install s3cmd
1.5.3 配置客戶端執行環境
1.53.1 s3cmd客戶端新增域名解析
#deploy節點 test@ceph-deploy:~/ceph-cluster$ cat /etc/hosts 127.0.0.1 localhost 127.0.1.1 ubuntu # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters 10.0.0.100 ceph-deploy.example.local ceph-deploy 10.0.0.101 ceph-mon1.example.local ceph-mon1 10.0.0.102 ceph-mon2.example.local ceph-mon2 10.0.0.103 ceph-mon3.example.local ceph-mon3 10.0.0.104 ceph-mgr1.example.local ceph-mgr1 10.0.0.105 ceph-mgr2.example.local ceph-mgr2 10.0.0.106 ceph-node1.example.local ceph-node1 10.0.0.107 ceph-node2.example.local ceph-node2 10.0.0.108 ceph-node3.example.local ceph-node3 10.0.0.109 ceph-node4.example.local ceph-node4 10.0.0.105 rgw.test.net
1.53.2 進行s3cm3配置
#deploy節點 test@ceph-deploy:~/ceph-cluster$ s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: 6LO8046SQ3DVGVKS84LX #建立使用者的時候的access key Secret Key: iiVFHXC6qc4iTnKVcKDVJaOLeIpl39EbQ2OwueRV #建立使用者的secret key Default Region [US]: Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: rgw.test.net:9900 Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw.test.net:9900/%(bucket) Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: No On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings: Access Key: 6LO8046SQ3DVGVKS84LX Secret Key: iiVFHXC6qc4iTnKVcKDVJaOLeIpl39EbQ2OwueRV Default Region: US S3 Endpoint: rgw.test.net:9900 DNS-style bucket+hostname:port template for accessing a bucket: rgw.test.net:9900/%(bucket) Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Not configured. Never mind. Save settings? [y/N] y Configuration saved to '/home/test/.s3cfg'
1.53.3建立bucket驗證許可權
#deploy節點 test@ceph-deploy:~/ceph-cluster$ s3cmd la test@ceph-deploy:~/ceph-cluster$ s3cmd mb s3://test Bucket 's3://test/' created test@ceph-deploy:~/ceph-cluster$ s3cmd ls 2021-08-31 08:08 s3://test
1.53.4 驗證上傳資料
#deploy節點 #上傳檔案 test@ceph-deploy:~$ s3cmd put /home/test/test.pdf s3://test/pdf/test.pdf #不寫檔名預設檔名 upload: '/home/test/test.pdf' -> 's3://test/pdf/test.pdf' [1 of 1] 4809229 of 4809229 100% in 1s 2.47 MB/s done #檢視檔案 test@ceph-deploy:~$ s3cmd la DIR s3://test/pdf/ #檢視檔案資訊 test@ceph-deploy:~$ s3cmd ls s3://test/pdf/ 2021-08-31 08:25 4809229 s3://test/pdf/test.pdf
1.53.5 驗證下載檔案
#deploy節點 test@ceph-deploy:~$ sudo s3cmd get s3://test/pdf/test.pdf /opt/ download: 's3://test/pdf/test.pdf' -> '/opt/test.pdf' [1 of 1] 4809229 of 4809229 100% in 0s 171.89 MB/s done test@ceph-deploy:~$ ll /opt/ total 4708 drwxr-xr-x 2 root root 4096 Aug 31 16:43 ./ drwxr-xr-x 23 root root 4096 Aug 22 15:29 ../ -rw-r--r-- 1 root root 4809229 Aug 31 08:25 test.pdf
1.53.6 刪除檔案
#deploy節點 test@ceph-deploy:~$ s3cmd ls s3://test/pdf/test.pdf 2021-08-31 08:25 4809229 s3://test/pdf/test.pdf test@ceph-deploy:~$ s3cmd rm s3://test/pdf/test.pdf delete: 's3://test/pdf/test.pdf' test@ceph-deploy:~$ s3cmd ls s3://test/pdf/test.pdf
二： Ceph crush進階
CRUSH演算法通過計算資料儲存位置來確定如何儲存和檢索資料。CRUSH使Ceph客戶機能夠直接與OSDs通訊，而不是通過集中的伺服器或代理。通過演算法確定的資料儲存和檢索方法，Ceph避免了單點故障、效能瓶頸和對其可伸縮性的物理限制。
2.1 PG與OSD對映調整
PG是一組物件的邏輯集合，通過複製它到不同的OSD上來提供儲存系統的可靠性。 根據Ceph池的複製級別，每個PG的資料會被複制並分發到Ceph叢集的多個OSD上。 可以將PG看成一個邏輯容器，這個容器包含多個物件，同時這個邏輯容器被對映到多個OSD上。
2.1.1 檢視當前狀態
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.01949 1.00000 20 GiB 102 MiB 81 MiB 14 KiB 21 MiB 20 GiB 0.50 0.86 89 up 1 hdd 0.01949 1.00000 20 GiB 130 MiB 95 MiB 27 KiB 35 MiB 20 GiB 0.63 1.10 98 up 2 hdd 0.01949 1.00000 20 GiB 129 MiB 96 MiB 6 KiB 34 MiB 20 GiB 0.63 1.09 83 up 3 hdd 0.01949 1.00000 20 GiB 106 MiB 71 MiB 13 KiB 35 MiB 20 GiB 0.52 0.90 87 up 4 hdd 0.01949 1.00000 20 GiB 128 MiB 94 MiB 8 KiB 33 MiB 20 GiB 0.62 1.08 91 up 5 hdd 0.01949 1.00000 20 GiB 123 MiB 88 MiB 23 KiB 35 MiB 20 GiB 0.60 1.04 91 up 6 hdd 0.01949 1.00000 20 GiB 121 MiB 86 MiB 8 KiB 35 MiB 20 GiB 0.59 1.02 84 up 7 hdd 0.01949 1.00000 20 GiB 119 MiB 91 MiB 18 KiB 28 MiB 20 GiB 0.58 1.00 95 up 8 hdd 0.01949 1.00000 20 GiB 72 MiB 43 MiB 18 KiB 29 MiB 20 GiB 0.35 0.61 91 up 9 hdd 0.01949 1.00000 20 GiB 129 MiB 93 MiB 6 KiB 37 MiB 20 GiB 0.63 1.09 92 up 10 hdd 0.01949 1.00000 20 GiB 141 MiB 111 MiB 11 KiB 30 MiB 20 GiB 0.69 1.19 106 up 11 hdd 0.01949 1.00000 20 GiB 120 MiB 87 MiB 17 KiB 33 MiB 20 GiB 0.59 1.01 100 up TOTAL 240 GiB 1.4 GiB 1.0 GiB 175 KiB 384 MiB 239 GiB 0.58 MIN/MAX VAR: 0.61/1.19 STDDEV: 0.08
2.1.2 修改WEIGHT並驗證
#deploy節點
#修改完會立即更新，速度取決於資料的大小，根據演算法進行分配
test@ceph-deploy:~/ceph-cluster$ ceph osd crush reweight osd.10 1.5 test@ceph-deploy:~/ceph-cluster$ ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.01949 1.00000 20 GiB 102 MiB 81 MiB 14 KiB 21 MiB 20 GiB 0.50 0.86 87 up 1 hdd 0.01949 1.00000 20 GiB 134 MiB 95 MiB 27 KiB 39 MiB 20 GiB 0.65 1.13 96 up 2 hdd 0.01949 1.00000 20 GiB 133 MiB 96 MiB 6 KiB 38 MiB 20 GiB 0.65 1.12 85 up 3 hdd 0.01949 1.00000 20 GiB 111 MiB 71 MiB 13 KiB 40 MiB 20 GiB 0.54 0.94 86 up 4 hdd 0.01949 1.00000 20 GiB 128 MiB 94 MiB 8 KiB 33 MiB 20 GiB 0.62 1.08 92 up 5 hdd 0.01949 1.00000 20 GiB 123 MiB 88 MiB 23 KiB 35 MiB 20 GiB 0.60 1.04 92 up 6 hdd 0.01949 1.00000 20 GiB 121 MiB 86 MiB 8 KiB 35 MiB 20 GiB 0.59 1.02 82 up 7 hdd 0.01949 1.00000 20 GiB 119 MiB 91 MiB 18 KiB 28 MiB 20 GiB 0.58 1.00 92 up 8 hdd 0.01949 1.00000 20 GiB 72 MiB 43 MiB 18 KiB 29 MiB 20 GiB 0.35 0.61 92 up 9 hdd 0.01949 1.00000 20 GiB 114 MiB 93 MiB 6 KiB 21 MiB 20 GiB 0.56 0.96 93 up 10 hdd 1.50000 1.00000 20 GiB 141 MiB 111 MiB 11 KiB 31 MiB 20 GiB 0.69 1.19 106 up 11 hdd 0.01949 1.00000 20 GiB 125 MiB 87 MiB 17 KiB 37 MiB 20 GiB 0.61 1.05 99 up TOTAL 240 GiB 1.4 GiB 1.0 GiB 175 KiB 387 MiB 239 GiB 0.58
2.1.2 修改REWEIGHT並驗證
#REWEIGHT的值範圍在0~1之間，值越小PG越小
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph osd reweight 9 0.6 reweighted osd.9 to 0.6 (9999) test@ceph-deploy:~/ceph-cluster$ ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.01949 1.00000 20 GiB 226 MiB 96 MiB 14 KiB 130 MiB 20 GiB 1.10 0.89 87 up 1 hdd 0.01949 1.00000 20 GiB 213 MiB 98 MiB 27 KiB 115 MiB 20 GiB 1.04 0.84 97 up 2 hdd 0.01949 1.00000 20 GiB 303 MiB 154 MiB 6 KiB 149 MiB 20 GiB 1.48 1.20 82 up 3 hdd 0.01949 1.00000 20 GiB 304 MiB 137 MiB 13 KiB 167 MiB 20 GiB 1.48 1.20 90 up 4 hdd 0.01949 1.00000 20 GiB 170 MiB 69 MiB 8 KiB 101 MiB 20 GiB 0.83 0.67 83 up 5 hdd 0.01949 1.00000 20 GiB 248 MiB 123 MiB 23 KiB 125 MiB 20 GiB 1.21 0.98 86 up 6 hdd 0.01949 1.00000 20 GiB 232 MiB 99 MiB 8 KiB 133 MiB 20 GiB 1.13 0.92 88 up 7 hdd 0.01949 1.00000 20 GiB 301 MiB 154 MiB 18 KiB 147 MiB 20 GiB 1.47 1.19 90 up 8 hdd 0.01949 1.00000 20 GiB 145 MiB 42 MiB 18 KiB 103 MiB 20 GiB 0.71 0.57 89 up 9 hdd 0.01949 0.59999 20 GiB 199 MiB 91 MiB 6 KiB 108 MiB 20 GiB 0.97 0.79 54 up 10 hdd 0.01949 1.00000 20 GiB 544 MiB 303 MiB 11 KiB 240 MiB 19 GiB 2.66 2.15 144 up 11 hdd 0.01949 1.00000 20 GiB 145 MiB 70 MiB 17 KiB 75 MiB 20 GiB 0.71 0.57 96 up TOTAL 240 GiB 3.0 GiB 1.4 GiB 175 KiB 1.6 GiB 237 GiB 1.23
2.2 crush 執行圖管理
通過工具將ceph的crush執行圖匯出並進行編輯，然後匯入
2.2.1 匯出crush執行圖
匯出的crush執行圖為二進位制格式，要使用crushtool工具轉換為文字格式後才能進行編輯
#deploy節點 test@ceph-deploy:~/ceph-cluster$ sudo mkdir /data/ceph -p test@ceph-deploy:~/ceph-cluster$ sudo ceph osd getcrushmap -o /data/ceph/crushmap 77
2.2.2 將執行圖轉換為文字
匯出的執行圖不能編輯，需要轉換為文字再進行檢視與編輯
#deploy節點 root@ceph-deploy:~# apt install -y ceph-base root@ceph-deploy:~# crushtool -d /data/ceph/crushmap > /data/ceph/crushmap.txt root@ceph-deploy:~# file /data/ceph/crushmap.txt test@ceph-deploy:~/ceph-cluster$ sudo vim /data/ceph/crushmap.txt # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 6 step take default step chooseleaf firstn 0 type host step emit }
2.2.3 將文字轉換成crush格式
#deploy節點 test@ceph-deploy:~/ceph-cluster$ sudo crushtool -c /data/ceph/crushmap.txt -o /data/ceph/newcrushmap
2.2.4 匯入新的crush
匯入的執行圖會立即覆蓋原有的執行圖並生效
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph osd setcrushmap -i /data/ceph/newcrushmap 78
2.2.5 驗證crush執行圖是否生效
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 6, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ]
三: Ceph dashboard 及監控
Dashboard（管理看板）是將多個儀表、圖表、報表等元件內容整合在一個面板上進行綜合顯示的功能模組，提供靈活的元件及面板定義，並且提供大量預設定的元件模板，方便使用者靈活選擇，提高工作效率。可以使分析結果更具有良好的直觀性、可理解性，快速掌握運營動態，為決策者做出決策提供更有利的資料支援。 3.1.1 啟用 dashboard 外掛Ceph mgr 是一個多外掛(模組化)的元件，其元件可以單獨的啟用或關閉
#新版本需要安裝 dashboard 安保，而且必須安裝在 mgr 節點 #mgr節點 test@ceph-mgr1:/etc/ceph$ apt-cache madison ceph-mgr-dashboard ceph-mgr-dashboard | 16.2.5-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages ceph-mgr-dashboard | 16.2.5-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main i386 Packages test@ceph-mgr1:/etc/ceph$ sudo apt install ceph-mgr-dashboard #deploy節點 #列出所以版塊 test@ceph-deploy:~/ceph-cluster$ ceph mgr module ls #啟動版塊 test@ceph-deploy:~/ceph-cluster$ ceph mgr module enable dashboard 注：模組啟用後還不能直接訪問，需要配置關閉 SSL 或啟用 SSL 及指定監聽地址。
3.1.2啟用 dashboard 模組
Ceph dashboard 在 mgr 節點進行開啟設定，並且可以配置開啟或者關閉 SSL#deploy節點 #關閉 SSL test@ceph-deploy:~/ceph-cluster$ ceph config set mgr mgr/dashboard/ssl false #指定 dashboard 監聽地址 test@ceph-deploy:~/ceph-cluster$ ceph config set mgr mgr/dashboard/ceph-mgr1/server_addr 10.0.0.104 #指定 dashboard 監聽埠 test@ceph-deploy:~/ceph-cluster$ ceph config set mgr mgr/dashboard/ceph-mgr1/server_port 9009 #驗證叢集狀態 test@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 635d9577-7341-4085-90ff-cb584029a1ea health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 4h) mgr: ceph-mgr1(active, since 3m), standbys: ceph-mgr2 mds: 2/2 daemons up, 2 standby osd: 12 osds: 12 up (since 4h), 12 in (since 3d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 329 pgs objects: 372 objects, 314 MiB usage: 1.2 GiB used, 239 GiB / 240 GiB avail pgs: 329 active+clean
3.13在 mgr 節點驗證埠與程序
#mgr節點 #檢查mgr服務是否正常執行，檢視埠資訊，如果不正常啟動，重啟一下服務 test@ceph-mgr1:~$ sudo systemctl restart ceph-mgr@ceph-mgr1.service test@ceph-mgr1:~$ ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:54113 0.0.0.0:* LISTEN 0 128 0.0.0.0:9443 0.0.0.0:* LISTEN 0 128 127.0.0.1:42569 0.0.0.0:* LISTEN 0 128 0.0.0.0:9900 0.0.0.0:* LISTEN 0 128 0.0.0.0:111 0.0.0.0:* LISTEN 0 5 10.0.0.104:9009 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 128 [::ffff:0.0.0.0]:2049 *:* LISTEN 0 128 [::]:43399 [::]:* LISTEN 0 128 [::]:111 [::]:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::1]:6010 [::]:*
3.14dashboard 訪問驗證
3.15設定 dashboard 賬戶及密碼
#deploy節點 test@ceph-deploy:~/ceph-cluster$ sudo touch pass.txt test@ceph-deploy:~/ceph-cluster$ echo "123456" > pass.txt test@ceph-deploy:~/ceph-cluster$ ceph dashboard set-login-credentials test -i pass.txt ****************************************************************** *** WARNING: this command is deprecated. *** *** Please use the ac-user-* related commands to manage users. *** ****************************************************************** Username and password updated
1.16 登入介面
1.17 登入成功介面
1.18dashboard SSL
如果要使用 SSL 訪問。則需要配置簽名證書。證書可以使用 ceph 命令生成，或是 opessl 命令生成1.18.1ceph 自簽名證書
#deploy節點 #生成證書 test@ceph-deploy:~/ceph-cluster$ ceph dashboard create-self-signed-cert Self-signed certificate created #啟用 SSL test@ceph-deploy:~/ceph-cluster$ ceph config set mgr mgr/dashboard/ssl true #檢視當前 dashboard 狀態 test@ceph-deploy:~/ceph-cluster$ ceph mgr services { "dashboard": "http://10.0.0.104:9009/" } #mgr節點 #重啟 mgr 服務 test@ceph-mgr1:~$ sudo systemctl restart ceph-mgr@ceph-mgr1 #再次驗證dashboard 狀態
test@ceph-deploy:~/ceph-cluster$ ceph mgr services
{
"dashboard": "https://10.0.0.104:9009/"
}
1.2通過 prometheus 監控 ceph node 節點
1.21部署 prometheus
#mgr節點 test@ceph-mgr1:~$ sudo mkdir /apps test@ceph-mgr1:~$ cd /apps test@ceph-mgr1:/apps$ sudo tar xf prometheus-2.23.0.linux-amd64.tar.gz test@ceph-mgr1:/apps$ sudo ln -sv /apps/prometheus-2.23.0.linux-amd64 /apps/prometheus #方便以後的升級 root@ceph-mgr1:~# cat /etc/systemd/system/prometheus.service [Unit] Description=Prometheus Server Documentation=https://prometheus.io/docs/introduction/overview/ After=network.target [Service] Restart=on-failure WorkingDirectory=/apps/prometheus/ ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml [Install] WantedBy=multi-user.target root@ceph-mgr1:~# systemctl daemon-reload root@ceph-mgr1:~# sudo systemctl start prometheus.service root@ceph-mgr1:~# sudo systemctl enable prometheus.service Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.
1.22訪問 prometheus
1.23部署 node_exporter
#node節點驗證各 node 節點的 node_exporter 資料
#3個節點都做這個操作
root@ceph-node1:~# mkdir /apps/ root@ceph-node1:~# cd /apps/ root@ceph-node1:/apps# tar xf node_exporter-1.0.1.linux-amd64.tar.gz root@ceph-node1:/apps# ln -sv /apps/node_exporter-1.0.1.linux-amd64 /apps/node_exporter root@ceph-node1:/apps# cat /etc/systemd/system/node-exporter.service [Unit] Description=Prometheus Node Exporter After=network.target [Service] ExecStart=/apps/node_exporter/node_exporter [Install] WantedBy=multi-user.target root@ceph-node1:/apps# systemctl daemon-reload root@ceph-node1:/apps# systemctl restart node-exporter root@ceph-node1:/apps# systemctl enable node-exporter Created symlink /etc/systemd/system/multi-user.target.wants/node-exporter.service → /etc/systemd/system/node-exporter.service.
1.24配置 prometheus server 資料並驗證
#mgr節點 root@ceph-mgr1:~# cd /apps/prometheus root@ceph-mgr1:/apps/prometheus# cat prometheus.yml - job_name: 'ceph-node-data' static_configs: - targets: ['10.0.0.106:9100','10.0.0.107:9100','10.0.0.108:9100'] root@ceph-mgr1:/apps/prometheus# systemctl restart prometheus.service
驗證節點
1.3通過 prometheus 監控 ceph 服務
Ceph manager 內部的模組中包含了 prometheus 的監控模組,並監聽在每個 manager 節點的9283 埠，該埠用於將採集到的資訊通過 http 介面向 prometheus 提供資料1.31啟用 prometheus 監控模組
#deploy節點 test@ceph-deploy:~/ceph-cluster$ ceph mgr module enable prometheus #mgr節點 root@ceph-mgr1:/apps/prometheus# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 128 0.0.0.0:9443 0.0.0.0:* LISTEN 0 128 0.0.0.0:9900 0.0.0.0:* LISTEN 0 128 127.0.0.1:42447 0.0.0.0:* LISTEN 0 128 0.0.0.0:111 0.0.0.0:* LISTEN 0 128 10.0.0.104:6800 0.0.0.0:* LISTEN 0 5 10.0.0.104:9009 0.0.0.0:* LISTEN 0 128 10.0.0.104:6801 0.0.0.0:* LISTEN 0 128 0.0.0.0:52241 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::1]:6010 [::]:* LISTEN 0 128 [::ffff:0.0.0.0]:2049 *:* LISTEN 0 128 *:9090 *:* LISTEN 0 5 *:9283 *:* LISTEN 0 128 [::]:36107 [::]:* LISTEN 0 128 [::]:111 [::]:*
1.32驗證 manager 資料
1.33配置 prometheus 採集資料
#mgr節點 root@ceph-mgr1:/apps/prometheus# cat prometheus.yml # my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: 'ceph-node-data' static_configs: - targets: ['10.0.0.106:9100','10.0.0.107:9100','10.0.0.108:9100'] - job_name: 'ceph-cluster-data' static_configs: - targets: ['10.0.0.104:9283'] root@ceph-mgr1:/apps/prometheus# systemctl restart prometheus.service
1.34驗證資料
1.4通過 grafana 顯示監控資料
通過 granfana 顯示對 ceph 的叢集監控資料及 node 資料 1.41安裝 grafanagrafana下載地址：Download Grafana | Grafana Labs
#deploy節點 root@ceph-deploy:~# sudo apt-get install -y adduser libfontconfig1 root@ceph-deploy:~# wget https://dl.grafana.com/oss/release/grafana_7.5.10_amd64.deb root@ceph-deploy:~# sudo dpkg -i grafana_7.5.10_amd64.deb root@ceph-deploy:~# systemctl enable grafana-server Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable grafana-server Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service. root@ceph-deploy:~# systemctl start grafana-server
1.42登入 grafana
賬號密碼預設admin
1.43配置資料來源
在 grafana 新增 prometheus 資料來源1.44匯入模板
模板地址：Grafana Dashboards - discover and share dashboards for Grafana. | Grafana Labs
1.45 效果圖