介紹：前期對ceph有一個簡單的介紹，但是內容太大，並不具體，接下來使用ceph-deploy部署一個Ceph集群，並做一些運維管理工作，深入的理解Ceph原理及工作工程！

一、環境準備

本次使用的虛擬機測試，使用7.6系統最小化安裝，CentOS Linux release 7.6.1810 (Core)

主機規劃：

網絡規劃：

192.168.2.0/24 集群網絡

172.25.254.0/24 OSD復制網絡+iscsi映射網絡

功能實現：

1. Ceph集群：monitor、manager、osd*2
2. iscsi：client
3. monitor：dashboard

主機前期準備：

每個節點都要做

修改主機名安裝必要軟件

hostnamectl set-hostname username
hostname  username
yum install -y net-tools wget vim
yum update

配置阿裏源

rm -f /etc/yum.repos.d/*
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
 

sed -i ‘/aliyuncs.com/d‘ /etc/yum.repos.d/*.repo
echo ‘#阿裏ceph源
> [ceph]
> name=ceph
> baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/
> gpgcheck=0
> [ceph-noarch]
> name=cephnoarch
> baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/
> gpgcheck=0
> [ceph-source]
> name=ceph-source
> baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS/
> gpgcheck=0
> #‘>/etc/yum.repos.d/ceph.repo
 yum clean all && yum makecache

配置時間同步

yum install ntp ntpdate ntp-doc

部署節點配置主機名

[root@ceph1 ~]# vi /etc/hosts

172.25.254.130  ceph1
172.25.254.131  ceph2
172.25.254.132  ceph3
172.25.254.133  ceph4
172.25.254.134  ceph5
172.25.254.135  ceph6               

配置部署節點到所有osd節點免密登錄：

[root@ceph1 ~]# useradd manager #創建非root用戶

[root@ceph1 ~]# echo redhat | passwd --stdin manager

[root@ceph1 ~]# echo "manager ALL = (root) NOPASSWD:ALL" > /etc/sudoers.d/manager

[root@ceph1 ~]# for i in {1..5}; do echo "====ceph${i}====";ssh root@ceph${i} ‘useradd -d /home/cephuser -m cephuser; echo "redhat" | passwd --stdin cephuser‘; done #所有osd節點創建cephuser用戶

[root@ceph1 ~]# for i in {1..5}; do echo "====ceph${i}====";ssh root@ceph${i} ‘echo "cephuser ALL = (root) NOPASSWD:ALL" > /etc/sudoers.d/cephuser‘; done

[root@ceph1 ~]# for i in {1..5}; do echo "====ceph${i}====";ssh root@ceph${i} ‘chmod 0440 /etc/sudoers.d/cephuser‘; done

[root@ceph1 ~]# su - manager

[cephuser@ceph1 ~]$ ssh-keygen -f ~/.ssh/id_rsa -N ‘‘

[cephuser@ceph1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

[cephuser@ceph1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub cephuser@172.25.254.131

[cephuser@ceph1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub cephuser@172.25.254.132

[cephuser@ceph1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub cephuser@172.25.254.133

[cephuser@ceph1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub cephuser@172.25.254.134

在部署節點ceph1上安徽在哪個匹配環境

[root@ceph1 ~]# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
[root@ceph1 ~]# python get-pip.py

修改部署節點的config文件

[cephuser@ceph1 ~]$ vi ~/.ssh/config

Host node1
  Hostname ceph1
  User cephuser
Host node2
  Hostname crph2
  User cephuser
Host node3
  Hostname ceph3
  User cephuser
Host node4
  Hostname ceph4
  User cephuser
Host node5
  Hostname ceph5
  User cephuser

[cephuser@ceph1 ~]$ chmod 600 .ssh/config

測試

[cephuserr@ceph1 ~]$ ssh cephuser@ceph2

[cephuser@ceph2 ~]$ exit

關閉selinux並配置防火墻：

[root@ceph1 ~]# sed -i ‘/^SELINUX=.*/c SELINUX=perimissive‘ /etc/selinux/config

[root@ceph1 ~]# sed -i ‘s/^SELINUXTYPE=.*/SELINUXTYPE=disabled/g‘ /etc/selinux/config

[root@ceph1 ~]# grep --color=auto ‘^SELINUX‘ /etc/selinux/config

SELINUX=perimissive
SELINUXTYPE=disabled

[root@ceph1 ~]# setenforce 0

[root@ceph1 ~]# firewall-cmd --list-all

public (active)
target: default
icmp-block-inversion: no
interfaces: ens33
sources: 
services: ssh dhcpv6-client
ports: 
protocols: 
masquerade: no
forward-ports: 
source-ports: 
icmp-blocks: 
rich rules: 

[root@ceph1 ~]# firewall-cmd --zone=public --add-port 80/tcp --permanent

[root@ceph1 ~]# firewall-cmd --zone=public --add-port 6800-7300/tcp --permanent

[root@ceph1 ~]# firewall-cmd --reload

創建集群

以下操作均在控制節點完成

[root@ceph1 ~]# su - cephuser

創建配置文件

[cephuser@ceph1 ~]$ cd

[cephuser@ceph1 ~]$ mkdir my-cluster

[cephuser@ceph1 ~]$ cd my-cluster

創建監控節點

[cephuser@ceph1 my-cluster]$ ceph-deploy new ceph2

[cephuser@ceph1 my-cluster]$ ll

-rw-rw-r--. 1 cephuser cephuser  197 Mar 14 23:13 ceph.conf
-rw-rw-r--. 1 cephuser cephuser 3166 Mar 14 23:13 ceph-deploy-ceph.log
-rw-------. 1 cephuser cephuser   73 Mar 14 23:13 ceph.mon.keyring

該目錄存在一個 Ceph 配置文件、一個 monitor 密鑰環和一個日誌文件。

配置網絡

[cephuser@ceph1 my-cluster]$ vi ceph.conf

public network = 172.15.254.024
cluster network = 192.168.2.0/24

安裝ceph：

[cephuser@ceph1 my-cluster]$ ceph-deploy install ceph1 ceph2 ceph3 ceph4 ceph5

初始化監控節點

[cephuser@ceph1 my-cluster]$ ceph-deploy mon create-initial

[cephuser@ceph1 my-cluster]$ ll

-rw-------. 1 cephuser cephuser     71 Mar 14 23:29 ceph.bootstrap-mds.keyring
-rw-------. 1 cephuser cephuser     71 Mar 14 23:29 ceph.bootstrap-mgr.keyring
-rw-------. 1 cephuser cephuser     71 Mar 14 23:29 ceph.bootstrap-osd.keyring
-rw-------. 1 cephuser cephuser     71 Mar 14 23:29 ceph.bootstrap-rgw.keyring
-rw-------. 1 cephuser cephuser     63 Mar 14 23:29 ceph.client.admin.keyring
-rw-rw-r--. 1 cephuser cephuser    197 Mar 14 23:29 ceph.conf
-rw-rw-r--. 1 cephuser cephuser 310977 Mar 14 23:30 ceph-deploy-ceph.log
-rw-------. 1 cephuser cephuser     73 Mar 14 23:29 ceph.mon.keyring

部署MGR

[cephuser@ceph1 my-cluster]$ ceph-deploy mgr create ceph2 ceph3 ceph4

運行卡住，重新來過

002 ceph的deploy部署