初始環境：

 

系統環境:centos73.10.0-514.26.2.el7.x86_64

機器數量：兩臺

硬碟：至少兩塊,一塊為系統盤，另一塊留作他用

命名規則：node1 node2

IP規劃：192.168.238.129 node1

192.168.238.130 node2

 

 

1.磁碟設定（裝完系統後，格式化另外的硬碟，兩節點都需要）

 

[[email protected] ~]#fdisk /dev/sdb#格式化磁碟

 

2.設定磁碟格式掛載硬碟

 

(on both nodes):

Note:

These examples are going to assume the brick is going to reside on /dev/sdb1.(兩節點都需要做)

[[email protected] ~]#mkfs.xfs -i size=512/dev/sdb1

[[email protected] ~]#mkdir -p /export/sdb1&& mount /dev/sdb1 /export/sdb1#建立掛載目錄和掛載

[[email protected] ~]#echo"/dev/sdb1 /export/sdb1 xfs defaults 0 0">> /etc/fstab#

加入開機啟動

[[email protected]

~]#mount -a && mount#掛載測試和掛載及結果檢視

3.安裝glusterfs

[[email protected] ~]#yum installcentos-release-gluster#安裝源

[[email protected] ~]#yuminstall glusterfs-server#安裝glusterfs

4.注意關掉防火牆

[[email protected] ~]#systemctlstop firewalld#臨時關閉，重啟還會啟動

[[email protected]e1 ~]#systemctldisable firewalld#永久關閉

 

5.啟動glusterfs

[[email protected] ~]#systemctl start glusterd#啟動gluster

[[email protected] ~]#systemctl enable glusterd#加入開機啟動

 

6.組建叢集

 

Replace nodename with hostname of theother server in the cluster, or IP address if you don’t have DNS or /etc/hostsentries

在/etc/hosts裡邊做設定或者在DNS，建議兩者都做，這裡沒有DNS解析伺服器，只在/etc/hosts裡邊坐繫結，注意不要忘記該機器的名字。

 

For Example：

 

[[email protected] gluster]#cat /etc/hosts

127.0.0.1localhostlocalhost.localdomain localhost4 localhost4.localdomain4

::1localhostlocalhost.localdomain localhost6 localhost6.localdomain6

192.168.238.129 node1

192.168.238.130 node2

192.168.238.133 node3

192.168.238.132 node4

192.168.238.134 node5

#在node1節點執行peer probe命令

[[email protected] ~]#glusterpeer probe node2#注意，如果探測不到請確認host解析做了沒有，防火牆是否關閉

#檢視剛剛加入的節點

[[email protected] ~]#glusterpeer status

[[email protected] ~]#gluster peer probe node2

peer probe: success.

[[email protected] ~]#gluster peer status

Number of Peers: 1

Configure your Gluster volume

刪除加入的節點（如果想刪除節點，執行該命令）

 

[[email protected] ~]#glusterpeer detach node2

在倆個主機執行

 

[[email protected] ~]#mkdir -p/export/sdb1/brick

[[email protected]~]#gluster volume create testvol replica 2 transport tcpnode1:/export/sdb1/brick node2:/export/sdb1/brick

volume create: testvol: success: please start the volume to accessdata

[[email protected] log]#gluster peer status

Number of Peers: 1

Hostname: node2

Uuid: 61fe987a-99ff-419d-8018-90603ea16fe7

State: Peer in Cluster (Connected)

[[email protected] log]#glustervolume info

Volume Name: testvol

Type: Replicate

Volume ID: bc637d83-0273-4373-9d00-d794a3a3d2e7

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 =2

Transport-type: tcp

Bricks:

Brick1: node1:/export/sdb1/brick

Brick2: node2:/export/sdb1/brick

Options Reconfigured:

transport.address-family: inet

nfs.disable: on

[[email protected] ~]#gluster volume start testvol#啟動volume

volume start: testvol: success

[[email protected] gluster]#gluster volume info#檢視Volume資訊

 

Volume Name: testvol

Type: Replicate

Volume ID: bc637d83-0273-4373-9d00-d794a3a3d2e7

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: node1:/export/sdb1/bricks

Brick2: node2:/export/sdb1/brick

Options Reconfigured:

transport.address-family: inet

nfs.disable: on

[[email protected] ~]#gluster volume status

 

Status of volume: testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Brick node1:/export/sdb1/brick491520Y10774

Brick node2:/export/sdb1/brickN/AN/ANN/A

Self-heal Daemon on localhostN/AN/AY998

Self-heal Daemon on node1N/AN/AY10794

Task Status of Volume testvol

------------------------------------------------------------------------------

There are no active volume tasks

注意點：上邊的環境為每個伺服器單塊硬碟的結構，如果是多塊硬碟的結構，應該為分別格式化，掛載，具體過程如下：

mkfs.xfs -i size=512 /dev/sdc1

mkdir /export/sdb1

mount /dev/sdc1 /export/sdc1

echo "/dev/sdc1 /export/sdc1 xfs defaults0 0">> /etc/fstab

mkdir -p /export/sdb1/brick#所有的節點執行

node1上執行：

每臺主機一塊硬碟執行的命令（上邊我們配置的模式）：

# gluster volume create testvol replica 2transport tcp node1:/export/sdb1/brick node2:/export/sdb1/brick

每臺主機兩塊硬碟執行的命令：

# gluster volume create testvol replica 2transport tcp node1:/export/sdb1/brick node2:/export/sdb1/bricknode1:/export/sdc1/bricknode2:/export/sdc1/brick

照理推即。

掛載測試

[[email protected] ~]# mkdir/mnt/gluster/

[[email protected] ~]#mount -tglusterfs node1:/testvol /mnt/gluster/

 

Expanding

Volumes(擴容)

To expand a volume

 

前提條件開始有node1和node2節點，後加node3和node4節點的配置同上，必須一次加一對主機，注意修改叢集裡各個主機的/etc/hosts檔案，新加入node節點

 

1）On the first server in the cluster,probe the server to which you want to add the new brick using the followingcommand:

# gluster peerprobe#執行的命令如下：

[[email protected] ~]#gluster peer probe node3

peer probe: success.

[[email protected] ~]#gluster peer probe node4

peer probe: success.

2）Add the brick using the followingcommand:

# gluster volumeadd-brick

[[email protected] ~]#gluster volume add-bricktestvol node3:/export/sdb1/brick node4:/export/sdb1/brick

volume add-brick:success#顯示成功

3）Check the volume information using the following command:（檢查命令如下）

[[email protected] ~]#gluster volume info

Volume Name: testvol

Type: Distributed-Replicate

Volume ID:09363405-1c7c-4eb1-b815-b97822c1f274

Status: Started

Snapshot Count: 0

Number of Bricks: 2 x2 = 4

Transport-type: tcp

Bricks:

Brick1:node1:/export/sdb1/brick

Brick2:node2:/export/sdb1/brick

Brick3:node3:/export/sdb1/brick

Brick4:node4:/export/sdb1/brick

Options Reconfigured:

transport.address-family:inet

nfs.disable: on

Rebalance the volumeto ensure that all files are distributed to the new brick.

You can use therebalance command as described in Rebalancing Volumes

[[email protected] ~]#gluster volume rebalancetestvol start

volume rebalance:testvol: success: Rebalance on testvol has been started successfully. Userebalance status command to check status of the rebalance process.

ID: c9d052e8-2b6c-40b0-8d77-52290bcdb61

To shrink a volume（線上收縮）

本例的操作

[[email protected] gluster]#gluster volume remove-bricktestvol node3:/export/sdb1/brick node4:/export/sdb1/brick force

Removing brick(s) can result in data loss. Do you want to Continue? (y/n)y

volume remove-brick commit force: success

詳見後邊的文件

1.Remove the brick using the following command:

# gluster volume remove-brickstart

For example, to remove server2:/exp2:

# gluster volume remove-brick test-volume server2:/exp2 force

Removingbrick(s) canresultindata loss.Doyou want toContinue? (y/n)

2.Enter "y" to confirm the operation. The commanddisplays the following message indicating that the remove brick operation issuccessfully started:

3.RemoveBrick successful

4.(Optional) View the status of the remove brick operation usingthe following command:

# gluster volume remove-brickstatus

For example, to view the status of removebrick operation on server2:/exp2 brick:

#glustervolumeremove-bricktest-volumeserver2:/exp2status

NodeRebalanced-filessizescannedstatus

-----------------------------------------------

617c923e-6450-4065-8e33-865e28d9428f34340162inprogress

5.Check the volume information using the following command:

# gluster volume info

The command displays information similar tothe following:

# gluster volume info

VolumeName:test-volume

Type:Distribute

Status:Started

Number ofBricks:3

Bricks:

Brick1:server1:/exp1

Brick3:server3:/exp3

Brick4:server4:/exp4

6.Rebalance the volume to ensure that all files are distributed tothe new brick.

You can use the rebalance command as described inRebalancing Volumes

 

主機上硬碟故障的處理方式

 

方法一：如果本機還有空閒的備用盤的處理方式

 

故障問題：

 

[[email protected] ~]#gluster volume status

Status of volume:testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Bricknode1:/export/sdb1/brick491520Y2684

Brick node2:/export/sdb1/brickN/AN/ANN/A#sdb1顯示不線上，故障

Bricknode1:/export/sdc1/brick491530Y2703

Bricknode2:/export/sdc1/brick491530Y2704

Bricknode3:/export/sdb1/brick491520Y2197

Bricknode4:/export/sdb1/brick491520Y2207

Bricknode3:/export/sdc1/brick491530Y2216

Bricknode4:/export/sdc1/brick491530Y2226

Self-heal Daemon onlocalhostN/AN/AY1393

Self-heal Daemon onnode1N/AN/AY3090

Self-heal Daemon onnode4N/AN/AY2246

Self-heal Daemon onnode3N/AN/AY2236

Task Status of Volumetestvol

------------------------------------------------------------------------------

Task: Rebalance

ID:8b3a04a0-0449-4424-a458-29f602571ea2

Status: completed

 

從上方看到Brick node2:/export/sdb1/brick不線上，出現了問題

 

解決：

1.建立新的資料目錄，將備用的硬碟格式化，掛載到系統中去（故障主機上執行）

[[email protected] ~]#mkfs.xfs -i size=512/dev/sdd1#格式化

[[email protected] ~]#mkdir /export/sdd1/brick-p#建立相關的目錄

[[email protected] ~]#mount /dev/sdd1 /export/sdd1#掛載

[[email protected] ~]#echo "/dev/sdd1 /export/sdd1 xfs defaults 00">> /etc/fstab#加入開機啟動

2.查詢故障點的目錄的擴充套件屬性（正常主機執行）

[[email protected] brick]#getfattr -d -m. -e hex /export/sdb1/brick/

getfattr: Removingleading '/' from absolute path names

# file:export/sdb1/brick/

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.testvol-client-1=0x000000000000000000000000

trusted.gfid=0x00000000000000000000000000000001

trusted.glusterfs.dht=0x0000000100000000000000003ffffffe

trusted.glusterfs.dht.commithash=0x3000

trusted.glusterfs.volume-id=0xe107222fa1134606a9a7fcb16e4c0709

3.掛載卷並觸發自愈（故障主機執行）

 

[[email protected] ~]#mount -t glusterfs node2:/testvol /mnt#掛載點隨便，不重複就可以，node2:/testvol為之前生成的卷

[[email protected] ~]#mkdir /mnt/test#新建一個卷中不存在的目錄並刪除，根據你的掛載點的位置變換執行

[[email protected] ~]#rmdir /mnt/test#刪除剛才新建立的目錄

[[email protected] ~]#setfattr -n trusted.non-existent-key -v abc /mnt#設定擴充套件屬性觸發自愈

[[email protected] ~]#setfattr-x trusted.non-existent-key /mnt#設定擴充套件屬性觸發自愈

4.檢查當前節點是否掛起

正常的主機執行

[[email protected] gluster]#getfattr -d -m. -e hex /export/sdb1/brick/#

/export/sdb1/brick/你建立brick的位置

getfattr: Removing leading '/' from absolute path names

# file: export/sdb1/brick/

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.testvol-client-1=0x000000000000000400000004<<---- xattrs are marked

from source brick node1:/export/sdb1/brick--->>

trusted.gfid=0x00000000000000000000000000000001

trusted.glusterfs.dht=0x0000000100000000000000003ffffffe

trusted.glusterfs.dht.commithash=0x3334343336363233303800

trusted.glusterfs.volume-id=0xe107222fa1134606a9a7fcb16e4c0709

故障主機執行（正常也可以）：

[[email protected] gluster]#gluster volume heal testvol info#檢視你testvol的資訊

Brick node1:/export/sdb1/brick

/

Status: Connected

Number of entries: 1

Brick node2:/export/sdb1/brick

Status: Transport endpoint is not connected

Number of entries: -#狀態顯示傳輸端點未連線

Brick node1:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node2:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node3:/export/sdb1/brick

Status: Connected

Number of entries: 0

Brick node4:/export/sdb1/brick

Status: Connected

Number of entries: 0

Brick node3:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node4:/export/sdc1/brick

Status: Connected

Number of entries: 0

5.使用強制提交完成修復操作

故障機執行

[[email protected] ~]#gluster volume replace-brick testvol node2:/export/sdb1/bricknode2:/export/sdd1/brick commit force

volume replace-brick: success: replace-brick commit force operationsuccessful#提示成功

[[email protected] ~]#gluster volume status

Status of volume: testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Brick node1:/export/sdb1/brick491520Y2684

Brick node2:/export/sdd1/brick491540Y10298#線上盤已經是sdd1，已經吧sdb1替換了

Brick node1:/export/sdc1/brick491530Y2703

Brick node2:/export/sdc1/brick491530Y2704

Brick node3:/export/sdb1/brick491520Y2197

Brick node4:/export/sdb1/brick491520Y2207

Brick node3:/export/sdc1/brick491530Y2216

Brick node4:/export/sdc1/brick491530Y2226

Self-heal Daemon on localhostN/AN/AY10307

Self-heal Daemon on node3N/AN/AY9728

Self-heal Daemon on node1N/AN/AY3284

Self-heal Daemon on node4N/AN/AY9736

Task Status of Volume testvol

------------------------------------------------------------------------------

Task:Rebalance

ID:8b3a04a0-0449-4424-a458-29f602571ea2

Status: notstarted

正常主機執行

[[email protected] gluster]#getfattr -d -m. -e hex /export/sdb1/brick/

getfattr: Removing leading '/' from absolute path names

# file: export/sdb1/brick/

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.testvol-client-1=0x000000000000000000000000<<---- Pending changelogs are cleared.

trusted.gfid=0x00000000000000000000000000000001

trusted.glusterfs.dht=0x0000000100000000000000003ffffffe

trusted.glusterfs.dht.commithash=0x3334343336363233303800

trusted.glusterfs.volume-id=0xe107222fa1134606a9a7fcb16e4c0709

[[email protected] ~]#gluster volume heal testvol info

Brick node1:/export/sdb1/brick

Status: Connected

Number of entries: 0

Brick node2:/export/sdd1/brick

Status: Connected

Number of entries: 0

Brick node1:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node2:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node3:/export/sdb1/brick

Status: Connected

Number of entries: 0

Brick node4:/export/sdb1/brick

Status: Connected

Number of entries: 0

Brick node3:/export/sdc1/brick

Status: Connected

Number of entries: 0

Brick node4:/export/sdc1/brick

Status: Connected

Number of entries: 0

也可以使用（上邊的為官方文件的修復過程，用下邊的也可以搞定）：

當某塊磁碟損壞後，需要換一塊新的磁盤迴去，這時叢集中恰好還預留了備用磁碟，因此用備用磁碟替換損壞的磁碟，命令如下兩條命令這個就可以搞定

掛載磁碟參考上邊（掛載點/export/sdd1）

[[email protected] ~]#gluster volume replace-brick voltest node2:/export/sdb1/bricknode2:/export/sdd1/brickcommit（前邊為故障盤，後邊為恢復盤）

方式二：跨主機的同步

假設node2的sdb1有問題

前提：加入新主機node5（兩塊硬碟，一塊系統盤，一塊掛在盤），node5磁碟格式化，掛載和gluster的安裝過程等準備工作同上，注意修改叢集裡各個主機的/etc/hosts檔案，新加入node節點：

將node5加入信任池

[[email protected] brick]#gluster peer probe node5

peer probe: success.

掛載磁碟

[[email protected] ~]#mkdir -p /export/sdb1 && mount /dev/sdb1 /export/sdb1

[[email protected]ode5 ~]#echo "/dev/sdb1 /export/sdb1 xfs defaults 0 0">> /etc/fstab

[[email protected] ~]#mount -a && mount

執行下邊的命令：

[[email protected] ~]#gluster volume replace-brick testvol node2:/export/sdb1/bricknode5:/export/sdb1/brickcommit force

volume replace-brick: success: replace-brick commit force operationsuccessful

 

替換後可以繼續使用，也可以在sdb1硬碟更換後後，資料倒回，命令如下

 

[[email protected] ~]#gluster volume replace-brick testvol node5:/export/sdb1/bricknode2:/export/sdb1/brick commit force

volume replace-brick: success: replace-brick commit force operationsuccessful

替換之前的狀態：

[[email protected] brick]# gluster volume status

Status of volume: testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Brick node1:/export/sdb1/brick491520Y2085

Brick

node5:/export/sdb1/brick491520Y18229

Brick node1:/export/sdc1/brick491530Y2076

Brick node2:/export/sdc1/brick491530Y2131

Brick node3:/export/sdb1/brick491520Y2197

Brick node4:/export/sdb1/brick491520Y2207

Brick node3:/export/sdc1/brick491530Y2216

Brick node4:/export/sdc1/brick491530Y2226

Self-heal Daemon on localhostN/AN/AY10565

Self-heal Daemon on node2N/AN/AY2265

Self-heal Daemon on node3N/AN/AY10416

Self-heal Daemon on node4N/AN/AY10400

Self-heal Daemon on node5N/AN/AY18238

Task Status of Volume testvol

------------------------------------------------------------------------------

Task:Rebalance

ID:8b3a04a0-0449-4424-a458-29f602571ea2

Status: notstarted

替換之後的狀態：

[[email protected] gluster]# gluster volume status

Status of volume: testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Brick node1:/export/sdb1/brick491520Y2085

Brick

node2:/export/sdb1/brick（過來了）491530Y10208

Brick node1:/export/sdc1/brick491530Y2076

Brick node2:/export/sdc1/brick491520Y3474

Brick node3:/export/sdb1/brick491520Y2197

Brick node4:/export/sdb1/brick491520Y2207

Brick node3:/export/sdc1/brick491530Y2216

Brick node4:/export/sdc1/brick491530Y2226

Self-heal Daemon on localhostN/AN/AY10684

Self-heal Daemon on node3N/AN/AY10498

Self-heal Daemon on node5N/AN/AY10075

Self-heal Daemon on node4N/AN/AY10488

Self-heal Daemon on node2N/AN/AY10201

Task Status of Volume testvol

------------------------------------------------------------------------------

Task:Rebalance

ID:8b3a04a0-0449-4424-a458-29f602571ea2

Status: notstarted

 

資料的平衡

 

一般平衡資料有如下兩種場景：

Fix Layout：重新定位layout，原來的layout不變，（新）資料寫入新的節點

Fix Layout and Migrate Data:重新定位layout的修改，並且遷移已經存在的資料

要注意的是當有新的節點加入的時候，必須做fix layout，不然新寫入的資料還是寫入到老的節點上去

 

1.To rebalance a volume to fix layout changes（fix

layout）

Start the rebalance operation on any one ofthe server using the following command:

# gluster volume rebalance fix-layout start

For example:

# gluster volume rebalance test-volumefix-layout start

Starting rebalance on volume test-volumehas been successful

本機的命令如下：

[[email protected]

gluster]#gluster volume rebalance testvolfix-layout start

volume rebalance: testvol: success:Rebalance on testvol has been started successfully. Use rebalance statuscommand to check status of the rebalance process.

ID: 0ea5aa16-b349-44ca-a51b-d5fcf47e1272

[[email protected]

gluster]#gluster volume rebalance testvol status

Nodestatusrun time in h:m:s

--------------------------------

localhostfix-layout completed0:0:0

node2fix-layout completed0:0:0

node3fix-layout completed0:0:0

node4fix-layout completed0:0:0

volume rebalance: testvol: success

2.To rebalance a volume to fix

layout and migrate the existing data（Fix Layout

and Migrate Data）

 

Start the rebalance operation on any one ofthe server using the following command:

# gluster volume rebalance start

For example:

# gluster volume rebalance test-volumestart

Starting rebalancing on volume test-volumehas been successful

Start the migration operation forcefully onany one of the servers using the following command:

# gluster volume rebalance start force

For example:

# gluster volume rebalance test-volumestart force

Starting rebalancing on volume test-volumehas been successful

本機操作命令：

[[email protected]

gluster]#gluster volume rebalance testvol start

volume rebalance: testvol: success:Rebalance on testvol has been started successfully. Use rebalance statuscommand to check status of the rebalance process.

ID: 2a47d454-fdc3-4d95-81ac-6981577d26e9

[[email protected]

gluster]#gluster volume rebalance testvol status

NodeRebalanced-filessizescannedfailuresskippedstatusrun time in h:m:s

------------------------------------------------------------------------------------------

localhost00Bytes3701completed0:00:00

node200Bytes000completed0:00:00

node300Bytes2700completed0:00:00

node400Bytes000completed0:00:00

volume rebalance: testvol: success

2.Stopping Rebalance Operation（需要的話）

You can stop the rebalance operation, asneeded.

Stop the rebalance operation using the followingcommand:

# gluster volume rebalancestop

For example:

# gluster volume rebalance test-volume stop

NodeRebalanced-filessizescannedstatus

-----------------------------------------------

617c923e-6450-4065-8e33-865e28d9428f59590244stopped

Stopped rebalance process on volumetest-volume

Stopping

Volumes

Stop the volume using the followingcommand:

# gluster volume stop

For example, to stop test-volume:

# gluster volume stop test-volume

Stopping volume will make its datainaccessible. Do you want to continue? (y/n)

Enter y to confirm the operation. Theoutput of the command displays the following:

Stopping volume test-volume has been successful

Deleting

Volumes

Delete the volume using the followingcommand:

# gluster volume delete

For example, to delete test-volume:

# gluster volume delete test-volume

Deleting volume will erase all informationabout the volume. Do you want to continue? (y/n)

Enter y to confirm the operation. Thecommand displays the following:

Deleting volume test-volume has beensuccessful

Triggering Self-Heal on Replicate

In replicate module, previously you had tomanually trigger a self-heal when a brick goes offline and comes back online,to bring all the replicas in sync. Now the pro-active self-heal daemon runs inthe background, diagnoses issues and automatically initiates self-healing every10 minutes on the files which requireshealing.

You can view the list of files that needhealing,the list of files which are currently/previouslyhealed,list of files which are in split-brain state, and you can manually triggerself-heal on the entire volume or only on the files which needhealing.

·Trigger self-heal only on the files which requireshealing:

# gluster volume heal

For example, to trigger self-heal on fileswhich requireshealingof test-volume:

# gluster volume heal test-volume

Heal operation onvolumetest-volumehas been successful

·Trigger self-heal on all the files of a volume:

# gluster volume heal full

For example, to trigger self-heal on all thefiles of of test-volume:

# gluster volume heal test-volume full

Heal operation onvolumetest-volumehas been successful

·View the list of files that needshealing:

# gluster volume heal info

For example, to view the list of files ontest-volume that needshealing:

# gluster volume heal test-volume info

Brick server1:/gfs/test-volume_0

Numberofentries:0

Brick server2:/gfs/test-volume_1

Numberofentries:101

/95.txt

/32.txt

/66.txt

/35.txt

/18.txt

/26.txt

/47.txt

/55.txt

/85.txt

...

·View the list of files that are self-healed:

# gluster volume heal info

healed

For example, to view the list of files ontest-volume that are self-healed:

# gluster volume heal test-volume info healed

BrickServer1:/gfs/test-volume_0

Numberofentries:0

BrickServer2:/gfs/test-volume_1

Numberofentries:69

/99.txt

/93.txt

/76.txt

/11.txt

/27.txt

/64.txt

/80.txt

/19.txt

/41.txt

/29.txt

/37.txt

/46.txt

...

·View the list of files of a particular volume on which theself-heal failed:

# gluster volume heal info

failed

For example, to view the list of files oftest-volume that are not self-healed:

# gluster volume heal test-volume info failed

BrickServer1:/gfs/test-volume_0

Numberofentries:0

BrickServer2:/gfs/test-volume_3

Numberofentries:72

/90.txt

/95.txt

/77.txt

/71.txt

/87.txt

/24.txt

...

·View the list of files of a particular volume which are insplit-brain state:

# gluster volume heal info split-brain

For example, to view the list of files oftest-volume which are in split-brain state:

# gluster volume heal test-volume info split-brain

BrickServer1:/gfs/test-volume_2

Numberofentries:12

/83.txt

/28.txt

/69.txt

...

BrickServer2:/gfs/test-volume_3

Numberofentries:12

/83.txt

/28.txt

/69.txt

...

問題處理：

1.提示State: Peer in Cluster (Disconnected)

[[email protected] gluster]# gluster peer status

Number of Peers: 1

Hostname: node2

Uuid: 61fe987a-99ff-419d-8018-90603ea16fe7

State: Peer in Cluster(Disconnected)

解決方式：檢視防火牆狀態和/etc/hosts檔案，防火牆也可以通過規則放行，不過還是關掉好，效能考慮

 

2.重建volume出錯

 

[[email protected]

sdb1]#gluster volume start testvol

volume start: testvol: failed: Failed toget extended attribute trusted.glusterfs.volume-id for brick dir /export/sdb1/brick.Reason : No data available#故障 重建volume出錯

處理方式： 檢視volume的資訊，刪除掉volume，清空資訊，重新建立volume

[[email protected]

sdb1]#gluster volume info

Volume Name: testvol

Type: Replicate

Volume ID:57a60503-c5ae-4671-b213-6f2a2f913615

Status: Created

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: node1:/export/sdb1/brick

Brick2: node2:/export/sdb1/brick

Options Reconfigured:

transport.address-family: inet

nfs.disable: on

[[email protected]

sdb1]#gluster volume delete testvol

Deleting volume will erase all informationabout the volume. Do you want to continue? (y/n) y

volume delete: testvol: success

[[email protected]

sdb1]#setfattr -x trusted.glusterfs.volume-id/export/sdb1/brick && setfattr -x trusted.gfid /export/sdb1/brick&& rm -rf /export/sdb1/brick/..glusterfs

setfattr: /export/sdb1/brick: No suchattribute

[[email protected]

sdb1]#setfattr -x trusted.glusterfs.volume-id/export/sdb1/brick && setfattr -x trusted.gfid /export/sdb1/brick&& rm -rf /export/sdb1/brick/..glusterfs

setfattr: /export/sdb1/brick: No suchattribute

[[email protected] sdb1]# gluster volume createtestvol replica 2 transport tcp node1:/export/sdb1/bricknode2:/export/sdb1/brick

volume create: testvol: success: pleasestart the volume to access data

[[email protected]

sdb1]#gluster volume start testvol

volume start: testvol: success

[[email protected]

sdb1]#gluster volume status

Status of volume: testvol

Gluster processTCP PortRDMA PortOnlinePid

------------------------------------------------------------------------------

Brick node1:/export/sdb1/brick491520Y2429

Brick node2:/export/sdb1/brick491520Y2211

Self-heal Daemon on localhostN/AN/AY2449

Self-heal Daemon on node2N/AN/AY2231

Task Status of Volume testvol

------------------------------------------------------------------------------

There are no active volume tasks

可以參照下邊的兩篇文章：

glusterfs volume create: testvol: failed: /data/brick1 ora prefix of it is already part of a volume

在建立volume的時候提示如下報錯

[[email protected] ~]# gluster volumecreate testvol 192.168.11.139:/data/brick1 192.168.11.140:/data/brick2forcevolume create: testvol: failed: /data/brick1 or a prefix of it is alreadypart of a volume

找到了國外的一篇部落格，據這篇部落格講。從glusterfs3.3開始，有一個新的變化就是會檢測這個目錄是否已經加入了volume。這就導致了很多gluster支援的問題。

假如你移除了一個volume的brick並且繼續使用這個volume，然後重新新增之前的這個brick，這時你的檔案狀態就會改變。就會造成一系列的問題。它們中的大多數會引起資料丟失。

假如你重新使用一個brick的話，請確保你知道自己在做什麼

解決辦法就是：

setfattr -x trusted.glusterfs.volume-id $brick_path

setfattr -x trusted.gfid $brick_path

rm -rf $brick_path/.glusterfs

1

2

3

4

5

6

7[[email protected] data]# setfattr -x trusted.glusterfs.volume-id /data/ctdb/

[[email protected] data]# setfattr -x trusted.gfid /data/ctdb/

[[email protected] data]# rm -rf /data/ctdb/.

./          ../         .glusterfs/

[[email protected]ster-node1 data]# rm -rf /data/ctdb/.glusterfs

[[email protected] data]# service glusterd restart

Starting glusterd:                                         [確定]

不要擔心提示說 屬性不存在，只要它不存在，那它就處於健康的狀態

最後，重啟一下glusterfs來確保它並沒有“記起來”舊的bricks

有些可能翻譯的不到位，可以檢視原文

https://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/

#glusterfs

error

）

GlusterFS: {path} or a prefix of it is already part of avolume

Starting

with GlusterFS 3.3, one change has been the check to see if a directory (or any

of it's ancestors) is already part of a volume. This is causing many support

questions in#gluster.

Thiswas implemented because if you remove a brick from a volume and continue to usethe volume, you can get file into a state where re-adding a former brick cancause all sort of problems, many of which can result in data loss.

Ifyou're going to reuse a brick, make sure you know what you're doing.

The Solution

Forthe directory (or any parent directories) that was formerly part of a volume, simply:

setfattr -x trusted.glusterfs.volume-id $brick_path

setfattr -x trusted.gfid $brick_path

rm -rf $brick_path/.glusterfs

Don'tworry if it says that the attribute does not exist. As long as it doesn'texist, you're in good shape.

Finally,restart glusterd to ensure it's not "remembering" the old bricks.

See

thebugzilla entryfor more details and see Jeff Darcy's article for more

information about how GlusterFS usesextended attributes.

[python]view plaincopy

1.[[email protected] glusterfs]# gluster volume create hello  replica 3  test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs

2.volume create: hello: success: please start the volume to access data

3.[[email protected] glusterfs]# gluster volume delete hello

4.Deleting volume will erase all information about the volume. Do you want tocontinue? (y/n) y

5.volume delete: hello: success

6.[[email protected] glusterfs]# gluster volume create gfs  replica 3  test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs

7.volume create: gfs: failed: Staging failed on test.144.Error: /data0/glusterfsisalready part of a volume

8.Staging failed on test.146.Error: /data0/glusterfsisalready part of a volume

9.Staging failed on test.145.Error: /data0/glusterfsisalready part of a volume

10.[[email protected] glusterfs]# ssh test.144 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'

11.setfattr: /data0/glusterfs/: No such attribute

12.[[email protected] glusterfs]# ssh test.145 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'

13.setfattr: /data0/glusterfs/: No such attribute

14.[[email protected] glusterfs]# ssh test.146 'setfattr -x trusted.glusterfs.volume-id /data0/glusterfs/ && setfattr -x trusted.gfid /data0/glusterfs/ && rm -rf /data0/glusterfs/..glusterfs'

15.setfattr: /data0/glusterfs/: No such attribute

16.[[email protected] glusterfs]#

17.[[email protected] glusterfs]# gluster volume create gfs  replica 3  test.144:/data0/glusterfs test.145:/data0/glusterfs test.146:/data0/glusterfs

18.volume create: gfs: success: please start the volume to access data

19.[[email protected] glusterfs]#

檢視node2防火牆規則是否適當，網路是否通順

3.之前brick在叢集裡邊，後來踢出來，再加報錯處理

[[email protected]

brick]#gluster volume add-brick testvolnode3:/export/sdb1/brick node4:/export/sdb1/brick node3:/export/sdc1/bricknode4:/export/sdc1/brick

volume add-brick: failed: Pre Validationfailed on node3. /export/sdb1/brick is already part of a volume

Pre Validation failed on node4./export/sdb1/brick is already part of a volume

[[email protected]

brick]#gluster volume add-brick testvolnode3:/export/sdb1/brick node4:/export/sdb1/brick node3:/export/sdc1/bricknode4:/export/sdc1/brick

volume add-brick: success

官方安裝手冊

Step 1 – Have atleast two nodes

·Fedora 22 (or later) on two nodes named"server1" and "server2"

·A working network connection

·At least two virtual disks, one for the OS installation,and one to be used to serve GlusterFS storage (sdb). This will emulate a realworld deployment, where you would want to separate GlusterFS storage from theOS install.

·Note: GlusterFS stores its dynamically generatedconfiguration files at /var/lib/glusterd. If at any point in time GlusterFS isunable to write to these files (for example, when the backing filesystem isfull), it will at minimum cause erratic behavior for your system; or worse,take your system offline completely. It is advisable to create separatepartitions for directories such as /var/log to ensure this does not happen.

Step 2 - Formatand mount the bricks

(on both nodes): Note: These examplesare going to assume the brick is going to reside on /dev/sdb1.

mkfs.xfs -isize=512/dev/sdb1

mkdir -p/data/brick1

echo'/dev/sdb1 /data/brick1 xfs defaults 1 2'>> /etc/fstab

mount -a&& mount

You should now see sdb1 mounted at/data/brick1

Step 3 -Installing GlusterFS

(on both servers) Install the software

yuminstallglusterfs-server

Start the GlusterFS management daemon:

service glusterd start

service glusterd status

glusterd.service -LSB:glusterfs server

Loaded: loaded (/etc/rc.d/init.d/glusterd)

Active: active (running) since Mon,13Aug201213:02:11-0700;2s ago

Process:19254ExecStart=/etc/rc.d/init.d/glusterd start (code=exited, status=0/SUCCESS)

CGroup: name=systemd:/system/glusterd.service

├19260/usr/sbin/glusterd -p /run/glusterd.pid

├19304/usr/sbin/glusterfsd --xlator-option georep-server.listen-port=24009-s localhost...

└19309/usr/sbin/glusterfs -f /var/lib/glusterd/nfs/nfs-server.vol -p/var/lib/glusterd/...

Step 4 -Configure the trusted pool

From "server1"

glusterpeer probe server2

Note: When using hostnames, the firstserver needs to be probed fromoneother server to setits hostname.

From "server2"

glusterpeer probe server1

Note: Once this pool has beenestablished, only trusted members may probe new servers into the pool. A newserver cannot probe the pool, it must be probed from the pool.

Step 5 - Set upa GlusterFS volume

On both server1 and server2:

mkdir -p /data/brick1/gv0

From any single server:

gluster volumecreategv0 replica2server1:/data/brick1/gv0 server2:/data/brick1/gv0

gluster volumestartgv0

Confirm that the volume shows"Started":

glustervolumeinfo

Note: If the volume is not started,clues as to what went wrong will be in log files under /var/log/glusterfs onone or both of the servers - usually in etc-glusterfs-glusterd.vol.log

Step 6 - Testingthe GlusterFS volume

For this step, we will use one of theservers to mount the volume. Typically, you would do this from an externalmachine, known as a "client". Since using this method would requireadditional packages to be installed on the client machine, we will use one ofthe servers as a simple place to test first, as if it were that"client".

mount -t glusterfs server1:/gv0 /mnt

foriin`seq -w 1 100`;docp -rp /var/log/messages /mnt/copy-test-$i; done

First, check the mount point:

ls-lA/mnt | wc -l

You should see 100 files returned. Next,check the GlusterFS mount points on each server:

ls -lA /data/brick1/gv0

You should see 100 files on each serverusing the method we listed here. Without replication, in a distribute onlyvolume (not detailed here), you should see about 50 files on each one.

1.Configure Firewall（最好關閉防火牆）

 

For the Gluster to communicate within a cluster either the firewallshave to be turned off or enable communication for each server.

iptables -IINPUT -p all -s `` -j ACCEPT

2. Configurethe trusted pool

 

Remember that the trusted pool is the term used to define a cluster

of nodes in Gluster. Choose a server to be your “primary” server. This is just

to keep things simple, you will generally want to run all commands from this

tutorial. Keep in mind, running many Gluster specific commands (like gluster

volume create) on one server in the cluster will execute the same command on

all other servers.（只需要在一臺機器上執行）

 

3. Replace

nodename with hostname of the other server in the cluster, or IP address if you

don’t have DNS or /etc/hosts entries. Let say we want to connect to node02:（在DNS和/etc/hosts中都要設定）

gluster peerprobe node02

Notice that running gluster peer statusfrom the second node shows that the first node has already been added.

 

4. Partition

the disk（磁碟處理）

 

4.1Assuming you have a emptydisk at /dev/sdb:

fdisk /dev/sdb

4.2 And then create a single XFS partition using fdisk

Format thepartition

mkfs.xfs -i size=512 /dev/sdb1

4.3 Add an entry to /etc/fstab

echo "/dev/sdb1/export/sdb1 xfs defaults 0 0">> /etc/fstab

4.4Mount the partition as aGluster "brick"

mkdir -p /export/sdb1 && mount -a&& mkdir -p /export/sdb1/brick

Set up a Gluster volume

 

The most basic Gluster volume type is a“Distribute only” volume (also referred to as a “pure DHT” volume if you wantto impress the folks at the water cooler). This type of volume simplydistributes the data evenly across the available bricks in a volume. So, if Iwrite 100 files, on average, fifty will end up on one server, and fifty willend up on another. This is faster than a “replicated” volume, but isn’t aspopular since it doesn’t give you two of the most sought after features ofGluster — multiple copies of the data, and automatic failover if something goeswrong.

 

1.To set up a replicated volume:（設定複製卷）

 

gluster volumecreate gv0 replica 2 node01.mydomain.net:/export/sdb1/bricknode02.mydomain.net:/export/sdb1/brick

 

Breakingthis down into pieces:

 

the first part says to create a glustervolume named gv0 (the name is arbitrary, gv0 was chosen simply because it’sless typing than gluster_volume_0).

make the volume a replica volume

keep a copy of the data on at least 2bricks at any given time. Since we only have two bricks total, this means eachserver will house a copy of the data.

we specify which nodes to use, and whichbricks on those nodes. The order here is important when you have more bricks.

It is possible (as of the most currentrelease as of this writing, Gluster 3.3) to specify the bricks in a such a waythat you would make both copies of the data reside on a single node. This wouldmake for an embarrassing explanation to your boss when your bulletproof,completely redundant, always on super cluster comes to a grinding halt when asingle point of failure occurs.

 

2.Now, we can check to make sure things are working as expected:

 

# gluster volume info

 

And you should see results similar to thefollowing:

Volume Name: gv0

Type: Replicate

Volume ID: 8bc3e96b-a1b6-457d-8f7a-a91d1d4dc019

Status: Created

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: node01.yourdomain.net:/export/sdb1/brick

Brick2: node02.yourdomain.net:/export/sdb1/brick

This shows us essentially what we justspecified during the volume creation. The one this to mention is the Status. Astatus of Created means that the volume has been created, but hasn’t yet beenstarted, which would cause any attempt to mount the volume fail.

 

3.Now, we should start the volume.

# gluster volumestart gv0

其他資料：

支援的資料型別：

Gluster does not support so called“structured data”, meaning live, SQL databases. Of course, using Gluster tobackup and restore the database would be fine - Gluster is traditionally betterwhen usingfile sizes at of least 16KB(with a sweet spot around 128KB or so).

 

結構化資料不支援，但是可以用來備份和重新儲存資料