1. 程式人生 > >Redis之——Redis叢集的擴充套件與測試

Redis之——Redis叢集的擴充套件與測試

一、簡單介紹

二、 安裝新的 Redis 節點,將用於擴充套件性測試

1、安裝 Redis3

在 192.168.1.117 虛擬機器上以同樣的方式安裝 Redis3, 並啟動兩個例項, 規劃如下:

主機名 IP 服務埠(預設6379) 叢集埠(服務埠值+10000) 主/從
liuyazhuang07 192.168.1.117 7117 17117 Master
liuyazhuang07 192.168.1.117 7118 17118 Slave
按規劃: 在 192.168.1.117 的防火牆中開啟相應的埠
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7117 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17117 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 7118 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 17118 -j ACCEPT

2、 Redis 安裝過程

# yum install gcc tcl
# cd /usr/local/src
# wget http://download.redis.io/releases/redis-3.2.9.tar.gz
# mkdir /usr/local/redis3
# tar -zxvf redis-3.2.9.tar.gz
# cd redis-3.2.9
# make PREFIX=/usr/local/redis3 install
# yum install ruby rubygems
# gem install redis

3、 建立叢集配置目錄,並拷貝 redid.conf 配置檔案到各節點配置目錄

192.168.1.117
# mkdir -p /usr/local/redis3/cluster/7117
# mkdir -p /usr/local/redis3/cluster/7118
# cp /usr/local/src/redis-3.2.9/redis.conf /usr/local/redis3/cluster/7117/redis-7117.conf
# cp /usr/local/src/redis-3.2.9/redis.conf /usr/local/redis3/cluster/7118/redis-7118.conf
提示: conf 配置檔案具體檢視連結http://download.csdn.net/detail/l1028386804/9854797
提供的配置檔案,主要增加了資料目錄 dir 屬性的配置。

4、 在 192.168.1.117 上使用如下命令啟動這 2 個 Redis 例項

# /usr/local/redis3/bin/redis-server /usr/local/redis3/cluster/7117/redis-7117.conf
# /usr/local/redis3/bin/redis-server /usr/local/redis3/cluster/7118/redis-7118.conf
# ps -ef | grep redis
root 4865 1 0 01:01 ? 00:00:00 /usr/local/redis3/bin/redis-server *:7117 [cluster]
root 4869 1 0 01:01 ? 00:00:00 /usr/local/redis3/bin/redis-server *:7118 [cluster]

三、 Redis 叢集的擴充套件性測試

1、 redis-trib.rb 命令介紹

[[email protected] src]# /usr/local/src/redis-3.2.9/src/redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>
import host:port
--from <arg>
set-timeout host:port milliseconds
del-node host:port node_id
create host1:port1 ... hostN:portN
--replicas <arg>
help (show this help)
add-node new_host:new_port existing_host:existing_port
--slave
--master-id <arg>
reshard host:port
--slots <arg>
--to <arg>
--yes
--from <arg>
fix host:port
check host:port
call host:port command arg arg .. arg
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working
node in the cluster.
redis-trib.rb 命令引數說明:
call:執行 redis 命令
create:建立一個新的叢集(上一節教程有介紹)
add-node: 將一個節點新增到叢集裡面,第一個是新節點 ip:port, 第二個是叢集中任意一個正常節點
ip:port, --master-id
reshard:重新分片
check:檢視叢集資訊
del-node:移除一個節點

2、 新增新的 Master 節點

add-node 將一個節點新增到叢集裡面, 第一個是新節點 ip:port, 第二個是任意一個已存在節點 ip:port
# /usr/local/src/redis-3.2.9/src/redis-trib.rb add-node 192.168.1.117:7117 192.168.1.111:7111
>>> Adding node 192.168.1.117:7117 to cluster 192.168.1.111:7111
Connecting to node 192.168.1.111:7111: OK
Connecting to node 192.168.1.116:7116: OK
Connecting to node 192.168.1.113:7113: OK
Connecting to node 192.168.1.112:7112: OK
Connecting to node 192.168.1.115:7115: OK
Connecting to node 192.168.1.114:7114: OK
>>> Performing Cluster Check (using node 192.168.1.111:7111)
M: cc50047487b52697d62b1a72b231b7c74e08e051 192.168.1.111:7111
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: b21ae6d0a3e614e53bbc52639173ec3ad68044b5 192.168.1.116:7116
slots: (0 slots) slave
replicates 041addd95fa0a15d98be363034e53dd06f69ef47
M: 041addd95fa0a15d98be363034e53dd06f69ef47 192.168.1.113:7113
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 712e523b617eea5a2ed8df732a50ff298ae2ea48 192.168.1.112:7112
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 55c0db5af1b917f3ce0783131fb8bab28920e1f3 192.168.1.115:7115
slots: (0 slots) slave
replicates 712e523b617eea5a2ed8df732a50ff298ae2ea48
S: 8a6ca1452d61f8b4726f0649e6ce49a6ec4afee2 192.168.1.114:7114
slots: (0 slots) slave
replicates cc50047487b52697d62b1a72b231b7c74e08e051
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Connecting to node 192.168.1.117:7117: OK
>>> Send CLUSTER MEET to node 192.168.1.117:7117 to make it join the cluster.
[OK] New node added correctly.
以上操作結果表示節點新增成功, 新增的節點不包含任何資料, 因為它沒有分配任何 slot。新加入的節點是一個 master 節點,當叢集需要將某個從節點升級為新的主節點時,這個新節點不會被選中。
為新節點分配雜湊槽(slot):你只需要指定叢集中其中一個節點的地址, redis-trib 就會自動找到叢集中的其他節點。目前 redis-trib只能在管理員的協助下完成重新分片的工作,命令如下:
# /usr/local/src/redis-3.2.9/src/redis-trib.rb reshard 192.168.1.111:7111
Connecting to node 192.168.1.111:7111: OK
Connecting to node 192.168.1.117:7117: OK
Connecting to node 192.168.1.116:7116: OK
Connecting to node 192.168.1.113:7113: OK
Connecting to node 192.168.1.112:7112: OK
Connecting to node 192.168.1.115:7115: OK
Connecting to node 192.168.1.114:7114: OK
>>> Performing Cluster Check (using node 192.168.1.111:7111)
M: cc50047487b52697d62b1a72b231b7c74e08e051 192.168.1.111:7111
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: badbc0ffde2a3700df7e179d23fa2762108eabba 192.168.1.117:7117
slots: (0 slots) master
0 additional replica(s)
S: b21ae6d0a3e614e53bbc52639173ec3ad68044b5 192.168.1.116:7116
slots: (0 slots) slave
replicates 041addd95fa0a15d98be363034e53dd06f69ef47
M: 041addd95fa0a15d98be363034e53dd06f69ef47 192.168.1.113:7113
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 712e523b617eea5a2ed8df732a50ff298ae2ea48 192.168.1.112:7112
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 55c0db5af1b917f3ce0783131fb8bab28920e1f3 192.168.1.115:7115
slots: (0 slots) slave
replicates 712e523b617eea5a2ed8df732a50ff298ae2ea48
S: 8a6ca1452d61f8b4726f0649e6ce49a6ec4afee2 192.168.1.114:7114
slots: (0 slots) slave
replicates cc50047487b52697d62b1a72b231b7c74e08e051
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 500
上面的提示是確認你打算移動的雜湊槽 slots 的數量(這裡槽數量設定為 500)除了移動的雜湊槽數量之外, redis-trib 還需要知道重新分片的目標(target node), 也就是負責接收這 500 個雜湊槽的節點。指定目標需要使用節點的 ID , 而不是 IP 地址和埠。 我們打算向上面新增的主節點來作為目標, 它的 IP 地址和埠是 192.168.1.117:7117,而節點 ID 則是badbc0ffde2a3700df7e179d23fa2762108eabba , 那麼我們應該向 redis-trib 提供節點的 ID :What is the receiving node ID? badbc0ffde2a3700df7e179d23fa2762108eabba接下來 redis-trib 會向你詢問重新分片的源節點(source node), 也就是要從哪個節點中取出 500 個雜湊槽,並將這些槽移動到目標節點上面。如果我們不打算從特定的節點上取出指定數量的雜湊槽, 那麼可以向 redis-trib 輸入 all , 這樣的話, 叢集中的所有主節點都會成為源節點, redis-trib 將從各個源節點中各取出一部分雜湊槽, 湊夠 500 個, 然後移動到目標節點上面:
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
輸入 all 並按下回車之後, redis-trib 將打印出雜湊槽的移動計劃:
Do you want to proceed with the proposed reshard plan (yes/no)? yes
如果你覺得沒問題的話, 就可以輸入 yes 並再次按回車確認, redis-trib 就會正式開始執行重新分片操作, 將指定的雜湊槽從源節點一個個地移動到目標節點上面
注意: 可以同步觀察重新分片是否會對客戶端的連續使用產生影響(結果:不影響)。移動前, 7117 上沒有 slot:


移動後,7117 上有 3 段 slot:

在重新分片操作執行完畢之後, 可以使用以下命令來檢查叢集是否正常:

# /usr/local/src/redis-3.2.9/src/redis-trib.rb check 192.168.1.111:7111

上面輸出的檢查結果顯示, 重新分片成功, 叢集狀態正常。也可以用以下命令再次檢視叢集的節點狀況:
# /usr/local/redis3/bin/redis-cli -c -p 7111 cluster nodes

以上叢集狀態輸出資訊解析:
(1)節點 ID
(2)IP:PORT
(3)節點狀態標識: master、 slave、 myself、 fail?、 fail
(4)如果是從節點,表示主節點的 ID;如果是主節點,則為 '-'
(5)叢集最近一次向各個節點發送 PING 命令後,過去多長時間還沒有接到回覆
(6)節點最近一次返回 PONG 的時間戳
(7)節點的配置紀元
(8)本節點的網路連線情況: connected、 disconnected
(9)如果是主節點,表示節點包含的槽

3、 新增新的 slave 節點

(1)新增節點
#/usr/local/src/redis-3.2.9/src/redis-trib.rb add-node 192.168.1.117:7118 192.168.1.111:7111
>>> Adding node 192.168.1.117:7118 to cluster 192.168.1.111:7111
Connecting to node 192.168.1.111:7111: OK
Connecting to node 192.168.1.117:7117: OK
Connecting to node 192.168.1.116:7116: OK
Connecting to node 192.168.1.113:7113: OK
Connecting to node 192.168.1.112:7112: OK
Connecting to node 192.168.1.115:7115: OK
Connecting to node 192.168.1.114:7114: OK
>>> Performing Cluster Check (using node 192.168.1.111:7111)
M: cc50047487b52697d62b1a72b231b7c74e08e051 192.168.1.111:7111
slots:11089-16383 (5295 slots) master
1 additional replica(s)
M: badbc0ffde2a3700df7e179d23fa2762108eabba 192.168.1.117:7117
slots:0-165,5461-5627,10923-11088 (499 slots) master
0 additional replica(s)
S: b21ae6d0a3e614e53bbc52639173ec3ad68044b5 192.168.1.116:7116
slots: (0 slots) slave
replicates 041addd95fa0a15d98be363034e53dd06f69ef47
M: 041addd95fa0a15d98be363034e53dd06f69ef47 192.168.1.113:7113
slots:166-5460 (5295 slots) master
1 additional replica(s)
M: 712e523b617eea5a2ed8df732a50ff298ae2ea48 192.168.1.112:7112
slots:5628-10922 (5295 slots) master
1 additional replica(s)
S: 55c0db5af1b917f3ce0783131fb8bab28920e1f3 192.168.1.115:7115
slots: (0 slots) slave
replicates 712e523b617eea5a2ed8df732a50ff298ae2ea48
S: 8a6ca1452d61f8b4726f0649e6ce49a6ec4afee2 192.168.1.114:7114
slots: (0 slots) slave
replicates cc50047487b52697d62b1a72b231b7c74e08e051
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Connecting to node 192.168.1.117:7118: OK
>>> Send CLUSTER MEET to node 192.168.1.117:7118 to make it join the cluster.
[OK] New node added correctly.

新增的 7118 為一個 master
(2)redis-cli 連線上新節點 shell,輸入命令:cluster replicate 對應 master 的 node-id
# /usr/local/redis3/bin/redis-cli -c -p 7118
127.0.0.1:7118>cluster replicate ab31611b3424990e2b9bbe73135cb4cb0ace394f
OK
線上新增 slave 時,需要 dump 整個 master 程序,並傳遞到 slave,再由 slave 載入 rdb 檔案到記憶體, rdb傳輸過程中 Master 可能無法提供服務,整個過程消耗大量 IO, 因此要小心操作。檢視執行結果:
127.0.0.1:7116> cluster nodes

這時 7117 已變成 7118 的 slave

4、 線上 reshard 資料

對於負載/資料不均勻的情況,可以線上 reshard slot 來解決,方法與新增新 master 的 reshard 一樣,只是需要 reshard 的 master 節點是老節點。

5、 刪除一個 slave 節點

# cd /usr/local/src/redis-3.2.9/src/
# ./redis-trib.rb del-node 192.168.1.117:7118 5256e05a17c106c93285a03aff1b1b9e7ca7bf0c

這時候, 用下以下再檢視叢集狀態,會發現該 slave 節點已成功移除
# /usr/local/redis3/bin/redis-cli -c -p 7116 cluster nodes

移除一個節點, 對應的節點程序也會被關閉。

6、 刪除一個 master 節點

刪除 master 節點之前首先要使用 reshard 移除該 master 的全部 slot,然後再刪除當前節點(目前只能把被刪除 master 的 slot 遷移到一個節點上),操作和分配 slot 類似,指定具體的 Source node 即可。

# /usr/local/src/redis-3.2.9/src/redis-trib.rb reshard 192.168.1.117:7117
Connecting to node 192.168.1.117:7117: OK
Connecting to node 192.168.1.112:7112: OK
Connecting to node 192.168.1.115:7115: OK
Connecting to node 192.168.1.114:7114: OK
Connecting to node 192.168.1.111:7111: OK
Connecting to node 192.168.1.116:7116: OK
Connecting to node 192.168.1.113:7113: OK
>>> Performing Cluster Check (using node 192.168.1.117:7117)
M: ab31611b3424990e2b9bbe73135cb4cb0ace394f 192.168.1.117:7117
slots:0-165,5461-5627,10923-11088 (499 slots) master
0 additional replica(s)
M: d2c6c159b07e8197e2c8d2eae8c847050159f602 192.168.1.112:7112
slots:5628-10922 (5295 slots) master
1 additional replica(s)
S: f34b28f1483f0c0d9543e93938fc12b8818050cb 192.168.1.115:7115
slots: (0 slots) slave
replicates d2c6c159b07e8197e2c8d2eae8c847050159f602
M: 48db78bcc55c4c3a3788940a6458b921ccf95d44 192.168.1.114:7114
slots:11089-16383 (5295 slots) master
1 additional replica(s)
S: 8dd55e9b4da9f62b9b15232e86553f1337864179 192.168.1.111:7111
slots: (0 slots) slave
replicates 48db78bcc55c4c3a3788940a6458b921ccf95d44
S: 1fd90d54090925afb4087d4ef94a1710a25160d6 192.168.1.116:7116
slots: (0 slots) slave
replicates 4e46bd06654e8660e617f7249fa22f6fa1fdff0d
M: 4e46bd06654e8660e617f7249fa22f6fa1fdff0d 192.168.1.113:7113
slots:166-5460 (5295 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
//輸入被刪除 master 的所有 slot 數量
How many slots do you want to move (from 1 to 16384)? 499
//接收 slot 的 master 節點 ID
What is the receiving node ID? 48db78bcc55c4c3a3788940a6458b921ccf95d44
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
//準備被刪除 master 節點的 node-id
Source node #1: ab31611b3424990e2b9bbe73135cb4cb0ace394f
Source node #2:done
Ready to move 499 slots.
Source nodes:
M: ab31611b3424990e2b9bbe73135cb4cb0ace394f 192.168.1.117:7117
slots:0-165,5461-5627,10923-11088 (499 slots) master
0 additional replica(s)
Destination node:
M: 48db78bcc55c4c3a3788940a6458b921ccf95d44 192.168.1.114:7114
slots:11089-16383 (5295 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 0 from ab31611b3424990e2b9bbe73135cb4cb0ace394f
Moving slot 1 from ab31611b3424990e2b9bbe73135cb4cb0ace394f
……
//輸入 yes 執行 reshard
Do you want to proceed with the proposed reshard plan (yes/no)? yes
移除該 Master 節點的所有 slot 後,重新檢視叢集狀態,會發現該節點不再佔用 slot:
# /usr/local/redis3/bin/redis-cli -c -p 7116 cluster nodes

確認已清空該 Master 節點的所有 slot 後就可以刪除該節點了(命令與刪除 slave 節點一樣):
# cd /usr/local/src/redis-3.2.9/src/
# ./redis-trib.rb del-node 192.168.1.117:7117 ab31611b3424990e2b9bbe73135cb4cb0ace394f
結果如下:

此時執行 ps -ef | grep redis 命令會發現,被移除的節點對應用的 Redis 例項也已經被關閉了


至此,Redis叢集的擴充套件與測試完畢。