Redis Cluster 初探(1)
Redis Cluster是Redis官方的叢集實現方案,在此之前已經有一些民間的第三方Redis叢集解決方案,如Twitter的Twenproxy,豌豆莢的Codis,與其不同的是,Redis Cluster並非使用Porxy的模式來連線叢集節點,而是使用無中心節點的模式來組建叢集,有一定效能優勢也有缺點,本文主要是我調研Redis Cluster的一些知識整理與經驗彙總。
首先我們來嘗試下搭建一個Redis Cluster叢集
前置準備
Redis Cluster需要Redis 3.0及以上版本才支援,此文釋出的時候,Redis的最高版本為3.0.5。
wget http://download.redis.io/releases/redis-3.0.5.tar.gz
tar -xvf redis-3.0.5.tar.gz
cd redis-3.0.5
make
編譯完Redis,生成的可執行檔案在redis-3.0.5/src
之中,為了方便使用,我們把可執行檔案的目錄加入PATH之中。
vim ~/.bashrc
# 增加以下內容
REDIS_HOME=/home/xiezefan/sofeware/redis-3.0.5/src
PATH=$REDIS_HOME:$PATH
export PATH
# 儲存後讓修改生效
source ~/.bashrc
要建立Redis Cluster,我們還需要安裝Ruby以及RubyGems。
yum install ruby
yum install gcc g++ make automake autoconf curl-devel openssl-devel zlib-devel httpd-devel apr-devel apr-util-devel sqlite-devel
yum install ruby-rdoc ruby-devel
yum install rubygems
gem install redis
建立叢集
此次此時我們需要建立8個節點,埠號7000~7007
複製redis-3.0.5/redis.conf
,修改一下內容
mkdir cluster
cp ~/sofeware/redis-3.0.5/redis.conf cluster/
vim cluster/redis.conf
# 修改以下內容
port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
批量複製7份,修改配置檔案的埠號。修改完畢後,分別進入各個節點的目錄中啟動redis
cd 7000
redis-server redis.conf
cd 7001
redis-server redis.conf
# 依次啟動7000-7007
...
至此,8個Redis全都啟動完畢,但是他們還處於彼此互相不知道彼此的階段。
[email protected]:~$ ps -ef | grep redis
xiezefan 13372 1 0 20:09 ? 00:00:08 redis-server *:7000 [cluster]
xiezefan 13376 1 0 20:09 ? 00:00:08 redis-server *:7001 [cluster]
xiezefan 13380 1 0 20:09 ? 00:00:08 redis-server *:7002 [cluster]
xiezefan 13382 1 0 20:09 ? 00:00:08 redis-server *:7003 [cluster]
xiezefan 13386 1 0 20:09 ? 00:00:08 redis-server *:7004 [cluster]
xiezefan 13390 1 0 20:09 ? 00:00:08 redis-server *:7005 [cluster]
xiezefan 13394 1 0 20:09 ? 00:00:08 redis-server *:7006 [cluster]
xiezefan 13400 1 0 20:09 ? 00:00:08 redis-server *:7007 [cluster]
下一步,我們要將7000-7005這六個節點連線成一個叢集。
redis-trib.rb create --replicas 1 10.211.55.4:7000 10.211.55.4:7001 10.211.55.4:7002 10.211.55.4:7003 10.211.55.4:7004 10.211.55.4:7005
該命令表示,將7000-7006節點建立一個叢集,冗餘為1,就是3主3從。切記,此處指定的IP會在client傳送move命令的時候返回,所以一定要指定為客戶端能訪問到的IP,例如下面這種IP是不可行的,client拿到的IP就位是127.0.0.1導致一直重定向失敗。
redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
輸入後,redis-trib自動分配給出一個slot的分配方案
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.211.55.4:7000
10.211.55.4:7001
10.211.55.4:7002
Adding replica 10.211.55.4:7003 to 10.211.55.4:7000
Adding replica 10.211.55.4:7004 to 10.211.55.4:7001
Adding replica 10.211.55.4:7005 to 10.211.55.4:7002
M: 9dfef549f7917794cbabaf96781ed0e19957c1f3 10.211.55.4:7000
slots:0-5460 (5461 slots) master
M: c14485e8d7f1f3ec5c505b41fe727b657c951d8d 10.211.55.4:7001
slots:5461-10922 (5462 slots) master
M: 8b308093e99f4299b8c18ab1dd81c5a83a3528c6 10.211.55.4:7002
slots:10923-16383 (5461 slots) master
S: 632409883570eb5cecf6089583fba64a41d1154f 10.211.55.4:7003
replicates 9dfef549f7917794cbabaf96781ed0e19957c1f3
S: edd31196e2980360b0738c57af95e1a69b0f9c9b 10.211.55.4:7004
replicates c14485e8d7f1f3ec5c505b41fe727b657c951d8d
S: 95063a99c5cf2cc4abc51ca1e8ff0f3b8d60271c 10.211.55.4:7005
replicates 8b308093e99f4299b8c18ab1dd81c5a83a3528c6
Can I set the above configuration? (type 'yes' to accept):
輸入yes後,叢集自動建立,建立完畢後,通過redis-cli進入任一節點,使用cluster nodes
命令可以檢視各個節點的狀態
[email protected]:~$ redis-cli -p 7000
127.0.0.1:7000>cluster nodes
c14485e8d7f1f3ec5c505b41fe727b657c951d8d 10.211.55.4:7001 master - 0 1449060968280 2 connected 5461-10922
95063a99c5cf2cc4abc51ca1e8ff0f3b8d60271c 10.211.55.4:7005 slave 8b308093e99f4299b8c18ab1dd81c5a83a3528c6 0 1449060968280 6 connected
edd31196e2980360b0738c57af95e1a69b0f9c9b 10.211.55.4:7004 slave c14485e8d7f1f3ec5c505b41fe727b657c951d8d 0 1449060967274 5 connected
8b308093e99f4299b8c18ab1dd81c5a83a3528c6 10.211.55.4:7002 master - 0 1449060966769 3 connected 10923-16383
632409883570eb5cecf6089583fba64a41d1154f 10.211.55.4:7003 slave 9dfef549f7917794cbabaf96781ed0e19957c1f3 0 1449060967274 4 connected
9dfef549f7917794cbabaf96781ed0e19957c1f3 10.211.55.4:7000 myself,master - 0 0 1 connected 0-5460
隨便輸入一個查詢指令get user.1
127.0.0.1:7000> get user.1
(error) MOVED 9645 10.211.55.4:7001
因為user.1所在的solt-9645在7001節點上,cluster給你傳送MOVED指令讓你去7001節點查詢資料,連線redis-cli的時候,使用-c
引數可以指定查詢時接收到MOVED指令自動跳轉
[email protected]:~$ redis-cli -c -p 7000
127.0.0.1:7000> get user.1
-> Redirected to slot [9645] located at 10.211.55.4:7001
(nil)
叢集擴容
現在我們已經有一個包含6個節點的叢集,我寫了段程式碼,往叢集寫入10W條測試資料。
現在模擬機器擴容場景,為叢集加入一個master節點7006和一個slave節點7007。
redis-trib.rb add-node 10.211.55.4:7006 10.211.55.4:7000
以上命令將7006節點接入7000所在的叢集。接下來,我們為7006增加一個slave節點。
redis-trib.rb add-node --slave 10.211.55.4:7007 10.211.55.4:7000
以上命令表示增加slave節點,將7006的節點加入7000節點所在的叢集中作為slave節點,隨機依附現有的master節點中slave最少的節點,如果需要再指定特別的master節點,使用
redis-trib.rb add-node --slave --master-id 23b412673af0506df6382353e3a65960d5b7e66d 10.211.55.4:7007 10.211.55.4:7000
其中的23b412673af0506df6382353e3a65960d5b7e66d
是7006節點的id,我們可以通過cluster nodes
命令檢視節點的id。
接下來我們用坐負載均衡,Slot是Redis Cluster資料承載的最小單位,我們可以指定將一定範圍的Slot轉移到新的節點來實現負載均衡。
Redis Cluster轉移一個Slot的步驟是:
- 在目標節點上宣告將從源節點上遷入Slot
CLUSTER SETSLOT <slot> IMPORTING <source_node_id>
- 在源節點上宣告將往目標節點遷出Slot
CLUSTER SETSLOT <slot> IMPORTING <source_node_id>
- 批量從源節點獲取KEY
CLUSTER GETKEYSINSLOT <slot> <count>
- 將獲取的Key遷移到目標節點
MIGRATE <target_ip> <target_port> <key_name> 0 <timeout>
- 重複步驟3,4直到所有資料遷移完畢
- 分別向雙方節點發送
CLUSTER SETSLOT <slot> NODE <target_node_id>
,該命令將會廣播給叢集其他節點,已經將Slot轉移到目標節點。 - 等待叢集狀態變為OK
CLUSTER INFO
中的cluster_state = ok
我編寫了一個指令碼來批量遷移Slot
#!/bin/bash
source_host=$1 # 源節點HOST
source_port=$2 # 源節點埠
target_host=$3 # 目標節點HOST
target_port=$4 # 目標節點埠
start_slot=$5 # 遷移節點的其實範圍
end_slot=$6 # 遷移節點的結束範圍
for slot in `seq ${start_slot} ${end_slot}`
do
redis-cli -c -h ${target_host} -p ${target_port} cluster setslot ${slot} importing `redis-cli -c -h ${source_host} -p ${source_port} cluster nodes | grep ${source_port} | awk '{print $1}'`
echo "Setslot importing ${slot} to ${target_host}:${target_port} success"
redis-cli -c -h ${source_host} -p ${source_port} cluster setslot ${slot} migrating `redis-cli -c -h ${target_host} -p ${target_port} cluster nodes | grep ${target_port} | awk '{print $1}'`
echo "Setslot migrating ${slot} from ${source_host}:${source_port} success"
while [ 1 -eq 1 ]
do
allkeys=`redis-cli -c -h ${source_host} -p ${source_port} cluster getkeysinslot ${slot} 10`
if [ -z "${allkeys}" ]
then
redis-cli -c -h ${source_host} -p ${source_port} cluster setslot ${slot} node `redis-cli -c -h ${target_host} -p ${target_port} cluster nodes | grep ${target_port} | awk '{print $1}'`
redis-cli -c -h ${target_host} -p ${target_port} cluster setslot ${slot} node `redis-cli -c -h ${source_host} -p ${target_port} cluster nodes | grep ${target_port} | awk '{print $1}'`
echo "Migrate slot ${slot} finish"
break
else
for key in ${allkeys}
do
redis-cli -c -h ${source_host} -p ${source_port} migrate ${target_host} ${target_port} ${key} 0 5000
echo "Migrate slot ${slot} key ${key} success"
done
fi
done
done
執行命令 bash rebalance-cluster.sh 10.211.55.4 7000 10.211.55.4 7006 0 1000
將7000節點上0-1000這個範圍內的Slot轉移到7006節點,通過cluster nodes
命令我們可以看到0-1000這個區間是slot已經從7000轉移到7006
[email protected]:~/sheel$ redis-cli -c -p 7000 cluster nodes
23b412673af0506df6382353e3a65960d5b7e66d 10.211.55.4:7006 master - 0 1449064402389 7 connected 0-1000
0c2954d21d7bcdae333f4fdecf468ce05aa25544 10.211.55.4:7001 master - 0 1449064400372 2 connected 5461-10922
384a3bb5bd9ecb2fc7db75c866abc715d7966f82 10.211.55.4:7002 master - 0 1449064401381 3 connected 10923-16383
c3e09d286ef2dce49843268b20832d65a5d516a1 10.211.55.4:7004 slave 0c2954d21d7bcdae333f4fdecf468ce05aa25544 0 1449064401885 5 connected
50737b4a91443ab1a34eec4ef99d4f6fe5d358f4 10.211.55.4:7005 slave 384a3bb5bd9ecb2fc7db75c866abc715d7966f82 0 1449064402389 6 connected
3c62cc6664bba378cceb8ae8e02f5d727deafe9d 10.211.55.4:7007 slave 23b412673af0506df6382353e3a65960d5b7e66d 0 1449064400878 7 connected
d6441916dcd89cbf431465d92dfc0eb3dd235295 10.211.55.4:7003 slave 6ee21c5d93a6d2f293a2df1b37e8c9c27cb55ad8 0 1449064402389 4 connected
6ee21c5d93a6d2f293a2df1b37e8c9c27cb55ad8 10.211.55.4:7000 myself,master - 0 0 1 connected 1001-5460