Redis Cluster 叢集部署
2.4:Redis Cluster 部署
2.4.1:部署 Redis Cluster 的前提
- 時間同步;
- Redis Node 均採用相同的硬體配置、相同的密碼、相同的 Redis 版本;
- Redis Node 必須開啟的引數:
- cluster-enabled yes:啟用叢集,開啟後 Redis 程序會有 Cluster 顯示;
- cluster-config-file nodes-6380.conf:指定叢集配置檔案,此檔案由 Redis Cluster 自動建立和維護,開啟即可,不需要任何手動配置;
- 所有 Redis Node 必須沒有任何資料;
2.4.2:拓撲圖
共 6 臺 Redis 伺服器組成 Redis Cluster,三主三從,即叢集中的 Redis Node 為 3 個;
奇數為 Master,偶數為 Slave,方便後期維護管理時辨別;
-
Masters:Redis1、Redis3、Redis5;
-
Slaves:Redis2、Redis4、Redis6;
各 Redis 伺服器編譯安裝 redis-4.0.14 和 redis-5.0.10(為了測試 redis 5 版本的叢集建立和維護);
2.4.3:各節點配置並啟動 Redis
編輯各節點的 Redis 配置檔案:
配置 masterauth(如果有 requirepass);
開啟叢集;指定叢集配置檔案;
masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf
各節點啟動為單機 Redis:
systemctl start redis && systemctl enable redis
2.4.4:建立叢集(Redis-3/4)
準備 redis-trib.rb 使用環境
Redis 3 和 4 版本建立叢集時,需要使用叢集管理工具 redis-trib.rb,這個工具是 Redis 官方推出的管理 Redis 叢集的工具,整合在 Redis 原始碼的 src 目錄下,是基於 Redis 提供的叢集命令而封裝的簡單、便捷、實用的操作工具;
redis-trib.rb 是 Redis 作者用 Ruby 開發完成的,CentOS yum 安裝的 Ruby 存在版本較低問題,需要編譯安裝 Ruby:
只需在一臺 Redis 伺服器上準備環境即可;
[[email protected] ~]# yum remove ruby rubygems -y
[[email protected] ~]# cd /usr/local/src
[[email protected] src]# wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
[[email protected] src]# tar xf ruby-2.5.5.tar.gz
[[email protected] src]# cd ruby-2.5.5/
[[email protected] ruby-2.5.5]# ./configure
[[email protected] ruby-2.5.5]# make -j 2
[[email protected] ruby-2.5.5]# make install
安裝 Ruby 的 Redis 模組:
[[email protected] ruby-2.5.5]# gem install redis
如果無法線上安裝 Ruby 的 Redis 模組,可以先下載 Redis 模組(https://rubygems.org/gems/redis),再離線安裝:
gem install -l redis-3.3.0.gem
建立 redis-trib.rb 軟連結:
[[email protected] ~]# ln -sv /usr/local/src/redis-4.0.14/src/redis-trib.rb /usr/bin/redis-trib
‘/usr/bin/redis-trib’ -> ‘/usr/local/src/redis-4.0.14/src/redis-trib.rb’
檢視 redis-trib.rb 使用幫助:
[[email protected] ~]# redis-trib
Usage: redis-trib <command> <options> <arguments ...>
create host1:port1 ... hostN:portN # 建立叢集,指定叢集中的節點;
--replicas <arg> # 指定每個Master的副本數量(即一個Master有幾個Slave);
check host:port # 檢查叢集狀態
info host:port # 檢視叢集資訊
fix host:port # 修復叢集
--timeout <arg>
reshard host:port # 重新分片(熱遷移slots);
--from <arg>
--to <arg>
--slots <arg>
--yes
--timeout <arg>
--pipeline <arg>
rebalance host:port # 平衡叢集中各節點的slot數量;
--weight <arg>
--auto-weights
--use-empty-masters
--timeout <arg>
--simulate
--pipeline <arg>
--threshold <arg>
add-node new_host:new_port existing_host:existing_port
# 向叢集中新增節點;
--slave
--master-id <arg>
del-node host:port node_id # 從叢集中刪除節點;
set-timeout host:port milliseconds # 設定節點的超時時間(單位毫秒);
call host:port command arg arg .. arg
# 在叢集的所有節點上執行命令;
import host:port # 匯入外部redis伺服器的資料到當前叢集;
--from <arg>
--copy
--replace
help (show this help)
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
為 Ruby Redis 模組配置 Redis 連線密碼
[[email protected] ~]# vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.2.5/lib/redis/client.rb
class Redis
class Client
# Defaults are also used for converting string keys to symbols.
DEFAULTS = {
……
password: 123456,
建立 Redis Cluster
建立時,配置在前的節點會成為 Master:
[[email protected] ~]# redis-trib create --replicas 1 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.201:6379
192.168.1.203:6379
192.168.1.205:6379
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
replicates 469e0c16495598b5fadba28f856203e206464b99
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
slots: (0 slots) slave
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
slots: (0 slots) slave
replicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
slots: (0 slots) slave
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
2.4.5:建立叢集(Redis-5)
Redis 5 版本可以直接使用 redis-cli 建立 Redis Cluster:
[[email protected] ~]# redis-cli -a 123456 --cluster create 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
slots: (0 slots) slave
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
slots: (0 slots) slave
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
slots: (0 slots) slave
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
redis-cli -a 123456 --cluster create 192.168.7.101:6379
192.168.7.101:6380 192.168.7.102:6379 192.168.7.102:6380 1
92.168.7.103:6379 192.168.7.103:6380 --cluster-replicas 1
2.4.6:驗證叢集建立
叢集中任一節點均可檢視叢集狀態和資訊;
CLUSTER 指令
檢視叢集狀態:
127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:127
cluster_stats_messages_pong_sent:125
cluster_stats_messages_meet_sent:3
cluster_stats_messages_sent:255
cluster_stats_messages_ping_received:121
cluster_stats_messages_pong_received:130
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:255
檢視 Node 對應關係:
127.0.0.1:6379> CLUSTER NODES
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609749076000 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] myself,slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609749076000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] master - 0 1609749077996 1 connected 0-5460
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609749075000 4 connected
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609749076973 5 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609749075960 3 connected 10923-16383
redis-trib.rb(Redis-3/4)
指定叢集中的任一節點均可檢視叢集的狀態資訊;
檢視叢集資訊:
[[email protected] ~]# redis-trib info 192.168.1.206:6379
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.201:6379 (469e0c16...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
檢查叢集狀態:
[[email protected] ~]# redis-trib check 192.168.1.203:6379
>>> Performing Cluster Check (using node 192.168.1.203:6379)
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
slots: (0 slots) slave
replicates 469e0c16495598b5fadba28f856203e206464b99
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
slots: (0 slots) slave
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
slots: (0 slots) slave
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
redis-cli(Redis-5)
[[email protected] ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
slots: (0 slots) slave
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
slots: (0 slots) slave
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
slots: (0 slots) slave
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
檢視叢集配置檔案
Redis-4.0.14:
[[email protected] ~]# cat /apps/redis/data/nodes-6379.conf
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
Redis-5.0.10:
[[email protected] ~]# cat /apps/redis5/data/nodes-6379.conf
df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:[email protected] master - 0 1609811160000 3 connected 10923-16383
c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:[email protected] master - 0 1609811161000 2 connected 5461-10922
b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:[email protected] slave c2b48542344ece56e69d8ca9404ee29a48ae7b8e 0 1609811162000 6 connected
90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:[email protected] slave feb6b43233fdcc01e0d5425fd03e0116b74f0833 0 1609811162884 5 connected
d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:[email protected] slave df921d013a6cbf1588c1ad51809435fe39f6c25c 0 1609811162000 4 connected
feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:[email protected] myself,master - 0 1609811160000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
2.4.7:測試叢集讀寫
叢集資料寫入
Redis1 節點上嘗試寫入:
提示 foo1 這個 key 的 CRC 結果被排程到了 13431 槽位,該槽位在 192.168.1.205:6379 節點上;
[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
(error) MOVED 13431 192.168.1.205:6379
Redis5 節點 寫入 foo1:
[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
OK
各節點只儲存各自槽位的資料,所以 Redis1 和 Redis3 都沒有剛剛寫入的 foo1:
[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> KEYS foo1
(empty list or set)
叢集資料讀取
因為各節點只儲存各自槽位的資料,所以資料的讀取也只能到相應的節點上進行;
而且叢集中的 Slave 節點,讀寫服務均不提供;
檢視 Redis5 的 Slave 節點:
Redis5 的 ID 為:1b897eda6acfb806eb27326f151e4067f0d47f7b;
可以查到對應的 Slave 為 Redis2(192.168.1.202);
[[email protected] ~]# cat /apps/redis/data/nodes-6379.conf
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
到 Redis2 讀取剛剛寫入到 Redis5 的 foo1:
可以查到有 foo1 的資料,但是讀取還是得到 Redis5 上;
127.0.0.1:6379> KEYS foo1
1) "foo1"
127.0.0.1:6379> GET foo1
(error) MOVED 13431 192.168.1.205:6379