1. 程式人生 > 其它 >Redis Cluster 叢集部署

Redis Cluster 叢集部署

技術標籤:Redisredis

2.4:Redis Cluster 部署

2.4.1:部署 Redis Cluster 的前提

  1. 時間同步;
  2. Redis Node 均採用相同的硬體配置、相同的密碼、相同的 Redis 版本;
  3. Redis Node 必須開啟的引數:
    • cluster-enabled yes:啟用叢集,開啟後 Redis 程序會有 Cluster 顯示;
    • cluster-config-file nodes-6380.conf:指定叢集配置檔案,此檔案由 Redis Cluster 自動建立和維護,開啟即可,不需要任何手動配置;
  4. 所有 Redis Node 必須沒有任何資料;

2.4.2:拓撲圖

在這裡插入圖片描述

共 6 臺 Redis 伺服器組成 Redis Cluster,三主三從,即叢集中的 Redis Node 為 3 個;
奇數為 Master,偶數為 Slave,方便後期維護管理時辨別;

  • Masters:Redis1、Redis3、Redis5;

  • Slaves:Redis2、Redis4、Redis6;

各 Redis 伺服器編譯安裝 redis-4.0.14 和 redis-5.0.10(為了測試 redis 5 版本的叢集建立和維護);

2.4.3:各節點配置並啟動 Redis

編輯各節點的 Redis 配置檔案:

配置 masterauth(如果有 requirepass);
開啟叢集;

指定叢集配置檔案;

masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf

各節點啟動為單機 Redis:

systemctl start redis && systemctl enable redis

2.4.4:建立叢集(Redis-3/4)

準備 redis-trib.rb 使用環境

Redis 3 和 4 版本建立叢集時,需要使用叢集管理工具 redis-trib.rb,這個工具是 Redis 官方推出的管理 Redis 叢集的工具,整合在 Redis 原始碼的 src 目錄下,是基於 Redis 提供的叢集命令而封裝的簡單、便捷、實用的操作工具;

redis-trib.rb 是 Redis 作者用 Ruby 開發完成的,CentOS yum 安裝的 Ruby 存在版本較低問題,需要編譯安裝 Ruby:

只需在一臺 Redis 伺服器上準備環境即可;

[[email protected] ~]# yum remove ruby rubygems -y
[[email protected] ~]# cd /usr/local/src
[[email protected] src]# wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
[[email protected] src]# tar xf ruby-2.5.5.tar.gz 
[[email protected] src]# cd ruby-2.5.5/
[[email protected] ruby-2.5.5]# ./configure
[[email protected] ruby-2.5.5]# make -j 2
[[email protected] ruby-2.5.5]# make install

安裝 Ruby 的 Redis 模組:

[[email protected] ruby-2.5.5]# gem install redis

如果無法線上安裝 Ruby 的 Redis 模組,可以先下載 Redis 模組(https://rubygems.org/gems/redis),再離線安裝:

gem install -l redis-3.3.0.gem

建立 redis-trib.rb 軟連結:

[[email protected] ~]# ln -sv /usr/local/src/redis-4.0.14/src/redis-trib.rb /usr/bin/redis-trib
‘/usr/bin/redis-trib’ -> ‘/usr/local/src/redis-4.0.14/src/redis-trib.rb’

檢視 redis-trib.rb 使用幫助:

[[email protected] ~]# redis-trib 
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN    # 建立叢集,指定叢集中的節點;
                  --replicas <arg>               # 指定每個Master的副本數量(即一個Master有幾個Slave);
  check           host:port						 # 檢查叢集狀態
  info            host:port						 # 檢視叢集資訊
  fix             host:port						 # 修復叢集
                  --timeout <arg>
  reshard         host:port						 # 重新分片(熱遷移slots);
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port						 # 平衡叢集中各節點的slot數量;
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
                  --simulate
                  --pipeline <arg>
                  --threshold <arg>
  add-node        new_host:new_port existing_host:existing_port
  												 # 向叢集中新增節點;
                  --slave
                  --master-id <arg>
  del-node        host:port node_id				 # 從叢集中刪除節點;
  set-timeout     host:port milliseconds		 # 設定節點的超時時間(單位毫秒);
  call            host:port command arg arg .. arg
  												 # 在叢集的所有節點上執行命令;
  import          host:port						 # 匯入外部redis伺服器的資料到當前叢集;
                  --from <arg>
                  --copy
                  --replace
  help            (show this help)

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

為 Ruby Redis 模組配置 Redis 連線密碼

[[email protected] ~]# vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.2.5/lib/redis/client.rb
class Redis
  class Client
    # Defaults are also used for converting string keys to symbols.
    DEFAULTS = {
    ……
    password: 123456,

建立 Redis Cluster

建立時,配置在前的節點會成為 Master:

[[email protected] ~]# redis-trib create --replicas 1 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.201:6379
192.168.1.203:6379
192.168.1.205:6379
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   replicates 469e0c16495598b5fadba28f856203e206464b99
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   slots: (0 slots) slave
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   slots: (0 slots) slave
   replicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   slots: (0 slots) slave
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2.4.5:建立叢集(Redis-5)

Redis 5 版本可以直接使用 redis-cli 建立 Redis Cluster:

[[email protected] ~]# redis-cli -a 123456 --cluster create 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379 --cluster-replicas 1

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   slots: (0 slots) slave
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   slots: (0 slots) slave
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   slots: (0 slots) slave
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
redis-cli -a 123456 --cluster create 192.168.7.101:6379
192.168.7.101:6380 192.168.7.102:6379 192.168.7.102:6380 1
92.168.7.103:6379 192.168.7.103:6380 --cluster-replicas 1

2.4.6:驗證叢集建立

叢集中任一節點均可檢視叢集狀態和資訊;

CLUSTER 指令

檢視叢集狀態:

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:127
cluster_stats_messages_pong_sent:125
cluster_stats_messages_meet_sent:3
cluster_stats_messages_sent:255
cluster_stats_messages_ping_received:121
cluster_stats_messages_pong_received:130
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:255

檢視 Node 對應關係:

127.0.0.1:6379> CLUSTER NODES
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609749076000 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] myself,slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609749076000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] master - 0 1609749077996 1 connected 0-5460
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609749075000 4 connected
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609749076973 5 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609749075960 3 connected 10923-16383

redis-trib.rb(Redis-3/4)

指定叢集中的任一節點均可檢視叢集的狀態資訊;

檢視叢集資訊:

[[email protected] ~]# redis-trib info 192.168.1.206:6379
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.201:6379 (469e0c16...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

檢查叢集狀態:

[[email protected] ~]# redis-trib check 192.168.1.203:6379
>>> Performing Cluster Check (using node 192.168.1.203:6379)
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   slots: (0 slots) slave
   replicates 469e0c16495598b5fadba28f856203e206464b99
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   slots: (0 slots) slave
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   slots: (0 slots) slave
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli(Redis-5)

[[email protected] ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   slots: (0 slots) slave
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   slots: (0 slots) slave
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   slots: (0 slots) slave
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

檢視叢集配置檔案

Redis-4.0.14:

[[email protected] ~]# cat /apps/redis/data/nodes-6379.conf 
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

Redis-5.0.10:

[[email protected] ~]# cat /apps/redis5/data/nodes-6379.conf 
df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:[email protected] master - 0 1609811160000 3 connected 10923-16383
c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:[email protected] master - 0 1609811161000 2 connected 5461-10922
b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:[email protected] slave c2b48542344ece56e69d8ca9404ee29a48ae7b8e 0 1609811162000 6 connected
90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:[email protected] slave feb6b43233fdcc01e0d5425fd03e0116b74f0833 0 1609811162884 5 connected
d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:[email protected] slave df921d013a6cbf1588c1ad51809435fe39f6c25c 0 1609811162000 4 connected
feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:[email protected] myself,master - 0 1609811160000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

2.4.7:測試叢集讀寫

叢集資料寫入

Redis1 節點上嘗試寫入:

提示 foo1 這個 key 的 CRC 結果被排程到了 13431 槽位,該槽位在 192.168.1.205:6379 節點上;

[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
(error) MOVED 13431 192.168.1.205:6379

Redis5 節點 寫入 foo1:

[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
OK

各節點只儲存各自槽位的資料,所以 Redis1 和 Redis3 都沒有剛剛寫入的 foo1:

[[email protected] ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> KEYS foo1
(empty list or set)

叢集資料讀取

因為各節點只儲存各自槽位的資料,所以資料的讀取也只能到相應的節點上進行;

而且叢集中的 Slave 節點,讀寫服務均不提供;

檢視 Redis5 的 Slave 節點:

Redis5 的 ID 為:1b897eda6acfb806eb27326f151e4067f0d47f7b;
可以查到對應的 Slave 為 Redis2(192.168.1.202);

[[email protected] ~]# cat /apps/redis/data/nodes-6379.conf 
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:[email protected] slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:[email protected] master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:[email protected] slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:[email protected] master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:[email protected] slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:[email protected] myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

到 Redis2 讀取剛剛寫入到 Redis5 的 foo1:

可以查到有 foo1 的資料,但是讀取還是得到 Redis5 上;

127.0.0.1:6379> KEYS foo1
1) "foo1"
127.0.0.1:6379> GET foo1
(error) MOVED 13431 192.168.1.205:6379