1. 程式人生 > >Redis叢集模式部署

Redis叢集模式部署

1. Redis部署

以下以Linux(CentOS)系統為例

1.1 下載和編譯

$ wget http://download.redis.io/releases/redis-4.0.7.tar.gz
$ tar xzf redis-4.0.7.tar.gz
$ cd redis-4.0.7
$ make

編譯完成後會在src目錄下生成Redis服務端程式redis-server和客戶端程式redis-cli

1.2 啟動服務

1、前臺執行

src/redis-server

該方式啟動預設為前臺方式執行,使用預設配置。

2、後臺執行

可以修改redis.conf

檔案的daemonize引數為yes,指定配置檔案啟動,例如:

vi redis.conf

# By default Redis does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
daemonize yes

指定配置檔案啟動。

src/redis-server redis.conf

例如:

#指定配置檔案後臺啟動
[root@kube-node-1 redis-4.0.7
]# src/redis-server redis.conf 95778:C 30 Jan 00:44:37.633 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 95778:C 30 Jan 00:44:37.634 # Redis version=4.0.7, bits=64, commit=00000000, modified=0, pid=95778, just started 95778:C 30 Jan 00:44:37.634 # Configuration loaded #檢視Redis程序 [root@kube-node-1 redis-4.0.7]# ps aux|grep redis
root 95779 0.0 0.0 145268 468 ? Ssl 00:44 0:00 src/redis-server 127.0.0.1:6379

更多啟動引數如下:

[[email protected]1 src]# ./redis-server --help
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --slaveof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel

1.3 客戶端測試

$ src/redis-cli
redis> set foo bar
OK
redis> get foo
"bar"

2. Redis叢集部署

Redis的叢集部署需要在每臺叢集部署的機器上安裝Redis(可參考上述的[1. Redis部署]),然後修改配置以叢集的方式啟動。

2.1 手動部署叢集

2.1.1 設定配置檔案及啟動例項

修改配置檔案redis.conf,叢集模式的最小化配置檔案如下:

#可選操作,該項設定後臺方式執行,
daemonize yes

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

更多叢集配置引數可參考預設配置檔案redis.conf中Cluster模組的說明

最小叢集模式需要三個master例項,一般建議起六個例項,即三主三從。因此我們建立6個以埠號命名的目錄存放例項的配置檔案和其他資訊。

mkdir cluster-test
cd cluster-test
mkdir 7000 7001 7002 7003 7004 7005

在對應埠號的目錄中建立redis.conf的檔案,配置檔案的內容可參考上述的叢集模式配置。每個配置檔案中的埠號port引數改為對應目錄的埠號。

複製redis-server的二進位制檔案到cluster-test目錄中,通過指定配置檔案的方式啟動redis服務,例如:

cd 7000
../redis-server ./redis.conf

如果是以前臺方式執行,則會在控制檯輸出以下資訊:

[82462] 26 Nov 11:56:55.329 * No cluster configuration found, I'm 97a3a64667477371c4479320d683e4c8db5858b1

每個例項都會生成一個Node ID,類似97a3a64667477371c4479320d683e4c8db5858b1,用來作為Redis例項在叢集中的唯一標識,而不是通過IP和Port,IP和Port可能會改變,該Node ID不會改變。

目錄結構可參考:

cluster-test/
├── 7000
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── 7001
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── 7002
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── 7003
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── 7004
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── 7005
│   ├── appendonly.aof
│   ├── dump.rdb
│   ├── nodes.conf
│   └── redis.conf
├── redis-cli
└── redis-server

2.1.2 redis-trib建立叢集

Redis的例項全部執行之後,還需要redis-trib.rb工具來完成叢集的建立,redis-trib.rb二進位制檔案在Redis包主目錄下的src目錄中,執行該工具依賴Ruby環境和gem,因此需要提前安裝。

1、安裝Ruby

yum -y install ruby rubygems

檢視Ruby版本資訊。

[root@kube-node-1 src]# ruby --version
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]

由於centos系統預設支援Ruby版本為2.0.0,因此執行gem install redis命令時會報以下錯誤。

[[email protected]1 src]# gem install redis
Fetching: redis-4.0.1.gem (100%)
ERROR:  Error installing redis:
    redis requires Ruby version >= 2.2.2.

解決方法是先安裝rvm,再升級ruby版本。

2、安裝rvm

curl -L get.rvm.io | bash -s stable

如果遇到以下報錯,則執行報錯中的gpg2 --recv-keys的命令。

[[email protected]1 ~]# curl -L get.rvm.io | bash -s stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   194  100   194    0     0    335      0 --:--:-- --:--:-- --:--:--   335
100 24090  100 24090    0     0  17421      0  0:00:01  0:00:01 --:--:-- 44446
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: 於 20170911日 星期一 045921秒 CST 建立的簽名,使用 RSA,鑰匙號 BF04FF17
gpg: 無法檢查簽名:沒有公鑰
Warning, RVM 1.26.0 introduces signed releases and automated check of signatures when GPG software found. Assuming you trust Michal Papis import the mpapis public key (downloading the signatures).

GPG signature verification failed for '/usr/local/rvm/archives/rvm-1.29.3.tgz' - 'https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc'! Try to install GPG v2 and then fetch the public key:

    gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

or if it fails:

    command curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -

the key can be compared with:

    https://rvm.io/mpapis.asc
    https://keybase.io/mpapis

NOTE: GPG version 2.1.17 have a bug which cause failures during fetching keys from remote server. Please downgrade or upgrade to newer version (if available) or use the second method described above.

執行報錯中的gpg2 --recv-keys的命令。

例如:

[[email protected]1 ~]# gpg2 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
gpg: 鑰匙環‘/root/.gnupg/secring.gpg’已建立
gpg: 下載金鑰‘D39DC0E3’,從 hkp 伺服器 keys.gnupg.net
gpg: /root/.gnupg/trustdb.gpg:建立了信任度資料庫
gpg: 金鑰 D39DC0E3:公鑰“Michal Papis (RVM signing) <[email protected].com>”已匯入
gpg: 沒有找到任何絕對信任的金鑰
gpg: 合計被處理的數量:1
gpg:           已匯入:1  (RSA: 1)

再次執行命令curl -L get.rvm.io | bash -s stable。例如:

[[email protected]1 ~]# curl -L get.rvm.io | bash -s stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   194  100   194    0     0    310      0 --:--:-- --:--:-- --:--:--   309
100 24090  100 24090    0     0  18230      0  0:00:01  0:00:01 --:--:--  103k
Downloading https://github.com/rvm/rvm/archive/1.29.3.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.3/1.29.3.tar.gz.asc
gpg: 於 20170911日 星期一 045921秒 CST 建立的簽名,使用 RSA,鑰匙號 BF04FF17
gpg: 完好的簽名,來自於“Michal Papis (RVM signing) <[email protected]>”
gpg:               亦即“Michal Papis <[email protected]>”
gpg:               亦即“[jpeg image of size 5015]”
gpg: 警告:這把金鑰未經受信任的簽名認證!
gpg:       沒有證據表明這個簽名屬於它所聲稱的持有者。
主鑰指紋: 409B 6B17 96C2 7546 2A17  0311 3804 BB82 D39D C0E3
子鑰指紋: 62C9 E5F4 DA30 0D94 AC36  166B E206 C29F BF04 FF17
GPG verified '/usr/local/rvm/archives/rvm-1.29.3.tgz'
Creating group 'rvm'

Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

以上表示執行成功,然後執行以下命令。

source /usr/local/rvm/scripts/rvm

檢視rvm庫中已知的ruby版本

rvm list known

例如:

[[email protected] ~]# rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-head] # security released on head
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p330]
[ruby-]1.9.3[-p551]
[ruby-]2.0.0[-p648]
[ruby-]2.1[.10]
[ruby-]2.2[.7]
[ruby-]2.3[.4]
[ruby-]2.4[.1]
ruby-head
...

3、升級Ruby

#安裝ruby
rvm install  2.4.0
#使用新版本
rvm use  2.4.0
#移除舊版本
rvm remove 2.0.0
#檢視當前版本
ruby --version

例如:

[root@kube-node-1 ~]# rvm install  2.4.0
Searching for binary rubies, this might take some time.
Found remote file https://rvm_io.global.ssl.fastly.net/binaries/centos/7/x86_64/ruby-2.4.0.tar.bz2
Checking requirements for centos.
Installing requirements for centos.
Installing required packages: autoconf, automake, bison, bzip2, gcc-c++, libffi-devel, libtool, readline-devel, sqlite-devel, zlib-devel, libyaml-devel, openssl-devel................................
Requirements installation successful.
ruby-2.4.0 - #configure
ruby-2.4.0 - #download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14.0M  100 14.0M    0     0   852k      0  0:00:16  0:00:16 --:--:--  980k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-2.4.0 - #validate archive
ruby-2.4.0 - #extract
ruby-2.4.0 - #validate binary
ruby-2.4.0 - #setup
ruby-2.4.0 - #gemset created /usr/local/rvm/gems/ruby-2.4.0@global
ruby-2.4.0 - #importing gemset /usr/local/rvm/gemsets/global.gems..............................
ruby-2.4.0 - #generating global wrappers........
ruby-2.4.0 - #gemset created /usr/local/rvm/gems/ruby-2.4.0
ruby-2.4.0 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.4.0 - #generating default wrappers........

[root@kube-node-1 ~]# rvm use  2.4.0
Using /usr/local/rvm/gems/ruby-2.4.0

[root@kube-node-1 ~]# rvm remove 2.0.0
ruby-2.0.0-p648 - #already gone
Using /usr/local/rvm/gems/ruby-2.4.0

[root@kube-node-1 ~]# ruby --version
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

4、安裝gem

gem install redis

例如:

[root@kube-node-1 ~]# gem install redis

Fetching: redis-4.0.1.gem (100%)
Successfully installed redis-4.0.1
Parsing documentation for redis-4.0.1
Installing ri documentation for redis-4.0.1
Done installing documentation for redis after 2 seconds
1 gem installed

5、執行redis-trib.rb命令

以上表示安裝成功,可以執行redis-trib.rb命令。

cd src 
#執行redis-trib.rb命令
./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \
> 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

引數create表示建立一個新的叢集,--replicas 1表示為每個master建立一個slave。

如果建立成功會顯示以下資訊

[OK] All 16384 slots covered

例如:

[[email protected] src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \
> 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: d5a834d075fd93eefab877c6ebb86efff680650f 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: 13d0c397604a0b2644244c37b666fce83f29faa8 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: be2718476eba4e56f696e56b75e67df720b7fc24 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: 3d02f59b34047486faecc023685379de7b38076c 127.0.0.1:7003
   replicates 13d0c397604a0b2644244c37b666fce83f29faa8
S: dedf672f0a75faf37407ac4edd5da23bc4651e25 127.0.0.1:7004
   replicates be2718476eba4e56f696e56b75e67df720b7fc24
S: 99c07119a449a703583019f7699e15afa0e41952 127.0.0.1:7005
   replicates d5a834d075fd93eefab877c6ebb86efff680650f
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: d5a834d075fd93eefab877c6ebb86efff680650f 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: be2718476eba4e56f696e56b75e67df720b7fc24 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 13d0c397604a0b2644244c37b666fce83f29faa8 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 3d02f59b34047486faecc023685379de7b38076c 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 13d0c397604a0b2644244c37b666fce83f29faa8
S: 99c07119a449a703583019f7699e15afa0e41952 127.0.0.1:7005
   slots: (0 slots) slave
   replicates d5a834d075fd93eefab877c6ebb86efff680650f
S: dedf672f0a75faf37407ac4edd5da23bc4651e25 127.0.0.1:7004
   slots: (0 slots) slave
   replicates be2718476eba4e56f696e56b75e67df720b7fc24
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2.1.3 部署結果驗證

1、客戶端驗證

使用客戶端redis-cli二進位制訪問某個例項,執行setget的測試。

$ redis-cli -c -p 7000
redis 127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
redis 127.0.0.1:7002> set hello world
-> Redirected to slot [866] located at 127.0.0.1:7000
OK
redis 127.0.0.1:7000> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
redis 127.0.0.1:7000> get hello
-> Redirected to slot [866] located at 127.0.0.1:7000
"world"

2、叢集狀態

使用cluster info命令檢視叢集狀態。

127.0.0.1:7000> cluster info
cluster_state:ok                       #叢集狀態
cluster_slots_assigned:16384           #被分配的槽位數
cluster_slots_ok:16384                 #正確分配的槽位
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6                  #當前節點
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:48273
cluster_stats_messages_pong_sent:49884
cluster_stats_messages_sent:98157
cluster_stats_messages_ping_received:49879
cluster_stats_messages_pong_received:48273
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:98157

3、節點狀態

使用cluster nodes命令檢視節點狀態。

127.0.0.1:7000> cluster nodes
be2718476eba4e56f696e56b75e67df720b7fc24 127.0.0.1:7002@17002 master - 0 1517303607000 3 connected 10923-16383
13d0c397604a0b2644244c37b666fce83f29faa8 127.0.0.1:7001@17001 master - 0 1517303606000 2 connected 5461-10922
3d02f59b34047486faecc023685379de7b38076c 127.0.0.1:7003@17003 slave 13d0c397604a0b2644244c37b666fce83f29faa8 0 1517303606030 4 connected
d5a834d075fd93eefab877c6ebb86efff680650f 127.0.0.1:7000@17000 myself,master - 0 1517303604000 1 connected 0-5460
99c07119a449a703583019f7699e15afa0e41952 127.0.0.1:7005@17005 slave d5a834d075fd93eefab877c6ebb86efff680650f 0 1517303607060 6 connected
dedf672f0a75faf37407ac4edd5da23bc4651e25 127.0.0.1:7004@17004 slave be2718476eba4e56f696e56b75e67df720b7fc24 0 1517303608082 5 connected

參考文章: