keepalived實現nginx高可用
1、環境說明
IP | 服務 | 作用 |
---|---|---|
192.168.1.101 | nginx + keepalived | master |
192.168.1.102 | nginx + keepalived | backup |
192.168.1.103 | 虛擬ip(VIP) |
- 說明:
系統:CentOS 6.10
master配一個,backup可以配置多個;
虛擬ip(VIP):192.168.1.103,對外提供服務的ip,也可稱作浮動ip
各個組件之間的關系圖如下:
tomcat的安裝不在本博客範圍之內;
2、nginx 安裝與配置
2.1、安裝nginx
master和backup所有節點都安裝
vim /etc/yum.repos.d/nginx.repo
添加如下內容:
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1
安裝
yum install nginx -y
2.2、master節點配置
2.2.1、刪除沒用的配置內容
vim /etc/nginx/conf.d/default.conf
改為如下:
server { listen 80; server_name localhost; access_log /var/log/nginx/host.access.log main; location / { root /usr/share/nginx/html; index index.html index.htm; } error_page 404 /404.html; error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } }
2.2.2、修改nginx默認顯示內容
vim /usr/share/nginx/html/index.html
只修改第14行內容,如下:
1 <!DOCTYPE html> 2 <html> 3 <head> 4 <title>Welcome to nginx!</title> 5 <style> 6 body { 7 width: 35em; 8 margin: 0 auto; 9 font-family: Tahoma, Verdana, Arial, sans-serif; 10 } 11 </style> 12 </head> 13 <body> 14 <h1>Welcome to nginx! test keepalived master!</h1> 15 <p>If you see this page, the nginx web server is successfully installed and 16 working. Further configuration is required.</p> 17 18 <p>For online documentation and support please refer to 19 <a href="http://nginx.org/">nginx.org</a>.<br/> 20 Commercial support is available at 21 <a href="http://nginx.com/">nginx.com</a>.</p> 22 23 <p><em>Thank you for using nginx.</em></p> 24 </body> 25 </html>
2.3、backup節點配置
只把/usr/share/nginx/html/index.html
的第14行改為如下,其它和master一致。
<h1>Welcome to nginx! test keepalived backup!</h1>
3、keepalived服務
3.1、keepalived 是什麽?
Keepalived 一方面具有配置管理LVS的功能,同時還具有對LVS下面節點進行健康檢查的功能,另一方面也可實現系統網絡服務的高可用功能,用來防止單點故障。
3.2、keepalived 工作原理
keepalived 是以 VRRP 協議為實現基礎,VRRP全稱Virtual Router Redundancy Protocol,即虛擬路由冗余協議。
虛擬路由冗余協議,可以認為是實現路由器高可用的協議,即將N臺提供相同功能的路由器組成一個路由器組,這個組裏面有一個master和多個backup,master上面有一個對外提供服務的vip(該路由器所在局域網內其他機器的默認路由為該vip),master會發組播vrrp包,用於高速backup自己還活著,當backup收不到vrrp包時就認為master宕掉了,這時就需要根據VRRP的優先級來選舉一個backup當master。這樣的話就可以保證路由器的高可用了。保證業務的連續性,接管速度最快可以小於1秒。
3.3、keepalived主要有三個模塊,分別是core、check和vrrp。
core模塊為keepalived的核心,負責主進程的啟動、維護以及全局配置文件的加載和解析。
check負責健康檢查,包括常見的各種檢查方式。
vrrp模塊是來實現VRRP協議的。
3.4、keepalived 與 zookeeper 高可用性區別
- Keepalived:
- 優點:簡單,基本不需要業務層面做任何事情,就可以實現高可用,主備容災。而且容災的宕機時間也比較短。
- 缺點:也是簡單,因為VRRP、主備切換都沒有什麽復雜的邏輯,所以無法應對某些特殊場景,比如主備通信鏈路出問題,會導致腦裂。同時keepalived也不容易做負載均衡。
- Zookeeper:
- 優點:可以支持高可用,負載均衡。本身是個分布式的服務。
- 缺點:跟業務結合的比較緊密。需要在業務代碼中寫好ZK使用的邏輯,比如註冊名字。拉取名字對應的服務地址等。
4、keepalived 配置
4.1、keepalived 安裝
master和backup所有節點都安裝
[root@node1 ~]# yum install keepalived -y
[root@node1 ~]# rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf # keepalived服務主配置文件
/etc/rc.d/init.d/keepalived # 服務啟動腳本(centos 7 之前的用init.d 腳本啟動,之後的systemd啟動)
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.2.13
... ...
/usr/share/man/man1/genhash.1.gz
/usr/share/man/man5/keepalived.conf.5.gz
/usr/share/man/man8/keepalived.8.gz
/usr/share/snmp/mibs/KEEPALIVED-MIB.txt
4.2、默認配置及說明
[root@node1 keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs { # 全局配置
notification_email { # 定義報警郵件地址
[email protected]
[email protected]
[email protected]
}
notification_email_from [email protected] # 定義發送郵件的地址
smtp_server 192.168.200.1 # 郵箱服務器
smtp_connect_timeout 30 # 定義超時時間
router_id LVS_DEVEL # 定義路由器標識信息,相同的局域網唯一,標識本節點的字條串,通常為hostname
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
<br># 虛擬 IP 配置 vrrp
vrrp_instance VI_1 { # 定義實例
state MASTER # 狀態參數 master/backup 僅表示說明,主節點為 MASTER, 對應的備份節點為 BACKUP
interface eth0 # 綁定虛擬 IP 的網絡接口,與本機 IP 地址所在的網絡接口相同, 我的是 eth0
virtual_router_id 51 # 虛擬路由的 ID 號, 兩個節點設置必須一樣, 可選 IP 最後一段使用, 相同的 VRID 為一個組,他將決定多播的 MAC 地址<br> mcast_src_ip 192.168.10.50 ## 本機 IP 地址
priority 100 # 優先級決定是主還是備 --> 越大越優先,節點優先級, 值範圍 0-254, MASTER 要比 BACKUP 高 <br> nopreempt # 優先級高的設置 nopreempt 解決異常恢復後再次搶占的問題
advert_int 1 # 主備心跳通訊時間間隔,組播信息發送間隔,兩個節點設置必須一樣, 默認 1s
authentication { # 認證授權,設置驗證信息,兩個節點必須一致
auth_type PASS
auth_pass 1111
}<br> <br> ## 將 track_script 塊加入 instance 配置塊 <br> <br> track_script {<br> chk_nginx ## 執行 Nginx 監控的服務 <br> }
virtual_ipaddress { # vip,設備之間使用的虛擬ip地址,可以定義多個 <br>
192.168.200.16
192.168.200.17
192.168.200.18
}
}
virtual_server 192.168.200.100 443 {
delay_loop 6
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.201.100 443 {
weight 1
SSL_GET {
url {
path /
digest ff20ad2481f97b1754ef3e12ecd3a9cc
}
url {
path /mrtg/
digest 9b3a0c85a887a256d6939da88aabd8cd
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
virtual_server 10.10.10.2 1358 {
delay_loop 6
lb_algo rr
lb_kind NAT
persistence_timeout 50
protocol TCP
sorry_server 192.168.200.200 1358
real_server 192.168.200.2 1358 {
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.200.3 1358 {
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
virtual_server 10.10.10.3 1358 {
delay_loop 3
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.200.4 1358 {
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.200.5 1358 {
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
4.3、master主負載均衡服務器配置
[root@master ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_01
}
## keepalived 會定時執行腳本並對腳本執行的結果進行分析,動態調整 vrrp_instance 的優先級。
# 如果腳本執行結果為 0,並且 weight 配置的值大於 0,則優先級相應的增加。
# 如果腳本執行結果非 0,並且 weight配置的值小於 0,則優先級相應的減少。
# 其他情況,維持原本配置的優先級,即配置文件中 priority 對應的值。
vrrp_script chk_nginx {
script "/etc/keepalived/nginx_check.sh" ## 檢測 nginx 狀態的腳本路徑
interval 2 ## 檢測時間間隔
weight -20 ## 如果條件成立,權重-20
}
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
## 將 track_script 塊加入 instance 配置塊
track_script {
chk_nginx ## 執行 Nginx 監控的服務
}
virtual_ipaddress {
192.168.1.103
}
}
... ...
[root@master ~]#
4.4、backup備負載均衡服務器配置
[root@slave ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_02
}
vrrp_script chk_nginx {
script "/etc/keepalived/nginx_check.sh"
interval 3
weight -20
}
vrrp_instance VI_1 {
state SLAVE
interface eth1
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
## 將 track_script 塊加入 instance 配置塊
track_script {
chk_nginx ## 執行 Nginx 監控的服務
}
virtual_ipaddress {
192.168.1.103
}
}
... ...
[root@slave ~]#
5、測試
5.1、編寫 nginx 檢測腳本
在所有的節點上面編寫Nginx
狀態檢測腳本/etc/keepalived/nginx_check.sh
(已在 keepalived.conf 中配置)
腳本要求:如果 nginx 停止運行,嘗試啟動,如果無法啟動則殺死本機的 keepalived 進程, keepalied將虛擬 ip 綁定到 BACKUP 機器上。
內容如下:
[root@master ~]# vim /etc/keepalived/nginx_check.sh
#!/bin/bash
set -x
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
echo `date`‘: nginx is not healthy, try to killall keepalived‘ >> /etc/keepalived/keepalived.log
killall keepalived
fi
[root@master ~]# chmod +x /etc/keepalived/nginx_check.sh
[root@master ~]# ll /etc/keepalived/nginx_check.sh
-rwxr-xr-x 1 root root 338 2019-02-15 14:11 /etc/keepalived/nginx_check.sh
5.2、啟動所有節點上的nginx和keepalived
-
啟動nginx
service nginx start
- 啟動keepalived
相關操作命令如下:chkconfig keepalived on # keepalived服務開機啟動 service keepalived start # 啟動服務 service keepalived stop # 停止服務 service keepalived restart # 重啟服務
keepalived正常運行後,會啟動3個進程,其中一個是父進程,負責監控其子進程。一個是vrrp子進程,另外一個是checkers子進程。
[root@master ~]# ps -ef | grep keepalived
root 3653 1 0 14:18 ? 00:00:00 /usr/sbin/keepalived -D
root 3654 3653 0 14:18 ? 00:00:02 /usr/sbin/keepalived -D
root 3655 3653 0 14:18 ? 00:00:03 /usr/sbin/keepalived -D
root 7481 3655 0 15:19 ? 00:00:00 /usr/sbin/keepalived -D
root 7483 1323 0 15:19 pts/0 00:00:00 grep --color=auto keepalived
[root@master ~]#
5.3、master主負載均衡服務器IP
信息:192.168.1.101
[root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether b0:51:8e:01:9b:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:20:ae:75 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.101/24 brd 192.168.1.255 scope global eth1
inet 192.168.1.103/32 scope global eth1
inet6 fe80::20c:29ff:fe20:ae75/64 scope link
valid_lft forever preferred_lft forever
[root@master ~]#
5.4、backup備負載均衡服務器查看IP
信息:192.168.1.102
[root@slave ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether b0:51:8e:01:9b:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:7d:6a:24 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.102/24 brd 192.168.1.255 scope global eth1
inet6 fe80::20c:29ff:fe7d:6a24/64 scope link
valid_lft forever preferred_lft forever
[root@slave ~]#
以上可以看到,虛擬ip(VIP)生效是在192.168.1.101
服務器上。
5.5、測試
通過VIP(192.168.1.103)來訪問nginx,結果如下:
以上可知,現在生效的nginx代理機器是1.101;我們停掉機器1.101上面的keepalived
[root@master ~]# service keepalived stop
停止 keepalived: [確定]
再使用VIP(192.168.1.103)訪問nginx服務,結果如下:
以上可知,現在生效的nginx代理機器是1.102;我們重啟機器1.101上面的keepalived
[root@master ~]# service keepalived start
正在啟動 keepalived: [確定]
再使用VIP(192.168.1.103)訪問nginx服務,結果如下:
至此,Keepalived + Nginx 實現高可用 Web 負載均衡搭建完畢!
keepalived實現nginx高可用