使用Nginx+keepalived反向代理Tomcat實現負載均衡
阿新 • • 發佈：2021-06-23
1. 測試概要
本次測試使用4臺虛擬機器進行Nginx+Keepalived實現Tomcat的負載均衡與反向代理；
1.1. 測試環境
作業系統：CentOS 7 Minimal Install（Linux version 3.10.0-1160.el7.x86_64 ）
虛擬機器VM1：Tomcat1（172.17.0.234）apache-tomcat-10.0.6 jdk-8u202-linux-x64.tar.gz
虛擬機器VM2：Tomcat2（172.17.0.235）apache-tomcat-10.0.6 jdk-8u202-linux-x64.tar.gz
虛擬機器VM3：Nginx+Keepalived(Master)（172.17.0.236）
虛擬機器VM4
VIP：172.17.0.238
1.2 系統設定（每臺虛擬機器都要執行以下操作）
（1）關閉防火牆
# systemctl stop firewalld ## 關閉firewalld服務，當前生效，重啟失效
# systemctl disable firewalld ## 禁止firewalld開機啟動
（2）禁用Selinux
# getenforce ## 獲取selinux的狀態 # setenforce 0 ## 臨時禁用selinux # vi /etc/selinux/config ## 修改selinux配置，禁止開機啟動，修改SELINUX=disabled （檔案內容如下） # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
（3）配置IP地址，保證測試機器間的網路是通的
# vi /etc/sysconfig/network-scripts/ifcfg-ens33 網絡卡配置檔案如下（根據實際情況配置IP地址）： TYPE=Ethernet BOOTPROTO=static DEFROUTE=yes NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=172.17.0.234 NETMASK=255.255.255.0 GATEWAY=172.17.0.254 DNS1=223.5.5.5 # service network restart ## 重啟網路使配置生效 # ping 172.17.0.235 ## VM之間相互ping一下確認通訊正常 PING 172.17.0.235 (172.17.0.235) 56(84) bytes of data. 64 bytes from 172.17.0.235: icmp_seq=1 ttl=64 time=0.732 ms 64 bytes from 172.17.0.235: icmp_seq=2 ttl=64 time=0.536 ms 64 bytes from 172.17.0.235: icmp_seq=3 ttl=64 time=0.819 ms
1.3 JDK安裝(VM1,VM2做同樣配置)
# tar -zxvf jdk-8u202-linux-x64.tar.gz ## 解壓jdk包
# mv jdk1.8.0_202/ /usr/local/ ## 將jdk移到/usr/local目錄下
# vi /etc/profile ## 修改java的環境變數
以下為檔案部分內容，在檔案末尾追加（JAVA_HOME,CLASSPATH,PATH）環境變數）：
HOSTNAME=`/usr/bin/hostname 2>/dev/null`
HISTSIZE=1000
if [ "$HISTCONTROL" = "ignorespace" ] ; then
export HISTCONTROL=ignoreboth
else
export HISTCONTROL=ignoredups
fi
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL
# By default, we want umask to get set. This sets it for login shell
# Current threshold for system reserved uid/gids is 200
# You could check uidgid reservation validity in
# /usr/share/doc/setup-*/uidgid file
if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
umask 002
else
umask 022
fi
for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
if [ -r "$i" ]; then
if [ "${-#*i}" != "$-" ]; then
. "$i"
else
. "$i" >/dev/null
fi
fi
done
unset i
unset -f pathmunge
export JAVA_HOME=/usr/local/jdk1.8.0_202
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
1.4 Apache安裝(VM1,VM2做同樣配置)
# cd /usr/local/
# tar -zxvf apache-tomcat-10.0.6.tar.gz
# cd /usr/local/apache-tomcat-10.0.6/bin
# ./startup.sh
# vi /usr/local/apache-tomcat-10.0.6/webapps/ROOT/index.jsp
編輯index.jsp檔案(VM1)
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>成功</title>
<h1>我是172.17.0.234（Tomcat-Master）
編輯index.jsp檔案(VM2)
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>成功</title>
</h1><h1>我是172.17.0.235（Tomcat-Backup）
1.5 通過訪問VM1，VM2的地址，確認Tomcat工作正常
★至此，VM1，VM2配置完成，接下來需要配置Nginx反向代理，測試環境注意一定要關閉firewalld和selinux，不然會帶來很多麻煩，當然，生產環境建議配置防火牆策略來打通主機間的網路，而不是粗暴的關閉防火牆。
2. Nginx和keepalived安裝部署（VM3，VM4同樣配置）
##通過yum安裝nginx和keepalived
# yum install keepalived
# yum install nginx
# vi /etc/nginx/nginx.conf
檔案內容如下：
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
sendfile on;
keepalive_timeout 65;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
upstream www.test.com{
server 172.17.0.234:8080 weight=5;
server 172.17.0.235:8080 weight=5;
}
server {
listen 80;
server_name www.test.com:8080;
location / {
proxy_pass http://www.test.com;
}
error_page 404 /404.html;
location = /404.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
# systemctl start nginx ##啟動nginx
# systemctl status nginx ##檢視Nginx啟動狀態
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2021-06-16 22:22:49 EDT; 6h ago
Process: 1527 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Process: 1524 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 1522 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 1529 (nginx)
Tasks: 3
CGroup: /system.slice/nginx.service
├─1529 nginx: master process /usr/sbin/nginx
├─1530 nginx: worker process
└─1531 nginx: worker process
Jun 16 22:22:49 localhost.localdomain systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jun 16 22:22:49 localhost.localdomain nginx[1524]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Jun 16 22:22:49 localhost.localdomain nginx[1524]: nginx: configuration file /etc/nginx/nginx.conf test is successful
Jun 16 22:22:49 localhost.localdomain systemd[1]: Started The nginx HTTP and reverse proxy server.
VM3,VM4反向代理到Tomcat，目前配置輪詢權重是相同的server 172.17.0.234:8080 weight=5; server 172.17.0.235:8080 weight=5;
VM3,VM4反向代理到Tomcat，目前配置輪詢權重是不相同的 server 172.17.0.234:8080 weight=2; server 172.17.0.235:8080 weight=8;
配置Keepalived服務
VM3配置：
# vi /etc/keepalived/keepalived.conf
檔案內容如下：
! Configuration File for keepalived
global_defs {
router_id nginxmaster
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 62
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.17.0.238
}
}
# systemctl start keepalived ##啟用keepalived
# systemctl status keepalived ##檢視keepalived啟動情況
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-06-17 08:25:46 EDT; 20s ago
Process: 7687 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 7688 (keepalived)
Tasks: 3
CGroup: /system.slice/keepalived.service
├─7688 /usr/sbin/keepalived -D
├─7689 /usr/sbin/keepalived -D
└─7690 /usr/sbin/keepalived -D
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:48 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:25:53 localhost.localdomain Keepalived_vrrp[7690]: Sending gratuitous ARP on ens33 for 172.17.0.238
VM4配置：
# vi /etc/keepalived/keepalived.conf
檔案內容如下：
! Configuration File for keepalived
global_defs {
router_id nginxbackup
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 62
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.17.0.238
}
}
# systemctl start keepalived ##啟用keepalived
# systemctl status keepalived ##檢視keepalived啟動情況
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-06-17 08:30:20 EDT; 6s ago
Process: 2870 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 2871 (keepalived)
Tasks: 3
CGroup: /system.slice/keepalived.service
├─2871 /usr/sbin/keepalived -D
├─2872 /usr/sbin/keepalived -D
└─2873 /usr/sbin/keepalived -D
Jun 17 08:30:20 localhost.localdomain Keepalived_healthcheckers[2872]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 17 08:30:20 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) setting protocol VIPs.
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
Jun 17 08:30:21 localhost.localdomain Keepalived_vrrp[2873]: Sending gratuitous ARP on ens33 for 172.17.0.238
通過模擬VM1，VM3關機，系統正常工作，恢復後，系統負載正常；