HaProxy+keepalived+mycat叢集高可用配置
部署圖
叢集部署圖的理解：
1、keepalived和haproxy必須裝在同一臺機器上（如172.17.210.210.83機器上，keepalived和haproxy都要安裝），keepalived負責為該伺服器搶佔vip（虛擬ip），搶佔到vip後，對該主機的訪問可以通過原來的ip（172.17.210.210.83）訪問，也可以直接通過vip（172.17.210.210.103）訪問。
2、172.17.210.64上的keepalived也會去搶佔vip，搶佔vip時有優先順序，配置keepalived.conf中的（priority 150 #數值愈大，優先順序越高,172.17.210.64上改為120，master和slave上該值配置不同）決 定。但是一般哪臺主機上的keepalived服務先啟動就會搶佔到vip，即使是slave，只要先啟動也能搶到。
3、haproxy負責將對vip的請求分發到mycat上。起到負載均衡的作用，同時haproxy也能檢測到mycat是否存活，haproxy只會將請求轉發到存活的mycat上。
4、如果一臺伺服器（keepalived+haproxy伺服器）宕機，另外一臺上的keepalived會立刻搶佔vip並接管服務。
如果一臺mycat伺服器宕機，haporxy轉發時不會轉發到宕機的mycat上，所以mycat依然可用。 Haproxy安裝 haproxy安裝 useraddhaproxy #wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.25.tar.gz # tar zxvf haproxy-1.4.25.tar.gz # cd haproxy-1.4.25 # make TARGET=linux26 PREFIX=/usr/local/haproxy ARCH=x86_64 # make install PREFIX=/usr/local/haproxy #cd /usr/local/haproxy #chown -R haproxy.haproxy *
haproxy.cfg haproxy.cfg #cd /usr/local/haproxy #touch haproxy.cfg #vi/usr/local/haproxy/haproxy.cfg global log 127.0.0.1 local0 ##記日誌的功能 maxconn 4096 chroot/usr/local/haproxy user haproxy group haproxy daemon defaults log global option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000 listen admin_status 172.17.210.103:48800 ##VIP stats uri/admin-status ##統計頁面 stats auth admin:admin mode http option httplog listen allmycat_service 172.17.210.103:8096 ##轉發到mycat的8066埠，即mycat的服務埠 mode tcp option tcplog option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www balance roundrobin server mycat_133 172.17.210.133:8066 check port 48700 inter 5s rise 2 fall 3 server mycat_134 172.17.210.134:8066 check port 48700 inter 5s rise 2 fall 3 srvtimeout 20000 listen allmycat_admin 172.17.210.103:8097 ##轉發到mycat的9066埠，及mycat的管理控制檯埠 mode tcp option tcplog option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www balance roundrobin server mycat_133 172.17.210.133:9066 check port 48700 inter 5s rise 2 fall 3 server mycat_83 172.17.210.134:9066 check port 48700 inter 5s rise 2 fall 3 srvtimeout 20000 haproxy記錄日誌
預設haproxy是不記錄日誌的，為了記錄日誌還需要配置syslog模組，在linux下是rsyslogd服務，yum –y install rsyslog先安裝rsyslog，然後 記錄haproxy日誌的配置 #cd /etc/rsyslog.d/ 如果沒有這個目錄，新建 #cd /etc #mkdir rsyslog.d #cd /etc/rsyslog.d/ #touch haproxy.conf #vi /etc/rsyslog.d/haproxy.conf $ModLoad imudp $UDPServerRun 514 local0.* /var/log/haproxy.log #vi /etc/rsyslog.conf 1、在#### RULES ####上面一行的地方加入以下內容： # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### 2、在local7.* /var/log/boot.log的下面加入以下內容（增加後的效果如下）： # Save boot messages also to boot.log local7.* /var/log/boot.log local0.* /var/log/haproxy.log
儲存，重啟rsyslog服務
service rsyslog restart
現在你就可以看到日誌（/var/log/haproxy.log）了
配置監聽mycat是否存活
在Mycat server1 Mycat server2上都需要新增檢測埠48700的指令碼，為此需要用到xinetd，xinetd為linux系統的基礎服務，
首先在xinetd目錄下面增加指令碼與埠的對映配置檔案
1、如果xinetd沒有安裝，使用如下命令安裝：
yum install xinetd -y
2、檢查/etc/xinetd.conf的末尾是否有這一句：includedir /etc/xinetd.d
沒有就加上，
3、檢查 /etc/xinetd.d資料夾是否存在，不存在也加上
#cd /etc
#mkdir xinetd.d
4、增加 /etc/xinetd.d/mycat_status 監聽mycat是否存活的配置 #cd /etc #mkdir xinetd.d #cd /etc/xinetd.d/ #touch mycat_status #vim /etc/xinetd.d/mycat_status service mycat_status { flags = REUSE socket_type = stream port = 48700 wait = no user = root server =/usr/local/bin/mycat_status log_on_failure += USERID disable = no }
5、/usr/local/bin/mycat_status指令碼
mycat_status指令碼 #!/bin/bash #/usr/local/bin/mycat_status.sh # This script checks if a mycat server is healthy running on localhost. It will # return: # # "HTTP/1.x 200 OK\r" (if mycat is running smoothly) # # "HTTP/1.x 503 Internal Server Error\r" (else) mycat=`/usr/local/mycat/bin/mycatstatus | grep'not running' | wc -l` if [ "$mycat" = "0" ]; then /bin/echo-e "HTTP/1.1 200 OK\r\n" else /bin/echo-e "HTTP/1.1 503 Service Unavailable\r\n" fi
4、/etc/services中加入mycat_status服務
加入mycat_status服務 #cd /etc #vi services 在末尾加入 mycat_status 48700/tcp # mycat_status 儲存 重啟xinetd服務 service xinetd restart
5、驗證mycat_status服務是否啟動成功
驗證mycat_status服務是否啟動成功 #netstat -antup|grep 48700 如果成功會現實如下內容： [[email protected] log]# netstat -antup|grep 48700 tcp 0 0 :::48700 :::* LISTEN 12609/xinetd
啟動haproxy
啟動haproxy前必須先啟動keepalived，否則啟動不了。
/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg
啟動haproxy異常情況
如果報以下錯誤：
[[email protected] bin]# /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg
[ALERT] 183/115915 (12890) : Starting proxy admin_status: cannot bind socket
[ALERT] 183/115915 (12890) : Starting proxy allmycat_service: cannot bind socket
[ALERT] 183/115915 (12890) : Starting proxy allmycat_admin: cannot bind socket
原因為：該機器沒有搶佔到vip
為了使用方便可以增加一個啟動，停止haproxy的指令碼
啟動指令碼starthap內容如下
#!/bin/sh
/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg &
停止指令碼stophap內容如下
#!/bin/sh
ps -ef | grep sbin/haproxy | grep -v grep |awk '{print $2}'|xargs kill -s 9
分別賦予啟動許可權
chmod +x starthap
chmod +x stophap
啟動後可以通過http://172.17.210.103:48800/admin-status (使用者名稱密碼都是admin，haproxy.cfg中配置的)
openssl安裝
openssl必須安裝，否則安裝keepalived時無法編譯，keepalived依賴openssl。
openssl安裝 tar zxvf openssl-1.0.1g.tar.gz ./config--prefix=/usr/local/openssl ./config-t make depend make make test make install ln -s /usr/local/openssl /usr/local/ssl
openssl配置 vi /etc/ld.so.conf #在/etc/ld.so.conf檔案的最後面，新增如下內容： /usr/local/openssl/lib vi /etc/profile export OPENSSL=/usr/local/openssl/bin export PATH=$PATH:$OPENSSL source /etc/profile yum installopenssl-devel -y #如無法yum下載安裝，請修改yum配置檔案
測試: ldd /usr/local/openssl/bin/openssl linux-vdso.so.1 => (0x00007fff996b9000) libdl.so.2 =>/lib64/libdl.so.2 (0x00000030efc00000) libc.so.6 =>/lib64/libc.so.6 (0x00000030f0000000) /lib64/ld-linux-x86-64.so.2 (0x00000030ef800000) which openssl /usr/bin/openssl openssl version OpenSSL 1.0.0-fips 29 Mar 2010 keepalived安裝
本文在172.17.30.64、172.17.30.83兩臺機器進行keepalived安裝 安裝 tar zxvf keepalived-1.2.13.tar.gz cd keepalived-1.2.13 ./configure--prefix=/usr/local/keepalived make make install cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived/etc/init.d/ mkdir /etc/keepalived cd /etc/keepalived/ cp /usr/local/keepalived/etc/keepalived/keepalived.conf/etc/keepalived mkdir-p /usr/local/keepalived/var/log keepalived配置 建檢查haproxy是否存活的指令碼 #新建redis檢查 mkdir /etc/keepalived/scripts cd /etc/keepalived/scripts
keepalived.conf:
vi /etc/keepalived/keepalived.conf
Master: Master ! Configuration Filefor keepalived vrrp_script chk_http_port { script"/etc/keepalived/scripts/check_haproxy.sh" interval 2 weight 2 } vrrp_instance VI_1 { state MASTER #172.17.210.83上改為Master interface eth0 #對外提供服務的網路介面 virtual_router_id 51 #VRRP組名，兩個節點的設定必須一樣，以指明各個節點屬於同一VRRP組 priority 150 #數值愈大，優先順序越高,172.17.210.84上改為120 advert_int 1 #同步通知間隔 authentication { #包含驗證型別和驗證密碼。型別主要有PASS、AH兩種，通常使用的型別為PASS，據說AH使用時有問題 auth_type PASS auth_pass 1111 } track_script { chk_http_port #呼叫指令碼check_haproxy.sh檢查haproxy是否存活 } virtual_ipaddress { #vip地址，這個ip必須與我們在lvs客戶端設定的vip相一致 172.17.210.103 dev eth0 scope globa } notify_master/etc/keepalived/scripts/haproxy_master.sh notify_backup/etc/keepalived/scripts/haproxy_backup.sh notify_fault /etc/keepalived/scripts/haproxy_fault.sh notify_stop /etc/keepalived/scripts/haproxy_stop.sh }
slave： slave ! Configuration Filefor keepalived vrrp_script chk_http_port { script"/etc/keepalived/scripts/check_haproxy.sh" interval 2 weight 2 } vrrp_instance VI_1 { state MASTER #172.17.210.83上改為Master interface eth1 #對外提供服務的網路介面 virtual_router_id 51 #VRRP組名，兩個節點的設定必須一樣，以指明各個節點屬於同一VRRP組 priority 120 #數值愈大，優先順序越高,172.17.210.64上改為120 advert_int 1 #同步通知間隔 authentication { #包含驗證型別和驗證密碼。型別主要有PASS、AH兩種，通常使用的型別為PASS，據說AH使用時有問題 auth_type PASS auth_pass 1111 } track_script { chk_http_port #呼叫指令碼check_haproxy.sh檢查haproxy是否存活 } virtual_ipaddress { #vip地址，這個ip必須與我們在lvs客戶端設定的vip相一致 172.17.210.103 dev eth1 scope globa } notify_master/etc/keepalived/scripts/haproxy_master.sh notify_backup/etc/keepalived/scripts/haproxy_backup.sh notify_fault /etc/keepalived/scripts/haproxy_fault.sh notify_stop /etc/keepalived/scripts/haproxy_stop.sh }
check_haproxy.sh
vi /etc/keepalived/scripts/check_haproxy.sh
指令碼含義：如果沒有haproxy程序存在，就啟動haproxy，停止keepalived check_haproxy.sh #!/bin/bash STARTHAPROXY="/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg" STOPKEEPALIVED="/etc/init.d/keepalived stop" LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log" echo "[check_haproxy status]" >> $LOGFILE A=`ps-C haproxy --no-header |wc-l` echo "[check_haproxy status]" >> $LOGFILE date >> $LOGFILE if [ $A -eq 0 ];then echo $STARTHAPROXY >> $LOGFILE $STARTHAPROXY >> $LOGFILE 2>&1 sleep5 fi if [ `ps -C haproxy --no-header |wc-l` -eq 0 ];then exit 0 else exit 1 fi
haproxy_master.sh(master和slave一樣) haproxy_master.sh #!/bin/bash STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy-f /usr/local/haproxy/haproxy.cfg` STOPHAPROXY=`ps-ef | grep sbin/haproxy | grep -vgrep |awk'{print $2}'|xargskill -s 9` LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log" echo "[master]" >> $LOGFILE date >> $LOGFILE echo "Being master...." >> $LOGFILE 2>&1 echo "stop haproxy...." >> $LOGFILE 2>&1 $STOPHAPROXY >> $LOGFILE 2>&1 echo "start haproxy...." >> $LOGFILE 2>&1 $STARTHAPROXY >> $LOGFILE 2>&1 echo "haproxy stared ..." >> $LOGFILE
haproxy_backup.sh(master和slave一樣) haproxy_backup.sh #!/bin/bash STARTHAPROXY=`/usr/local/haproxy/sbin/haproxy-f /usr/local/haproxy/haproxy.cfg` STOPHAPROXY=`ps-ef | grep sbin/haproxy | grep -vgrep |awk'{print $2}'|xargskill -s 9` LOGFILE="/usr/local/keepalived/var/log/keepalived-haproxy-state.log" echo "[backup]" >> $LOGFILE date >> $LOGFILE echo "Being backup...." >> $LOGFILE 2>&1 echo "stop haproxy...." >> $LOGFILE 2>&1 $STOPHAPROXY >> $LOGFILE 2>&1 echo "start haproxy...." >> $LOGFILE 2>&1 $STARTHAPROXY >> $LOGFILE 2>&1 echo "haproxy stared ..." >> $LOGFILE
haproxy_fault.sh(master和slave一樣) haproxy_fault.sh #!/bin/bash LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log echo "[fault]" >> $LOGFILE date >> $LOGFILE
haproxy_stop.sh(master和slave一樣) haproxy_stop.sh #!/bin/bash LOGFILE=/usr/local/keepalived/var/log/keepalived-haproxy-state.log echo "[stop]" >> $LOGFILE date >> $LOGFILE