簡介

    haproxy為高效能的反向代理伺服器，在向後端server排程方面支援很多的排程演算法：roundrobin，source，uri，url_param，hdr()，leastconn等，且支援動態權重調整的一致性hash，在後端為cache server時很好的提高cache命中率；所以相比nginx反向代理，haproxy排程功能更為強大，另外haproxy提供自帶的gui介面，方便直接在web頁面管理前、後端伺服器，只需要在配置檔案中開啟即可。

    nginx自身在處理web靜態頁面方面非常強大，其採用模組化設計，有較好的擴充套件性，但不支援模組動態裝解除安裝（Tengine支援模組動態裝解除安裝），nginx具有高可靠性，支援熱部署，低記憶體消耗等優點，其在設計之初就吸納了最新的技術，如支援事件驅動、非同步IO等，併發處理能力強大。

    apache結合php處理動態頁面功能強大，php可直接以模組的形式載入到httpd，而nginx需要通過fastcgi協議與後端php通訊。所以結合各元件優點，我們這裡採用haproxy(支援https)+nginx(靜態)+lamp(動態)架構，最後通過keepalived實現對haproxy高可用。此外本實驗中對phpBB全站點使用https協議。

下圖為本次實驗架構圖：

                                                                                                                        圖1

                                                                                                                    圖2

架構圖說明

這裡有2幅架構圖，圖1中，後端各主機均在同一網段，這也是我們這次實驗所採用的架構；圖2中後端static server group和dynamic server group不在同一網段，二者之間需要路由器相連，Router的地址：eth0 --> 192.168.100.80 eth1 --> 192.168.200.80非但如此，採用圖2架構還需在haproxy和各後端server上新增靜態路由：

haproxy1：  route add -net 192.168.200.0/24 gw 192.168.100.80 dev eth1

haproxy2：  route add -net 192.168.100.0/24 gw 192.168.200.80 dev eth1

Nginx1：      routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0

Nginx2：      routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0

LAMP1：      routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0

LAMP2：      routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0

為簡便，這裡我們採用架構圖1。這裡我們用virtualbox準備6臺linux主機，各主機IP地址、Gateway及所需安裝軟體等如下表所配置，這裡我們將192.168.0.0/24網路模擬為公網，192.168.100.0/24為伺服器內網網段。前端為2臺haproxy組成的反向代理伺服器，並實現雙主高可用：所謂雙主，即haproxy1和haproxy2都正常工作，均同時為後端server提供反向代理和排程功能，使用者可通過DNS解析後的192.168.0.50和192.168.0.51兩個地址訪問我們的網站。客戶端請求的所有phpBB靜態頁面由前端的haproxy排程至後端的2臺nginx組成的伺服器組響應；使用者請求的所有phpBB動態頁面由前端的haproxy排程至後端的2臺lamp組成的伺服器組響應。

6臺linux主機上所需資源如下：

軟體安裝

按照上面的清單，分別在各主機上部署所需軟體，對於清單中HAProxy需要安裝ntpd伺服器的原因在於keepalived的執行需要後端server與前端伺服器時鐘保持同步，否則對後端server的健康狀態檢測會出現異常。

①、haproxy

haproxy在centos6.8上可以直接使用yum方式：

[[email protected] ~]# yum install -y haproxy

目前系統提供的預設版本是1.5.18，且支援openssl：

[[email protected] ~]# rpm -q haproxy ; ldd $(which haproxy)| grep ssl

haproxy-1.5.18-1.el6.x86_64

libssl.so.10 => /usr/lib64/libssl.so.10(0x00007f05e085c000)

如果是手動編譯安裝，則需要先在官網下載程式包，並新增對openssl的支援，下面是手動編譯步驟：

[[email protected] ~]# yum install -y openssl openssl-devel readline-devel pcre-devel libssl-dev libpcre3
[[email protected] ~]# wget http://www.haproxy.org/download/1.6/src/haproxy-1.6.13.tar.gz
[[email protected] ~]# tar xf haproxy-1.6.13.tar.gz ; cd haproxy-1.6.13
[[email protected] haproxy-1.6.13]# make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
[[email protected] haproxy-1.6.13]# ldd $(which haproxy) | grep ssl
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f05e085c000) 
[[email protected] haproxy-1.6.13]# make install PREFIX=/usr/local/haproxy
[[email protected] haproxy-1.6.13]# vi /etc/profile.d/haproxy.sh
#!/bin/bash
export PATH=$PATH:/usr/local/haproxy/sbin/
[[email protected] haproxy-1.6.13]# source /etc/profile.d/haproxy.sh

建立配置檔案目錄及拷貝並修改啟動指令碼：

[[email protected] haproxy-1.6.13]# mkdir -p /etc/haproxy
[[email protected] haproxy-1.6.13]# cp examples/haproxy.init /etc/rc.d/init.d/haproxy
[[email protected] haproxy-1.6.13]# vi /etc/rc.d/init.d/haproxy
BIN=/usr/local/haproxy/sbin/$BASENAME
CFG=/etc/$BASENAME/$BASENAME.cfg
PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME

至此haproxy已經安裝完畢，最後我們根據架構圖提供haproxy的配置檔案：

[[email protected] ~]# vi /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     40000
    user        haproxy
    group       haproxy
    daemon
    tune.ssl.default-dh-param 2048
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option 		    	        http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  http_frontend
    bind   *:80
    bind *:443 ssl crt /etc/haproxy/haproxy.pem  # 網站證書檔案
    mode http
    log global
    option httpclose
    option logasap
    option dontlognull
    option forwardfor except 127.0.0.1 header X-Client if-none
    capture request header Host len 20
    capture request header Referer len 60
    reqadd X-Forwarded-Proto:\ https
    #rspadd Via:\ haproxy.example.com
    acl url_static path_beg -i /static /images /javascript /stylesheets
    acl url_static path_end -i .jpg .jpeg .png .bmp .gif .css .js
    redirect scheme https if !{ ssl_fc }       # 全站採用https協議
    use_backend static_servers if url_static
    default_backend dynamic_servers
    
    
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static_servers
    mode http
    balance    roundrobin
    #option     httpchk  HEAD  /healthchk.html
    server     static1 192.168.100.64:80 check inter 1000 rise 2 fall 4 weight 1 maxconn 6000
    server     static2 192.168.100.65:80 check inter 1000 rise 2 fall 4 weight 1 maxconn 6000
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
backend dynamic_servers
    mode http
    balance    roundrobin
    #cookie     dynamic_cookie insert nocache indirect
    server     https_dynamic1 192.168.200.70:80 check inter 2000 rise 2 fall 4 weight 1 maxconn 1000 
    server     https_dynamic2 192.168.200.71:80 check inter 2000 rise 2 fall 4 weight 1 maxconn 1000 
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }

listen stats
    bind *:9090
    stats enable
    stats uri /haproxy?stats
    stats hide-version
    stats auth admin:yourpassword
    stats admin if TRUE

②、haproxy.cfg配置說明

haproxy的配置檔案位於：/etc/haproxy/haproxy.cfg，該配置檔案中的一個重點是新增對https協議的支援，為了實現該目的，需要做如下操作：

我們知道，haproxy代理ssl有三種方式：第一種，haproxy 自身提供ssl 證書，僅是客戶端與haproxy之間傳輸https協議，而haproxy與後端的web server仍然傳輸http；第二種為SSL透傳，haproxy 本身工作於tcp傳輸層，https由後端server直接處理，但haproxy不對https報文做任何修改，這就意味著後端server失去了對客戶端IP、埠及使用協議的記錄；第三種，SSL連線在負載均衡器處終止，按需求調整，然後作為新的SSL連線代理到後端伺服器。這可能會提供最大的安全性和傳送客戶端資訊的能力。這樣做的代價是更多的CPU能耗和稍複雜的配置。實驗中我們採用第一種，所以我們需要提供haproxy的ssl證書檔案：

因為是實驗，所以我們需要生成自簽署證書cacert.pem、應用程式haproxy證書haproxy.crt,應用程式haproxy私鑰haproxy.key等檔案，具體私鑰及證書生成步驟這裡不再詳述。最後將haproxy.crt和haproxy.key合併為一個檔案：

[[email protected] ~]# cd /etc/haproxy

[[email protected] haproxy]# cat haproxy.crt haproxy.key | tee haproxy.pem

而上面生成的haproxy.pem正是haproxy.cfg所需要的：

bind *:443 ssl crt /etc/haproxy/haproxy.pem

③、keepalived安裝與配置

我們需要在前端部署有haproxy的兩臺server上安裝keepalived，其安裝過程較為簡單，直接使用yum安裝即可：

[[email protected] ~]# yum install -y keepalived

[[email protected] ~]# rpm -q keepalived

keepalived-1.2.13-5.el6_6.x86_64

haproxy1上的keepalived配置：

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id www.example.com

}

vrrp_script chk_haproxy {
   script "killall -0 haproxy &> /dev/null"
   interval 1
   weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 256f6df6
    }
    virtual_ipaddress {
       192.168.0.50/24 dev eth0 label eth0:0
    }
    track_script {  
        chk_haproxy
    } 
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}

vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 52
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 19afa2c5
    }
    virtual_ipaddress {
       192.168.0.51/24 dev eth0 label eth0:1
    }
    track_script {  
        chk_haproxy
    } 
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}

haproxy2上的keepalived配置：

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id centos66.example.com
}
vrrp_script chk_haproxy {
   script "killall -0 haproxy &> /dev/null"
   interval 1
   weight -20
   }

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 256f6df6
    }
    virtual_ipaddress {
        192.168.0.50/24 dev eth0 label eth0:0
    }
    track_script {  
        chk_haproxy
    } 
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 19afa2c5
    }
    virtual_ipaddress {
        192.168.0.51/24 dev eth0 label eth0:1
    }
    track_script {  
        chk_haproxy
    } 
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}

訊息通知指令碼notify.sh：

#!/bin/bash
#Date:2017-10-07
vip=192.168.0.50
contact="[email protected]"

notify() {
   mailsubject="`hostname` to be $1: $vip floating"
   mailbody="`date +'%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"
   echo "$mailbody" | mail -s "$mailsubject" $contact
}
case "$1" in 
master)
    notify master
    exit 0
    ;;
backup)
    notify backup
    exit 0
    ;;
fault)
    notify fault
    exit 0
    ;;
*)
    echo "Usage: `basename $0` {master|backup|fault}"
    exit 1 
    ;;
esac

④、ntpd的安裝與配置

由於前端keepalived與後端各server通訊需要保證時鐘同步，所以我們需要在前端的其中一臺主機上安裝ntpd伺服器，注意本次實驗採用的是架構圖1，如果是架構圖2，那麼前端的2臺haproxy伺服器上均要部署ntpd。

[[email protected] ~]# yum install -y ntpd
[[email protected] ~]# vi /etc/ntpd.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

restrict ntp1.aliyun.com
restrict ntp2.aliyun.com
restrict ntp3.aliyun.com
restrict ntp4.aliyun.com
restrict ntp5.aliyun.com
# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1 
restrict -6 ::1
restrict 192.168.100.0 mask 255.255.255.0 nomodify
server ntp1.aliyun.com prefer
server ntp2.aliyun.com
server ntp3.aliyun.com
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys

安裝完成後啟動即可：service ntpd start ;chkconfig ntpd on

後端各server新增crontab任務計劃：

[[email protected] ~]# crontab -e

MAILTO=""

*/10 * * * * /usr/sbin/ntpdate 192.168.100.16 &> /dev/null

這樣設定完成後，後端的各server每10分鐘就會與前端的ntpd進行一次時鐘同步。

⑤、Static Server Group

靜態伺服器組的組成是由各linux主機上部署nginx來完成，在centos6.8上也可以直接使用yum來安裝：

[[email protected] ~]# yum install -y nginx

然後對後端各主機上的nginx配置檔案進行配置：

[[email protected] ~]# vi /etc/nginx/nginx.conf
# nginx.conf
user  nginx;
worker_processes  1;
worker_rlimit_nofile 51200;
pid /var/run/nginx.pid;
events {
    worker_connections  51200;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    error_log /var/log/nginx/error.log  notice;

    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /var/www/html/phpBB;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

      }

  } 

⑥、Dynamic Server Group

在後端各server上需要部署httpd+mariadb+php，在centos7上安裝過程比較簡單，直接使用yum安裝即可，這裡不再具體給出各軟體安裝步驟，我們需要在/etc/httpd/conf.d中新增虛擬主機：

[[email protected] ~]# cd /etc/httpd/conf.d/
[[email protected] conf.d]# vi vhosts.conf
<VirtualHost *:80>
   DocumentRoot "/var/www/html"
   ServerName  ftp.example.com
</VirtualHost>

<Directory "/var/www/html/phpBB">
   Options Includes ExecCGI FollowSymLinks
   AllowOverride None
   Require all granted
</Directory>

<VirtualHost *:80>
   DocumentRoot "/var/www/html/phpBB"
   ServerName  www.example.com:80
</VirtualHost>

另外需要將配置檔案：/etc/httpd/conf/httpd.conf中的"DocumentRoot"這一行註釋掉並修改日誌格式：

#DocumentRoot "/var/www/html"
LogFormat "%{X-Client}i %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
#SetEnvIf Request_URI "^/test.html(/.*)?$" dontlog
#CustomLog "logs/access_log" combined env=!dontlog

⑦、部署phpBB

從phpBB官網下載安裝包，將安裝包分別拷貝至後端各server：Nginx1,Ngine2,LAMP1,LAMP2上的/var/www/html目錄下，解壓縮並修改所屬使用者和組，這裡以後端的LAMP1來演示，其它各server上的操作步驟與下面類似：

[[email protected] ~]# cp phpBB-3.2.0.tar.bz2 /var/www/html
[[email protected] ~]# cd /var/www/html
[[email protected] html]# tar xf phpBB-3.2.0.tar.bz2
[[email protected] html]# ls
phpBB3 
[[email protected] html]# chown -R apache:apache phpBB3
[[email protected] html]# ln -s phpBB3 phpBB
[[email protected] html]# chown -R apache:apache phpBB

最後我們在LAMP1和LAMP2上建立資料庫

MariaDb[(none)] > create database phpBB;
MariaDb[(none)] > grant all privileges on phpBB.* to [email protected] '127.0.0.1' identified by 'yourpassword';
MariaDb[(none)] > flush privileges;

上面建立了phpBB資料庫名為：phpBB，資料庫使用者名稱：phpBB，密碼：yourpassword，這在後續的phpBB頁面安裝過程中需要。

至此各server配置已經完成，最後就需要我們在瀏覽器安裝phpBB了，在瀏覽器中輸入www.example.com進入安裝。