drbd+nfs+heartbeat高可用
DRBD是一種基於軟件的、基於網絡的塊復制存儲解決方案
DRBD在IP網絡傳輸，所有在集中使用DRBD作為共享存儲設備，不需要任何硬件投資，可以節約很多成本
NFS1
IP1:10.10.10.166 心跳和數據傳輸網卡 不配置網關，添加路由即可
添加路由：route add -host IP dev eth0並且寫在rc.local內
VIP:10.10.10.150
DRBD至少使用兩個分區需要單獨分出來
NFS2
IP1:10.10.10.167
VIP:10.10.10.50
配置主機名、IP、hosts文件、關閉selinux、iptables
安裝drbd源
CentOS 7
yum install glibc* -y
# rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
CentOS 6
rpm -Uvh elrepo-release-6-6.el6.elrepo.noarch.rpm
或者
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
配置阿裏雲源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
CentOS 6
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
CentOS 7
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache
配置DRBD
兩臺主機分別磁盤分區
parted -s /dev/sdb mklabel gpt #分區表轉換為GPT
parted -s /dev/sdb mkpart primary 0% 80%
parted -s /dev/sdb mkpart primary 81% 100%
打印分區結果
[[email protected] ~]# parted /dev/sdb p
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name 標誌
1 1049kB 17.2GB 17.2GB primary
2 17.4GB 21.5GB 4079MB primary
安裝DRBD
分出兩個區，一個數據同步使用sdb1，一個記錄日誌sdb2
fdisk /dev/ddb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
artition number (1-4): 1
First cylinder (1-2610, default 1): 回車
Last cylinder, +cylinders or +size{K,M,G} (1305-2610, default 2610):+10G
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
在執行執行一次分出sdb2 w保存分區
升級內核yum install kernel kernel-devel kernel-headers -y
yum install kmod-drbd83 drbd83 -y
加載DRBD到內核
modprobe drbd
加載不成功執行depmod然後重啟
echo "modprobe drbd > /dev/null 2>&1" > /etc/sysconfig/modules/drbd.modules
檢查是否安裝成功
lsmod | grep -i drbd
查看drbd.ko安裝路徑
modprobe -l | grep -i drbd
安裝成功之後drbd相關工具(drbdadm,drbdsetup)被安裝在/sbin/目錄下
配置drbd
vim /etc/drbd.confi
#include "drbd.d/global_common.conf";
#include "drbd.d/*.res";
global {
usage-count no;
}
common {
syncer { rate 200M; }
}
resource nfsha {
protocol C;
startup {
wfc-timeout 120;
degr-wfc-timeout 120;
}
net {
cram-hmac-alg "sha1";
shared-secret "nfs-ha";
}
disk {
on-io-error detach;
fencing resource-only;
}
device /dev/drbd0;
on nfs1 {
disk /dev/sdb1;
address 10.10.10.166:7788;
meta-disk internal;
}
on nfs2 {
disk /dev/sdb1;
address 10.10.10.167:7788;
meta-disk internal;
}
}
解釋：
global {
usage-count no; #允不允許官方統計
}
common {
syncer {rate 200M; } //設置主備節點同步時速率的最大值單位是字節
}
resource r0 { //資源名稱
protocol C; //使用的協議
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
wfc-timeout 120;
degr-wfc-timeout 120;
}
net {
cram-hmac-alg "sha1"; //DRBD同步時使用的驗證方式和密碼信息
shared-secret "nfsha";
}
disk { //使用dpod功能保證在數據不同步時不進行切換
on-io-error detach;
fencing resource-only;
}
device /dev/drbd0;
on master-drbd { //每個主機的說明以on開頭後面是hostname(與uname -n一樣)
disk /dev/sdb1; // drbd0使用的磁盤分區是/dev/sdb1
address 192.168.100.10:7788; //
meta-disk internal; //DRBD的源數據存方式
}
on slave-drbd {
disk /dev/sdb1;
address 192.168.100.20:7788;
meta-disk internal;
}
}
復制配置信息到另一臺主機
啟動DRBD(同時)
首先創建供DRBD記錄信息的數據塊
兩臺主機分別執行drbdadm create-md nfsha或者drbdadm create-md all
啟動DRBD service drbd start
查看節點狀態
cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5242684
ro表示角色信息，第一次啟動時都為Secondary(備)狀態
ds表示磁盤信息，Inconsistent表示磁盤數據不一致狀態
ns表示網絡發送的數據包信息
dw表示磁盤寫信息
dr表示磁盤讀信息
設置主用節點
drbdsetup /dev/drbd0 primary -o
drbdadm primary r0或者 drbdadm primary all
設置從 drbdadm secondary nfsha
啟動之後再次查看狀態
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:4812800 nr:0 dw:0 dr:4813472 al:0 bm:293 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:429884
[=================>..] sync‘ed: 91.9% (416/5116)M
finish: 0:00:10 speed: 40,304 (38,812) K/sec
sync‘ed：同步進度
等待一段時間再次查看(同步時間可能很長)
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:5242684 nr:0 dw:0 dr:5243356 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
ds狀態變為UpToDate/UpToDate說明成功了
掛載DRBD設備
mount只能用在主設備上，因此只要主設備才能格式化掛載。
要在備設備上掛載，必須先卸載主，在升級備為主，然後再掛載
主設備進行格式化
mkfs.ext4 /dev/drbd0
tune2fs -c -1 /dev/drbd0
掛載
mkdir /data
mount /dev/drbd0 /data
關閉DRBD開機自啟動
測試
dd if=/dev/zero of=/data/test.file bs=100M count=2
查看備用節點是否同步
主操作
umount /data
drbdadm secondary all
備用節點升級為主
drbdadm primary nfsha
mount /dev/drbd0 /data
掛載查看是否含有test.file
安裝Heartbeat
配置vip
cd /etc/sysconfig/network-scripts/
cp ifcfg-eth0 ifcfg-eth0:0
DEVICE=eth0:0
ONBOOT=yes
IPADDR=10.10.10.50
NETMASK=255.255.255.255
yum install pam-devel -y
yum install python-devel -y
yum install gcc-c++ -y
yum install glib* -y
yum install libxslt* -y
yum install tkinter* -y
yum install elfutils* -y
yum install lm_sensors* -y
yum install perl-Compress* perl-libwww* perl-HTML* perl-XML* perl-Net* perl-IO* perl-Digest* -y
yum install bzip2* -y
yum install ncurses* -y
yum install imake* -y
yum install autoconf* -y
yum install flex -y
yum install beecrypt* -y
yum install net-snmp* -y
yum install perl-LDAP-0.40-1.el6.noarch.rpm -y
yum install perl-Parse-* perl-Mail-DKIM* -y
yum install libnet* -y
yum install openssl openssl-devel -y
tar xf libnet-1.1.2.1.tar.gz
cd libnet
./configure
make &&make install
cd ..
tar xf heartbeat-2.0.7.tar.gz
cd heartbeat-2.0.7
./ConfigureMe configure --enable-fatal-warnings=no --disable-swig --disable-snmp-subagent
./ConfigureMe make --enable-fatal-warnings=no || gmake
make install
cd /usr/share/doc/heartbeat-2.0.7/
cp ha.cf haresources authkeys /etc/ha.d/
cd /etc/ha.d/
設置heartbeat配置文件
(nfs1)
編輯ha.cf，添加下面配置：
# vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 5
ucast eth0 10.10.10.167 # 指定對方網卡及IP
auto_failback off
node nfs1 nfs2
(nfs2)
編輯ha.cf，添加下面配置：
# vi /etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 5
ucast eth0 10.10.10.166
auto_failback off
node nfs1 nfs2
編輯雙機互聯驗證文件authkeys，添加以下內容：(node1,node2)
# vi /etc/ha.d/authkeys
auth 1
1 crc
給驗證文件600權限
# chmod 600 /etc/ha.d/authkeys
編輯集群資源文件：(nfs1，nfs2)
# vi /etc/ha.d/haresources
nfs1 IPaddr::10.10.10.50/24/eth0 drbddisk::nfsha Filesystem::/dev/drbd0::/data::ext4 killnfsd
這裏ip為虛ip 註意網卡（eth0）資源名（nfsha）和掛載點（data）
註：該文件內IPaddr,Filesystem等腳本存放路徑在/etc/ha.d/resource.d/下，也可在該目錄下存放服務啟動腳本（例如：mysql,www），將相同腳本名稱添加到/etc/ha.d/haresources內容中，從而跟隨heartbeat啟動而啟動該腳本。
IPaddr::10.10.10.0/24/eth0：用IPaddr腳本配置對外服務的浮動虛擬IP
drbddisk::nfsha：用drbddisk腳本實現DRBD主從節點資源組的掛載和卸載
Filesystem::/dev/drbd0::/data::ext4：用Filesystem腳本實現磁盤掛載和卸載
編輯腳本文件killnfsd,用來重啟NFS服務：(node1,node2)
# vi /etc/ha.d/resource.d/killnfsd
killall -9 nfsd; /etc/init.d/nfs restart;exit 0
賦予755執行權限：
chmod 755 /etc/ha.d/resource.d/killnfsd
創建DRBD腳本文件drbddisk:(nfs1，nfs2)
編輯drbddisk，添加下面的腳本內容
# vi /etc/ha.d/resource.d/drbddisk
#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# Copright 2003-2008 LINBIT Information Technologies
# Philipp Reisner, Lars Ellenberg
#
###
DEFAULTFILE="/etc/default/drbd"
DRBDADM="/sbin/drbdadm"
if [ -f $DEFAULTFILE ]; then
. $DEFAULTFILE
fi
if [ "$#" -eq 2 ]; then
RES="$1"
CMD="$2"
else
RES="all"
CMD="$1"
fi
## EXIT CODES
# since this is a "legacy heartbeat R1 resource agent" script,
# exit codes actually do not matter that much as long as we conform to
# http://wiki.linux-ha.org/HeartbeatResourceAgent
# but it does not hurt to conform to lsb init-script exit codes,
# where we can.
# http://refspecs.linux-foundation.org/LSB_3.1.0/
#LSB-Core-generic/LSB-Core-generic/iniscrptact.html
####
drbd_set_role_from_proc_drbd()
{
local out
if ! test -e /proc/drbd; then
ROLE="Unconfigured"
return
fi
dev=$( $DRBDADM sh-dev $RES )
minor=${dev#/dev/drbd}
if [[ $minor = *[!0-9]* ]] ; then
# sh-minor is only supported since drbd 8.3.1
minor=$( $DRBDADM sh-minor $RES )
fi
if [[ -z $minor ]] || [[ $minor = *[!0-9]* ]] ; then
ROLE=Unknown
return
fi
if out=$(sed -ne "/^ *$minor: cs:/ { s/:/ /g; p; q; }" /proc/drbd); then
set -- $out
ROLE=${5%/**}
: ${ROLE:=Unconfigured} # if it does not show up
else
ROLE=Unknown
fi
}
case "$CMD" in
start)
# try several times, in case heartbeat deadtime
# was smaller than drbd ping time
try=6
while true; do
$DRBDADM primary $RES && break
let "--try" || exit 1 # LSB generic error
sleep 1
done
;;
stop)
# heartbeat (haresources mode) will retry failed stop
# for a number of times in addition to this internal retry.
try=3
while true; do
$DRBDADM secondary $RES && break
# We used to lie here, and pretend success for anything != 11,
# to avoid the reboot on failed stop recovery for "simple
# config errors" and such. But that is incorrect.
# Don‘t lie to your cluster manager.
# And don‘t do config errors...
let --try || exit 1 # LSB generic error
sleep 1
done
;;
status)
if [ "$RES" = "all" ]; then
echo "A resource name is required for status inquiries."
exit 10
fi
ST=$( $DRBDADM role $RES )
ROLE=${ST%/**}
case $ROLE in
Primary|Secondary|Unconfigured)
# expected
;;
*)
# unexpected. whatever...
# If we are unsure about the state of a resource, we need to
# report it as possibly running, so heartbeat can, after failed
# stop, do a recovery by reboot.
# drbdsetup may fail for obscure reasons, e.g. if /var/lock/ is
# suddenly readonly. So we retry by parsing /proc/drbd.
drbd_set_role_from_proc_drbd
esac
case $ROLE in
Primary)
echo "running (Primary)"
exit 0 # LSB status "service is OK"
;;
Secondary|Unconfigured)
echo "stopped ($ROLE)"
exit 3 # LSB status "service is not running"
;;
*)
# NOTE the "running" in below message.
# this is a "heartbeat" resource script,
# the exit code is _ignored_.
echo "cannot determine status, may be running ($ROLE)"
exit 4 # LSB status "service status is unknown"
;;
esac
;;
*)
echo "Usage: drbddisk [resource] {start|stop|status}"
exit 1
;;
esac
exit 0
賦予755執行權限：
chmod 755 /etc/ha.d/resource.d/drbddisk
啟動HeartBeat服務
在兩個節點上啟動HeartBeat服務，先啟動node1：(node1,node2)
service heartbeat start
現在從其他機器能夠ping通虛IP 192.168.0.190，表示配置成功
配置NFS:(nfs1,nfs2)
安裝nfs yum install nfs-utils rpcbind
編輯exports配置文件，添加以下配置：
# vi /etc/exports
/data *(rw,no_root_squash)
重啟NFS服務：
service rpcbind restart
service nfs restart
chkconfig rpcbind on
chkconfig nfs off
註：這裏設置NFS開機不要自動運行，因為/etc/ha.d/resource.d/killnfsd 該腳本會控制NFS的啟動。
五、測試高可用
1、正常熱備切換
在客戶端掛載NFS共享目錄
# mount -t nfs 192.168.0.190:/store /tmp
模擬將主節點node1 的heartbeat服務停止，則備節點node2會立即無縫接管；測試客戶端掛載的NFS共享讀寫正常。
此時備機node2上的DRBD狀態：
# service drbd status
drbd driver loaded OK; device status:
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by [email protected], 2015-05-12 21:05:41
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /store ext4
異常宕機切換
強制關機，直接關閉node1電源
node2節點也會立即無縫接管，測試客戶端掛載的NFS共享讀寫正常。
此時node2上的DRBD狀態：
# service drbd status
drbd driver loaded OK; device status:
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by [email protected], 2015-05-12 21:05:41
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Unknown UpToDate/DUnknown C /store ext4
nfs heartbeat、drbd不要設置成開機自啟動
主宕機後先啟動drbd 然後再起的heartbeat
drbd+nfs+heartbeat高可用