資料庫管理與高可用----MHA高可用配置及故障切換

實驗思路:
1.MHA架構
1）資料庫安裝
2)一主兩從
3)MHA搭建
故障模擬
1）主庫失效
2）備選主庫成為主庫
3)從庫2將備選主庫指向為主庫
案例環境
1.本案例環境
伺服器CentOS7.4(64位)MHA-manager/192.168.100.130
管理節點，安裝manager元件
伺服器CentOS7.4(64位)Mysq11/192.168.100.150 Master 節點，安裝node元件
伺服器CentOS7.4(64位)Mysql2/192.168.100.160 Slave 節點，安裝node元件
伺服器CentOS7.4(64位)Mysql3/192.168.100.170 Slave節點，安裝node元件
這裡作業系統是CentOS7版本，所以這裡下載MHA版本是0.57版本。
2.案例需求
本案例要求通過MHA監控MySQL資料庫在故障時進行自動切換，不影響業務。
3.案例實現思路
1)安裝MySQL資料庫
2)配置MySQL一主兩從
3)安裝MHA軟體
4)配置無密碼認證
5)配置 MySQL MHA高可用
6)模擬master故障切換
在三臺MySQL 節點上分別安裝資料庫，MySQL版本請使用5.6.36，cmake版本請使
用2.8.6。下面只在Mysql1上面做演示，安裝過程如下。

1.安裝編譯依賴的環境
[root@Mysql1~]# yum install ncurses-devel gcc-c++ perl-Module-Install -y

2.安裝gmake編譯軟體
[root@Mysql1~ ]# tar zxvf cmake-2.8.6.tar.gz
[root@Mysql1 ~]# cd cmake-2.8.6
[root@Mysql1 cmake-2.8.6]# ./configure
[root@Mysql1 cmake-2.8.6]# gmake && gmake install

3.安裝MySQL資料庫
[root@Mysql1~ ]# tar zxvf mysql-5.6.36.tar.gz
[root@Mysql1 ~]# cd mysql-5.6.36

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITII_EXTRA_CHARSETS=all \
-DSYSCONFDTR=/etc

[root@Mysql1 mysql-5.6.36]# make && make install
[root@Mysql1 mysql-5.6.36]# cp support-files/my-default.cnf /etc/my.cnf
[root@Mysql1 mysql-5.6.36]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@Mysql1 ~]# chmod +x /etc/rc.d/init.d/mysqld
[root@MysqI1 ~]# chkconfig --add mysqld
[root@Mysql1 ~]# echo  "PATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
[root@Mysql1 ~]# source /etc/profile
[root@Mysql1 ~]# groupadd mysql
[root@Mysql1 ~]# useradd -M -s /sbin/nologin mysql -g mysql
[root@Mysql1 ~]# chown -R mysql.mysql /usr/local/mysql

source /etc/profile
groupadd mysql
useradd -M -s /sbin/nologin mysql -g mysql
chown -R mysql.mysql /usr/local/mysql

[root@Mysql1 ~]# mkdir -p /data/mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql

4.修改Master的主配置檔案/etc/my.cnf檔案，三臺伺服器的server-id不能一樣
[root@Mysql1 ~]# cat /etc/my.cnf
[mysqld]
server-id = 1
log_bin = master-bin
log-slave-updates = true
配置從伺服器:
在/etc/my.cnf中修改或者增加下面內容。
[root@Mysql2 ~]# vim /etc/my.cnf
server-id = 2
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

5.Mysql1、Mysql2、Mysql3分別做兩個軟連結
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysql /usr/sbin/
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
6、Mysql1、Mysql2、Mysql3啟動MySQL。

systemctl stop firewalld
setenforce 0
/usr/local/mysql/bin/mysqld_safe --user=mysql &
netstat -ntap | grep 3306

7.配置MySQL一主兩從

1).MySQL主從配置相對比較簡單。需要注意的是授權。步驟如下:
在所有資料庫節點上授權兩個使用者，一個是從庫同步使用，另外一個是manager使用。
grant replication slave on*.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;
grant replication slave on *.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;

2).下面三條授權按理論是不用新增的，但是做案例實驗環境時候通過MHA檢查MySQL主從有報錯，
報兩個從庫通過主機名連線不上主庫，所以所有資料庫加上下面的授權。
grant all privileges on *.* to 'mha'@'Mysql1' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql2' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql3' identified by 'manager';
flush privileges;
3).在Mysq1主機上檢視二進位制檔案和同步點
mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000001 |     1294 |              |                  |                   |
+-------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

4.接下來在Mysql2和Mysql3分別執行同步。
mysql> change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000010',master_log_pos=1294;
flush privileges;
start slave;
stop slave;
start slave;
5.檢視IO和SQL執行緒都是yes代表同步是否正常。
mysql> show slave status\G;
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
mysql> show master status;
"檢視二進位制日誌路徑"
mysql> show variables like 'datadir';
+---------------+--------------+
| Variable_name | Value        |
+---------------+--------------+
| datadir       | /home/mysql/ |
+---------------+--------------+
必須設定兩個從庫為只讀模式:
set global read_only=1;
flush privileges;
注意:設定完成直接驗證主從複製功能

--------------------------安裝MHA軟體----------------------------
1.所有伺服器上都安裝MHA依賴的環境，首先安裝epel源。
[root@MHA-manager ~]# yum install epel-release --nogpgcheck -y

yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN

2.MHA 軟體包對於每個作業系統版本不一樣，這裡CentOS7.4必須選擇0.57版本，
在<注意:所有伺服器>上必須先安裝node元件，最後在MHA-manager節點上安裝manager元件，
因為manager依賴node元件，下面都是在Mysql1上操作演示安裝node元件。
//上傳mha4mysql-node-0.57.tar.gz
[root@Mysql1~]# tar zxvf mha4mysql-node-0.57.tar.gz
[root@Mysql1 ~]# cd mha4mysql-node-0.57
[root@Mysql1 mha4mysql-node-o.57]# perl Makefile.PL 
[root@Mysql1 mha4mysql-node-o.57]# make
[root@Mysql1 mha4mysql-node-0.57]# make && make install

3.僅在MHA-manager上安裝manager元件(!注意:一定要先安裝node元件才能安裝manager元件)
[root@Mysql1 mha4mysql-node-0.57]# cd ~
[root@MHA-manager ~]# tar zxvf mha4mysql-manager-0.57.tar.gz
[root@MHA-manager~]# cd mha4mysql-manager-0.57
[root@MHA-manager mha4mysql-manager-0.57]# perl Makefile.PL
[root@MHA-manager mha4mysql-manager-0.57]# make
[root@MHA-manager mha4mysql-manager-o.57]# make && make install

manager安裝後在/usr/local/bin下面會生成幾個工具，主要包括以下幾個:
masterha_check_ssh檢查MHA的SSH配置狀況
masterha_check_repl檢查MySQL複製狀況
masterha_manger啟動manager的指令碼
masterha_check_status檢測當前MHA執行狀態
masterha_master_monitor檢測master是否宕機
masterha_master_switch控制故障轉移（自動或者手動)
masterha_conf_host新增或刪除配置的server資訊
masterha_stop關閉manager

4.node安裝後也會在/usr/local/bin下面會生成幾個指令碼(這些工具通常由MHA
Manager的指令碼觸發，無需人為操作）主要如下:
save_binary_logs儲存和複製master的二進位制日誌
apply_diff_relay_logs 識別差異的中繼日誌事件並將其差異的事件應用於其他的slave
filter_mysqlbinlog去除不必要的ROLLBACK事件(MHA已不再使用這個工具)
purge_relay_logs清除中繼日誌(不會阻塞SQL執行緒)

5.配置無密碼認證
1).在manager上配置到所有資料庫節點的無密碼認證
[root@MHA-manager ~]# ssh-keygen -t rsa    //一路按回車鍵
[root@MHA-manager ~]# ssh-copy-id 192.168.100.150   //輸入yes 及150的root密碼
[root@MHA-manager ~]# ssh-copy-id 192.168.100.160
[root@MHA-manager ~]# ssh-copy-id 192.168.100.170
2).在Mysql1上配置到資料庫節點Mysql2和Mysql3的無密碼認證
[root@Mysql1 ~]# ssh-keygen -t rsa
[root@MysqI1 ~]# ssh-copy-id 192.168.100.160
[root@Mysql1 ~]# ssh-copy-id 192.168.100.170
3).在Mysql2上配置到資料庫節點MysqI1和Mysql3的無密碼認證
[root@Mysql2 ~]# ssh-keygen -t rsa
[root@Mysql2 ~]# ssh-copy-id 192.168.100.150
[root@Mysql2 ~]# ssh-copy-id 192.168.100.170
4).在Mysql3上配置到資料庫節點Mysql1和Mysql2的無密碼認證
[root@Mysql3 ~]# ssh-keygen -t rsa
[root@Mysql3 ~]# ssh-copy-id 192.168.100.150
[root@Mysql3 ~]# ssh-copy-id 192.168.100.160

6.配置MHA

1).在manager節點上覆制相關指令碼到/usr/local/bin目錄。
[root@mha-manager~]# cp -ra /root/mha4mysql-manager-0.57/samples/scripts /usr/local/bin
//拷貝後會有四個執行檔案
[root@MHA-manager ~]#ll/usr/local/bin/scripts/
總用量32
-rwxr-xr-x 1 mysql mysql 36485月312015 master_ip_failover#自動切換時VIP管理的指令碼
-rwxr-xr-x 1 mysql mysql 98725月2509:07 master_ip_online_change #線上切換時vip的管理
-rwxr-xr-x 1 mysql mysql118675月312015 power_manager #故障發生後關閉主機的指令碼
-rwxr-Xr-x 1 mysql mysql 13605月312015 send_report #因故障切換後傳送報警的指令碼

2).複製上述的自動切換時VIP管理的指令碼到/usr/local/bin目錄，這裡使用指令碼管理VIP,
[root@mha-manager~]# cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin

3).修改內容如下:(刪除原有內容，直接複製)

[root@MHA-manager ~]# vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
#############################新增內容部分#########################################
my $vip = '192.168.100.200';
my $brdc = '192.168.100.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
#my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;";
#my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key";
##################################################################################
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

4).建立MHA軟體目錄並拷貝配置檔案。
[root@MHA-manager ~]# cd /usr/local/bin/scripts/
[root@MHA-manager scripts]# cp master_ip_online_change /usr/local/bin/
[root@MHA-manager scripts]# cp send_report /usr/local/
[root@MHA-manager ~]# mkdir /etc/masterha
[root@MHA-manager ~]# cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha
[root@MHA-manager ~]# vim /etc/masterha/app1.cnf

[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/usr/local/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=manager
ping_interval=1
remote_workdir=/tmp
repl_password=123
repl_user=myslave
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.100.160 -s 192.168.100.170
shutdown_script=""
ssh_user=root
user=mha

[server1]    #master節點
hostname=192.168.100.150
port=3306

[server2]    #salve1
hostname=192.168.100.160
port=3306
candidate_master=1
check_repl_delay=0

[server3]    #slave2
hostname=192.168.100.170
port=3306

################配置項解釋
[server default]
manager_log=/var/log/masterha/app1/manager.log  'manager日誌'
manager_workdir=/var/log/masterha/app1    'manager工作目錄'
master_binlog_dir=/usr/local/mysql/data    '#master儲存binlog的位置，這裡的路徑要與master裡配置的binlog的路徑一直，以便mha能找到,這裡也對應編譯安裝資料庫的資料目錄'
master_ip_failover_script=/usr/local/bin/master_ip_failover  '設定自動failover時候的切換指令碼，也就是上邊的那個指令碼'
master_ip_online_change_script=/usr/local/bin/master_ip_online_change '設定手動切換時候的切換指令碼'
password=manager  '#設定mysql中root使用者的密碼，這個密碼是前文中建立監控使用者的那個密碼'
ping_interval=1  '設定監控主庫，傳送ping包的時間間隔，預設是3秒，嘗試三次沒有迴應的時候自動進行rail'
remote_workdir=/tmp  '設定遠端mysql在發生切換時binlog的儲存位置'
repl_password=123  '設定複製使用者的密碼'
repl_user=myslave   '設定複製使用者的使用者'
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.179.124 -s 192.168.179.125 '設定從伺服器的地址'
shutdown_script="" '設定故障發生後關閉故障主機指令碼（該指令碼的主要作用是關閉主機放在發生腦裂,這裡沒有使用)'
ssh_user=root  '設定ssh的登入使用者名稱'
user=mha

[server1]
hostname=192.168.179.123
port=3306

[server2]
candidate_master=1  '設定為候選master'
hostname=192.168.179.124
check_repl_delay=0  '預設情況下如果一個slave落後master 100M的relay logs 的話，mha將不會選擇該slave作為一個新的master'
port=3306

[server3]
hostname=192.168.179.125
port=3306

5).測試ssh無密碼認證，如果正常最後會輸出successfully，如下所示。
[root@mha-manager~]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
Fri Aug 28 12:42:32 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Aug 28 12:42:32 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Starting SSH connection tests..
Fri Aug 28 12:42:38 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.150:22) to [email protected](192.168.100.160:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.150:22) to [email protected](192.168.100.170:22)..
Fri Aug 28 12:42:38 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.160:22) to [email protected](192.168.100.150:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.160:22) to [email protected](192.168.100.170:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.170:22) to [email protected](192.168.100.150:22)..
Fri Aug 28 12:42:34 2020 - [debug]   ok.
Fri Aug 28 12:42:34 2020 - [debug]  Connecting via SSH from [email protected](192.168.100.170:22) to [email protected](192.168.100.160:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [info] All SSH connection tests passed successfully.

//這一步有問題說明祕鑰分發問題

[root@mha-manager ~]# masterha_check_repl -conf=/etc/masterha/app1.cnf

Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.160..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.170..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking master_ip_failover_script status:
Fri Aug 28 16:20:51 2020 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.100.150 --orig_master_ip=192.168.100.150 --orig_master_port=3306 

IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===

Checking the Status of the script.. OK 
Fri Aug 28 16:20:51 2020 - [info]  OK.
Fri Aug 28 16:20:51 2020 - [warning] shutdown_script is not defined.
Fri Aug 28 16:20:51 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

//注意:第一次配置需要去master上手動開啟虛擬IP
[root@Mysql1 ~]# /sbin/ifconfig ens33:1 192.168.100.200/24

7.啟動MHA

[root@mha-manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

--remove_dead_master_conf該引數代表當發生主從切換後，老的主庫的ip將會從配置檔案中移除
--manger_log日誌存放位置
--ignore_last_failover在預設情況下，如果MHA 檢測到連續發生宕機，且兩次宕機間隔不足8小時的話，則不會進行Failover，之所以這樣限制是為了避免ping-pong效應。
該引數代表忽略上次MHA觸發切換產生的檔案，預設情況下，MHA 發生切換後會在日誌記目錄，也就是上面設定的日誌app1.failover.complete檔案，
下次再次切換的時候如果發現該目錄下存在該檔案將不允許觸發切換，除非在第一次切換後收到刪除該檔案，為了方便，這裡設定為-ignore_last_failover。
隔不足8小時的話，則不會進行Failover，之所以這樣限制是為了避免ping-pong效應。該
引數代表忽略上次MHA觸發切換產生的檔案，預設情況下，MHA 發生切換後會在日誌記
目錄，也就是上面設定的日誌app1.failover.complete檔案，下次再次切換的時候如果發現
該目錄下存在該檔案將不允許觸發切換，除非在第一次切換後收到刪除該檔案，為了方便，
這裡設定為-ignore_last_failover.

8.檢視MHA狀態，可以看到當前的master是Mysql1節點。
[root@mha-manager~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:70465) is running(0:PING_OK), master:192.168.100.150

9.檢視MHA日誌，也以看到當前的master是192.168.100.150，如下所示。
[root@mha-manager ~]# cat /var/log/masterha/app1/manager.log
.......
IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===
.........

10.檢視 Mysql1的VIP地址192.168.100.200是否存在?，這個VIР地址不會因為
manager 節點停止MHA 服務而消失。
[root@mysql1 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.150  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::7264:7578:de4e:190  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9433:6cdb:ee1d:7b3d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)
        RX packets 101492  bytes 130137031 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 43150  bytes 7281255 (6.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)

------------------------------------驗證------------------------------

[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
//啟用監控觀察日誌記錄
[root@MysqI1~]# pkill -9 mysql /檢視master變化
[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.100.150(192.168.100.150:3306)
The latest slave 192.168.100.160(192.168.100.160:3306) has all relay logs for recovery.
Selected 192.168.100.160(192.168.100.160:3306) as a new master.
192.168.100.160(192.168.100.160:3306): OK: Applying all logs succeeded.
192.168.100.160(192.168.100.160:3306): OK: Activated master IP address.
192.168.100.170(192.168.100.170:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.100.170(192.168.100.170:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.100.160(192.168.100.160:3306)
192.168.100.160(192.168.100.160:3306): Resetting slave info succeeded.
Master failover to 192.168.100.160(192.168.100.160:3306) completed successfully.  'master已經成功切換到備用160上面'
[root@mha-manager bin]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:72487) is running(0:PING_OK), master:192.168.100.160
[root@mha-manager bin]# 

[root@mysql2 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.160  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::1bbb:ba05:a579:42dd  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
        RX packets 100963  bytes 130156228 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40046  bytes 6979445 (6.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
虛擬地址也切換到mysql2
故障模擬:
在主庫上:
pkill mysqld
可以看到從庫的狀態，其中之一肯定有切換到主庫的
切換備選主庫的演算法:
1.一般判斷從庫的是從(position/GTID）判斷優劣，資料有差異，最接近於master的slave，成為備選主。
2.資料一致的情況下，按照配置檔案順序，選擇備選主庫。
3.設定有權重(candidate_master=1)，按照權重強制指定備選主。
1）預設情況下如果一個slave落後master 100M的relay logs的話，即使有權重，也會失效。
2）如果check_repl_delay=0的話，即使落後很多日誌，也強制選擇其為備選主。

故障修復步驟:
1.修復db
/etc/init.d/mysqld start
2修復主從
>change master to master_host='192.168.100.150',master_port=3306,master_auto_position=1,master_user='mha',master_passwd='manager',master_log_file='master-bin.000001'
change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=1294;
>start slave;
3.修改配置檔案(再把這個記錄新增進去，因為它檢測掉失效時候會自動消失)
vi /etc/masterha/app1.cnf
[server1]
hostname=192.168.195.129
port=3306
4.啟動manager(在manager那臺機器上)
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --gnore _last failover < /dev/null

dos2unix /usr/local/bin/master_ip_failover解決中英字不相容報錯的問題