【PHP】你使用過redis做非同步佇列麼,是怎麼用的?有什麼缺點?
一、需求源:
僅作為如下需求而測試,不參與應用業務架構討論。
1.MySQL 3臺server 1主1從1備,keepalived保持vip在主庫節點,在主節點和從節點均啟動keepalived,3臺機器均部署orchestrator。定義 mysqlA 為主,mysqlB為從,mysqlC為備。
2.MySQL節點故障:主節點失效,orchestrator自動切換新的主節點,保持高可用。
3.keepalived 監測本機是否為主節點,如果為主則獲取vip。
4.失效節點恢復後作為從節點加入叢集。(無要求自動加入,但為滿足相對自動化,已寫指令碼通過keepalived檢查並加入。)
二、部署過程:
node OS: centos7.8
mysql version: 5.7.31
keepalived version: 2.1.5
node A : mysqlA 192.168.127.142
node B: mysqlB 192.168.127.143
node C: mysqlC 192.168.127.124
VIP : 192.168.127.150
disable firewalld
setenforce 0
3臺機器配置新增hosts
echo '192.168.127.142 mysqlA' >> /etc/hosts echo '192.168.127.143 mysqlB' >> /etc/hostsecho '192.168.127.144 mysqlC' >> /etc/hosts
3臺機器互免密登入。
ssh-keygen #一直回車 ssh-copy-id mysqlA ssh-copy-id mysqlB ssh-copy-id mysqlC
因配置了Raft,有多個Orchestrator配置高可用,orchestrator-client會自動選擇leader。
3臺機器執行 vi /etc/profile,最後新增
export ORCHESTRATOR_API="mysqlA:3000/api mysqlB:3000/api mysqlC:3000/api"
軟體列表,各node均需要
[root@mysqlA ~]# ll -rw-r--r--. 1 root root 1081559 Nov 25 06:14 keepalived-2.1.5.tar.gz -rw-r--r--. 1 root root 9116 Nov 25 06:14 mysql57-community-release-el7-8.noarch.rpm -rw-r--r--. 1 root root 10467081 Nov 25 06:14 orchestrator-3.2.3-1.x86_64.rpm -rw-r--r--. 1 root root 10043463 Nov 25 06:14 orchestrator-cli-3.2.3-1.x86_64.rpm -rw-r--r--. 1 root root 15153 Nov 25 06:14 orchestrator-client-3.2.3-1.x86_64.rpm
2.1 keepalived
解壓並安裝
[root@mysqlA ~]# tar -zxf keepalived-2.1.5.tar.gz [root@mysqlA ~]# cd keepalived-2.1.5 [root@mysqlA keepalived-2.1.5]# ./configure && make && make install
安裝keepalive遇到錯誤1:
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
解決錯誤1:
[root@mysqlA keepalived-2.1.5]# yum -y install openssl-devel
編制keepalive.conf
[root@mysqlA ~]# mkdir /etc/keepalived [root@mysqlA ~]# cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/ [root@mysqlA ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { # notification_email { # [email protected] # [email protected] # [email protected] # } # notification_email_from [email protected] # smtp_server 192.168.200.1 # smtp_connect_timeout 30 # router_id LVS_DEVEL # vrrp_skip_check_adv_addr # vrrp_strict # vrrp_garp_interval 0 # vrrp_gna_interval 0 #執行指令碼的使用者,預設是keepalived_script,無則建立,此處用root執行 script_user root #開啟安全指令碼 enable_script_security } #開啟check_run,指令碼是檢測mysql叢集中,誰是master,誰就佔用vip。 #interval 指令碼執行間隔時間 vrrp_script check_run { script "/usr/local/orchestrator/mysqlA_check_master_vip.sh" interval 2 } #state 為BACKUP,2臺mysql一致; #interface:名稱為網絡卡名稱 #virtual_route_id:2臺mysql一致; #priority:2臺mysql一致 #track_script:對應上面vrrp_script的值 #notify_backup:當前keepalived狀態為backup時,即不佔用VIP時,通過指令碼檢查mysql狀態,並自動以 從 身份加入到叢集裡 vrrp_instance mysql { state BACKUP interface ens33 virtual_router_id 51 priority 80 nopreempt advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.127.150/24 } track_script { check_run } notify_backup /usr/local/orchestrator/join_MysqlCluster.sh }View Code keepalived.conf
mysqlB 配置檔案相同,注意修改vrrp_script check_run 中的指令碼名字,改為:mysqlB_check_master_vip.sh 。
將keepalived新增到系統服務,AB機均執行。
[root@mysqlA keepalived-2.1.5]# pwd /root/keepalived-2.1.5 [root@mysqlA keepalived-2.1.5]# cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/ [root@mysqlA init.d]# pwd /root/keepalived-2.1.5/keepalived/etc/init.d [root@mysqlA init.d]# cp keepalived /etc/init.d/
至此,keepalived暫告一段落。
2.2mysql
使用yum安裝。3臺機器均執行。僅做主從複製配置,詳細優化引數未設定,要閱讀官方天書補充。
rpm -Uvh mysql57-community-release-el7-8.noarch.rpm
#對應的yum安裝mysql版本在http://repo.mysql.com查詢,el為centos及rhel
yum clean all && yum makecache yum install gcc gcc-c++ openssl-devel mysql mysql-server mysql-devel -y [root@mysqlA ~]# mkdir /data/sqlite -p [root@mysqlA ~]# chown -R mysql:mysql /data/sqlite
mysql安裝登入密碼,查詢:
[root@mysqlA ~]# systemctl start mysqld grep 'temporary password' /var/log/mysqld.log
登入mysql後,修改密碼:
mysql> SET PASSWORD = PASSWORD('P@ssw0rd');
建立使用者,並授權
mysql> CREATE USER 'orchestrator'@'%' IDENTIFIED BY 'P@ssw0rd'; mysql> grant SUPER,REPLICATION CLIENT,replication slave on *.* to 'orchestrator'@'192.168.127.%' identified by 'P@ssw0rd'; mysql> flush privileges;
退出mysql,編制mysql配置檔案
[root@mysqlA ~]# cat /etc/my.cnf
[mysqld] symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid #server-id 每臺機器不一致,現在以IP後位設定 server-id = 142 log-bin=master-bin binlog_format=row binlog-ignore-db = mysql,information_schema,performance_schema #report_host 為主機名 report_host = mysqlA #開啟GTID gtid-mode=ON enforce-gtid-consistency=ON #mysqlC的log-slave-update不配置。 log-slave-updates=1 skip_slave_start=1View Code my.cnf
#在A及B機必須開啟的引數
log_slave_updates=1
所有主機重啟mysql服務
[root@mysqlA ~]# systemctl start mysqld
所有主機各自新增mysql_config_editor,登入mysql時不用輸入密碼
mysql_config_editor set --login-path=local --user=root --port=3306 --password
#檢視本機mysql_config_editor,注意local對應之前設定的--login-path
#my_print_defaults -s local
#重置所有
#mysql_config_editor reset
#測試
#mysql --login-path=local
設定主從複製,初定架構為:A--->B--->C
mysqlB:
[root@mysqlB ~]# mysql --login-path=local mysql> change master to master_host='mysqlA',master_port=3306,master_user='orchestrator',master_password='P@ssw0rd',master_auto_position=1; mysql> start slave; mysql> show slave status\G
關注如圖兩項
mysqlC:
[root@mysqlC ~]# mysql --login-path=local mysql> change master to master_host='mysqlB',master_port=3306,master_user='orchestrator',master_password='P@ssw0rd',master_auto_position=1; mysql> start slave; mysql> show slave status\G
檢視下面兩項值均為Yes,即表示設定從伺服器成功。
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
沒正式使用之前,驗證可同步後,先停掉所有mysql的複製關係。等orchestrator和keepalived正式配置完整後,在啟用複製。生產環境慎用 reset master;。
stop slave;
reset slave all;
reset master;
至此,MySQL暫告一段落。
2.3orcheatrator
3臺機器均安裝。
下載https://github.com/openark/orchestrator/releases
使用yum安裝rpm包
[root@mysqlA ~]# yum install epel-release [root@mysqlA ~]# yum install orchestrator-3.2.3-1.x86_64.rpm orchestrator-client-3.2.3-1.x86_64.rpm orchestrator-cli-3.2.3-1.x86_64.rpm -y
配置
將/usr/local/orchestrator/orchestrator-sample.conf.json 複製到/etc目錄下。
[root@mysqlA ~]# cp /usr/local/orchestrator/orchestrator-sample.conf.json /etc/orchestrator.conf.json
編輯 /etc/orchestrator.conf.json
{ "Debug": true, "EnableSyslog": false, "ListenAddress": ":3000", "MySQLTopologyUser": "orchestrator", *** ###與前面新增的mysql使用者對應*** "MySQLTopologyPassword": "P@ssw0rd", *** ###密碼對應*** "MySQLTopologyCredentialsConfigFile": "", "MySQLTopologySSLPrivateKeyFile": "", "MySQLTopologySSLCertFile": "", "MySQLTopologySSLCAFile": "", "MySQLTopologySSLSkipVerify": true, "MySQLTopologyUseMutualTLS": false, "BackendDB": "sqlite", *** ###使用sqlite,沒有使用mysql庫,可改用*** "SQLite3DataFile": "/data/sqlite/orchestrator.db", *** ###sqlite檔案絕對路徑,有資料夾即可。*** "MySQLConnectTimeoutSeconds": 1, "DefaultInstancePort": 3306, "DiscoverByShowSlaveHosts": true, "InstancePollSeconds": 5, "DiscoveryIgnoreReplicaHostnameFilters": [ "a_host_i_want_to_ignore[.]example[.]com", ".*[.]ignore_all_hosts_from_this_domain[.]example[.]com", "a_host_with_extra_port_i_want_to_ignore[.]example[.]com:3307" ], "UnseenInstanceForgetHours": 240, "SnapshotTopologiesIntervalHours": 0, "InstanceBulkOperationsWaitTimeoutSeconds": 10, "HostnameResolveMethod": "default", "MySQLHostnameResolveMethod": "@@hostname", "SkipBinlogServerUnresolveCheck": true, "ExpiryHostnameResolvesMinutes": 60, "RejectHostnameResolvePattern": "", "ReasonableReplicationLagSeconds": 10, "ProblemIgnoreHostnameFilters": [], "VerifyReplicationFilters": false, "ReasonableMaintenanceReplicationLagSeconds": 20, "CandidateInstanceExpireMinutes": 60, "AuditLogFile": "", "AuditToSyslog": false, "RemoveTextFromHostnameDisplay": ".mydomain.com:3306", "ReadOnly": false, "AuthenticationMethod": "basic", *** ###開啟web時使用basic方式認證,避免免密開啟*** "HTTPAuthUser": "admin", *** ###認證使用者*** "HTTPAuthPassword": "P@ssw0rd", *** ###認證密碼*** "AuthUserHeader": "", "PowerAuthUsers": [ "*" ], "ClusterNameToAlias": { "127.0.0.1": "test suite" }, "ReplicationLagQuery": "", "DetectClusterAliasQuery": "SELECT SUBSTRING_INDEX(@@hostname, '.', 1)", "DetectClusterDomainQuery": "", "DetectInstanceAliasQuery": "", "DetectPromotionRuleQuery": "", "DataCenterPattern": "[.]([^.]+)[.][^.]+[.]mydomain[.]com", "PhysicalEnvironmentPattern": "[.]([^.]+[.][^.]+)[.]mydomain[.]com", "PromotionIgnoreHostnameFilters": [], "DetectSemiSyncEnforcedQuery": "", "ServeAgentsHttp": false, "AgentsServerPort": ":3001", "AgentsUseSSL": false, "AgentsUseMutualTLS": false, "AgentSSLSkipVerify": false, "AgentSSLPrivateKeyFile": "", "AgentSSLCertFile": "", "AgentSSLCAFile": "", "AgentSSLValidOUs": [], "UseSSL": false, "UseMutualTLS": false, "SSLSkipVerify": false, "SSLPrivateKeyFile": "", "SSLCertFile": "", "SSLCAFile": "", "SSLValidOUs": [], "URLPrefix": "", "StatusEndpoint": "/api/status", "StatusSimpleHealth": true, "StatusOUVerify": false, "AgentPollMinutes": 60, "UnseenAgentForgetHours": 6, "StaleSeedFailMinutes": 60, "SeedAcceptableBytesDiff": 8192, "PseudoGTIDPattern": "", "PseudoGTIDPatternIsFixedSubstring": false, "PseudoGTIDMonotonicHint": "asc:", "DetectPseudoGTIDQuery": "", "BinlogEventsChunkSize": 10000, "SkipBinlogEventsContaining": [], "ReduceReplicationAnalysisCount": true, "FailureDetectionPeriodBlockMinutes": 5, "RecoveryPeriodBlockSeconds": 30, "RecoveryIgnoreHostnameFilters": [], "RecoverMasterClusterFilters": [ "*" ], "RecoverIntermediateMasterClusterFilters": [ "*" ], "OnFailureDetectionProcesses": [ "echo '`date +'%Y-%m-%d %T'` Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log" ], "PreGracefulTakeoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Planned takeover about to take place on {failureCluster}. Master will switch to read_only' >> /tmp/recovery.log" ], "PreFailoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log" ], "PostFailoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` (for all types) Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}; failureClusterAlias:{failureClusterAlias}' >> /tmp/recovery.log", "/usr/local/orchestrator/orch_hook.sh {failureType} {failureClusterAlias} {failedHost} {successorHost} >> /tmp/orch.log" ***###新增指令碼,當機器master出現硬體故障,或mysql軟體故障,即刻漂移VIP,並切換主*** ], "PostUnsuccessfulFailoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Unsuccessful Failover ' >> /tmp/recovery.log"], "PostMasterFailoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log" ], "PostIntermediateMasterFailoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log" ], "PostGracefulTakeoverProcesses": [ "echo '`date +'%Y-%m-%d %T'` Planned takeover complete' >> /tmp/recovery.log" ], "CoMasterRecoveryMustPromoteOtherCoMaster": true, "DetachLostSlavesAfterMasterFailover": true, "ApplyMySQLPromotionAfterMasterFailover": true, "PreventCrossDataCenterMasterFailover": false, "PreventCro***egionMasterFailover": false, "MasterFailoverDetachReplicaMasterHost": false, "MasterFailoverLostInstancesDowntimeMinutes": 0, "PostponeReplicaRecoveryOnLagMinutes": 0, "OSCIgnoreHostnameFilters": [], "GraphiteAddr": "", "GraphitePath": "", "GraphiteConvertHostnameDotsToUnderscores": true, "ConsulAddress": "", "ConsulAclToken": "", "PseudoGTIDPattern": "drop view if exists .*?`_pseudo_gtid_hint__", "PseudoGTIDMonotonicHint": "asc:", "DetectPseudoGTIDQuery": "select count(*) as pseudo_gtid_exists from meta.pseudo_gtid_status where anchor = 1 and time_generated > now() - interval 2 day", "RaftEnabled":true, ***###啟用高可用*** "RaftDataDir":"/usr/local/orchestrator", *** ###orchestrator目錄*** "RaftBind":"192.168.127.142", ***###本機的IP*** "DefaultRaftPort":10008, "RaftNodes":[ ***###3臺機的IP*** "192.168.127.142", "192.168.127.143", "192.168.127.144" ] }View Code orchestrator.conf
複製到其他機器,注意修改檔案的本機IP
[root@mysqlA ~]# scp /etc/orchestrator.conf.json root@mysqlB:/etc
指令碼:
路徑:/usr/local/orchestrator
mysqlA:orch_hook.sh、mysqlA_check_master_vip.sh、join_MysqlCluster.sh
mysqlB:orch_hook.sh、mysqlB_check_master_vip.sh、join_MysqlCluster.sh
orch_hook.sh用途:用於檢查master狀態,如master硬體故障或mysql軟體故障,即切換VIP至從(orchestrator已經自動切換從為新主);
mysqlA_check_master_vip.sh用途:檢查叢集MysqlCluster中,誰是master,誰即佔用VIP;雖然和orch_hook.sh用途相似,但是上一個指令碼沒有檢查MysqlCluster叢集master的功能,當主恢復故障後,是獨立master出現,但不在叢集裡。
join_MysqlCluster.sh:檢查本機是否在叢集裡。若不在,檢查本機mysql狀態,狀態OK自動以 從 庫身份加入MysqlCluster叢集,並自動啟用複製。
orch_hook.sh,在orchestrator配置檔案上呼叫:
#!/bin/bash isitdead=$1 cluster=$2 oldmaster=$3 newmaster=$4 logfile="/var/log/orch_hook.log" # list of clusternames #clusternames=(rep blea lajos) if [[ $isitdead == "DeadMaster" ]]; then ssh $oldmaster "systemctl restart keepalived" elif [[ $isitdead == "DeadIntermediateMasterWithSingleSlaveFailingToConnect" ]]; then ssh $oldmaster "systemctl restart keepalived" elif [[ $isitdead == "DeadIntermediateMaster" ]]; then ssh $oldmaster "systemctl restart keepalived" fiView Code orch_hook.sh
mysqlA_check_master_vip.sh,在keepalived配置檔案上呼叫,先本地執行驗證無誤:
#!/bin/bash master_node=`orchestrator-client -b admin:P@ssw0rd -c which-cluster-master -i MysqlCluster |awk -F ":" '{print $1}'` #echo "The cluster-master is $master_node" #mysqlB_check_master_vip.sh 的不同引數僅為下列兩項,互換即可 local_node=mysqlA remote_node=mysqlB #echo $local_node vipadd=192.168.127.150 if [ $master_node == $local_node ];then local_ip=`ip a show dev ens33|grep -w inet|awk '{print $2}'|awk -F '/' '{print $1}'` if [[ "$local_ip" =~ "$vipadd" ]];then # echo "vip is in master" exit 0 else echo "master vip is down, restarting now" ssh $remote_node "systemctl restart keepalived" # ssh $remote_node "systemctl start keepalived" fi else local_ip=`ip a show dev ens33|grep -w inet|awk '{print $2}'|awk -F '/' '{print $1}'` if [[ "$local_ip" =~ "$vipadd" ]];then # echo "vip is in local,release..." systemctl restart keepalived # systemctl start keepalived else # echo "vip don't in local,exit..." exit 0 fi fiView Code check_master_vip.sh
join_MysqlCluster.sh:在keepalived配置檔案上呼叫
#!/bin/bash mein=`orchestrator-client -b admin:P@ssw0rd -c which-cluster-instances -i MysqlCluster |awk -F ":" '{print $1}'` local_node=`hostname` if [[ "$mein" =~ "$local_node" ]];then echo "the node in cluster" >> /data/nodestatus.txt exit 0 eles echo "will be check status and " >> /data/nodestatus.txt fi CHECK_TIME=3 #mysql is working MYSQL_OK is 1 , mysql down MYSQL_OK is 0 MYSQL_OK=1 #function to check mysql status function check_mysql_helth (){ mysql --login-path=local -e "show status;" >/dev/null 2>&1 if [ $? = 0 ] ;then MYSQL_OK=1 else MYSQL_OK=0 fi return $MYSQL_OK } while [ $CHECK_TIME -ne 0 ] do let "CHECK_TIME -= 1" check_mysql_helth if [ $MYSQL_OK = 1 ] ; then CHECK_TIME=0 #mysql status is OK,then try join MysqlCluster #get MysqlCluster master_node master_node=`orchestrator-client -b admin:P@ssw0rd -c which-cluster-master -i MysqlCluster |awk -F":" '{print $1}'` #copy to master_node,set read_only,and start slave; mysql --login-path=local -e "change master to master_host='$master_node',master_port=3306,master_user='orchestrator',master_password='P@ssw0rd',master_auto_position=1;set global read_only=1;start slave;" #check slave status, shuoult be check Slave_IO_Running & Slave_SQL_Running Slave_IO_Run_status=`mysql --login-path=local -e "show slave status\G" |grep Slave_IO_Running |awk -F ":" '{print $2}'` if [ $Slave_IO_Run_status == Yes ]; then echo "Slave_IO_Runing is Yes" else echo "Slave_IO_Runing is Error" exit 1 fi Slave_SQL_Run_status=`mysql --login-path=local -e "show slave status\G" |grep Slave_SQL_Running |awk -F ":" '{print $2}'` if [ $Slave_SQL_Run_status == Yes ]; then echo "Slave_SQL_Running is Yes" else echo "Slave_SQL_Running is Error" exit 1 fi exit 0 fi if [ $MYSQL_OK -eq 0 ] && [ $CHECK_TIME -eq 0 ] then #pkill keepalived exit 1 fi sleep 10 doneView Code join_MysqlCluster.sh
3臺機器啟動 orchestrator服務
[root@mysqlA~]# systemctl start orchestrator [root@mysqlB~]# systemctl start orchestrator [root@mysqC~]# systemctl start orchestrator
登入web介面,由於未啟用eepalived,所以用其中1臺IP登入,並做發現:
Discover a new instance
輸入mysql地址或主機名
發現完畢
重新整理,回到叢集,重新命名叢集一個別名
新的叢集別名,要和指令碼上的變數一致
回到叢集
各節點狀態及自動生成的複製拓撲。
直接在拓撲圖上修改從庫的複製關係:
拖動修改主庫,官方翻譯為優雅的進行主從切換,切換後,舊主為從,需要重新啟動slave,手動,暫無寫指令碼檢測並啟動。
臨時寫了一個 優雅切換主從後的 新從 啟動複製指令碼
在 新從 伺服器上 ./orch_hook2.sh
#!/bin/bash mysql --login-path=local -e "stop slave;" new_master_node=`orchestrator-client -b admin:P@ssw0rd -c which-cluster-master -i MysqlCluster |awk -F":" '{print $1}'` mysql --login-path=local -e "change master to master_host='$new_master_node',master_port=3306,master_user='orchestrator',master_password='P@ssw0rd',master_auto_position=1;set global read_only=1;start slave;"View Code orch_hook2.sh
結語
##測試過程排錯讓人迷失方向---必須讀懂官方文件---感謝搜到的國內幾篇好文##
參考資料:
https://github.com/openark/orchestrator
https://www.cnblogs.com/zhoujinyi/p/10394389.html
https://zhuanlan.zhihu.com/p/139294374
1、我在構建MySQL的時候,確實是門外漢,太多配置引數、優化引數都不知道是什麼,有什麼用。像GTID,不啟用的話,主從切換複製每次要show master status,記錄File、和Position。測試過程還遇到 資料庫複製正常,但不同步、reset master後的恢復、匯出匯入恢復...一言難盡。
2、keepalived 回首一看,也有5-6年沒碰過這玩意了,配置檔案很多引數都不知道怎麼用了。呼叫指令碼那裡栽了幾個跟斗。
3、orchestrator,今年首次認識這個東西,然後一直思考這個東西有什麼用,能自動切換,能恢復拓撲,能。。。還以為主庫掛了之後能自動恢復,結果只是我以為。
4、指令碼也是寫的太爛了,本來指令碼能力就差...