1. 程式人生 > >pt-table-checksum檢測不出主從差異處理

pt-table-checksum檢測不出主從差異處理

ola ble 發布 crash recursion log ext open png

幾個月前寫過pt-table-checksum 3.0.4檢測不出主從差異數據,當時的解決方案是使用舊版本,另一個挫方法是自行設置binlog_format=‘STATEMENT‘。現在已經發布到3.0.9版本,原以為官方修復那個問題,結果還是一樣的坑~
最近幾版pt-table-checksum在binlog_format=‘row‘,且主從存在差異數據時,卻檢測不出主從差異。原因就是主上沒有SET @@binlog_format := ‘STATEMENT‘,導致下面兩個核心語句不是以statement格式記錄,從庫不會進行CRC32相關運算,主從永遠一致~

技術分享圖片
# pt-table-checksum 3.0
.4檢測不出差異數據 [root@ZST2 ~]# /usr/local/bin/pt-table-checksum --nocheck-binlog-format --nocheck-replication-filters --recursion-method=hosts --replicate=replcrash.checksums --databases=replcrash --tables=py_user,py_user_innodb --host=192.168.85.132 --port=3306 --user=mydba --password=mysql5721 TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE
05-08T09:41:15 0 0 8 1 0 0.257 replcrash.py_user 05-08T09:41:16 0 0 67740 5 0 1.056 replcrash.py_user_innodb # 兩個核心語句 REPLACE INTO `replcrash`.`checksums` (db, tbl, chunk, chunk_index, lower_boundary, upper_boundary, this_cnt, this_crc) SELECT replcrash
, py_user, 1, NULL, NULL, NULL, COUNT(*) AS cnt, COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS(#, `uid`, convert(`name` using utf8mb4), `add_time`, convert(`server_id` using utf8mb4), CONCAT(ISNULL(`name`), ISNULL(`add_time`), ISNULL(`server_id`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `replcrash`.`py_user` /*checksum table*/ UPDATE `replcrash`.`checksums` SET chunk_time = 0.004092, master_crc = 5abbd632, master_cnt = 8 WHERE db = replcrash AND tbl = py_user AND chunk = 1
View Code

通過對比舊版本(2.2.3)的/usr/local/bin/pt-table-checksum代碼,發現只需稍微調整代碼即可正常使用

技術分享圖片
# pt-table-checksum 3.0.4 只需註釋掉第9335行和9364行,這層邏輯應該應用於任何情況下,因此不需要使用if判斷
[root@ZST2 ~]# vim /usr/local/bin/pt-table-checksum
...
9335       #if ( $o->get(‘check-binlog-format‘) ) {
9336         # https://bugs.launchpad.net/percona-toolkit/+bug/919352
9337         # The tool shouldnt blindly attempt to change binlog_format;
9338         # instead, it should check if its already set to STATEMENT.
9339         # This is becase starting with MySQL 5.1.29, changing the format
9340         # requires a SUPER user.
9341         if ( VersionParser->new($dbh) >= 5.1.5 ) {
9342            $sql = SELECT @@binlog_format;
9343            PTDEBUG && _d($dbh, $sql);
9344            my ($original_binlog_format) = $dbh->selectrow_array($sql);
9345            PTDEBUG && _d(Original binlog_format:, $original_binlog_format);
9346            if ( $original_binlog_format !~ /STATEMENT/i ) {
9347               $sql = q{/*!50108 SET @@binlog_format := ‘STATEMENT‘*/};
9348               eval {
9349                  PTDEBUG && _d($dbh, $sql);
9350                  $dbh->do($sql);
9351               };
9352               if ( $EVAL_ERROR ) {
9353                  die "Failed to $sql: $EVAL_ERROR\n"
9354                     . "This tool requires binlog_format=STATEMENT, "
9355                     . "but the current binlog_format is set to "
9356                     ."$original_binlog_format and an error occurred while "
9357                     . "attempting to change it.  If running MySQL 5.1.29 or newer, "
9358                     . "setting binlog_format requires the SUPER privilege.  "
9359                     . "You will need to manually set binlog_format to ‘STATEMENT‘ "
9360                     . "before running this tool.\n";
9361               }
9362            }
9363         }
9364       #}
...

# 查看pt版本
[root@ZST2 ~]# /usr/local/bin/pt-table-checksum --version
pt-table-checksum 3.0.4
[root@ZST2 ~]# 
# 修改後檢測出主從不一致
[root@ZST2 ~]# /usr/local/bin/pt-table-checksum --nocheck-binlog-format --nocheck-replication-filters --recursion-method=hosts --replicate=replcrash.checksums --databases=replcrash --tables=py_user,py_user_innodb --host=192.168.85.132 --port=3306 --user=mydba --password=mysql5721
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
05-08T09:51:32      0      1        8       1       0   0.030 replcrash.py_user
05-08T09:51:33      0      0    67740       5       0   0.561 replcrash.py_user_innodb
[root@ZST2 ~]# 

# general-log信息
[root@ZST1 ~]# cat /data/mysql/mysql3306/data/ZST1.log |more
...
2018-05-08T01:51:32.315064Z        30 Query     SHOW VARIABLES LIKE version%
2018-05-08T01:51:32.318337Z        30 Query     SHOW ENGINES
2018-05-08T01:51:32.319229Z        30 Query     SHOW VARIABLES LIKE innodb_version
2018-05-08T01:51:32.322518Z        30 Query     SELECT @@binlog_format
2018-05-08T01:51:32.323019Z        30 Query     /*!50108 SET @@binlog_format := ‘STATEMENT‘*/
2018-05-08T01:51:32.323453Z        30 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2018-05-08T01:51:32.324012Z        30 Query     SHOW VARIABLES LIKE wsrep_on
2018-05-08T01:51:32.327830Z        30 Query     SELECT @@SERVER_ID
2018-05-08T01:51:32.328416Z        30 Query     SHOW SLAVE HOSTS
2018-05-08T01:51:32.344178Z        30 Query     SHOW VARIABLES LIKE wsrep_on
2018-05-08T01:51:32.347571Z        30 Query     SELECT @@SERVER_ID
2018-05-08T01:51:32.351692Z        30 Query     SHOW VARIABLES LIKE wsrep_on
2018-05-08T01:51:32.355034Z        30 Query     SELECT @@SERVER_ID
2018-05-08T01:51:32.359543Z        30 Query     SHOW DATABASES LIKE replcrash
2018-05-08T01:51:32.360406Z        30 Query     CREATE DATABASE IF NOT EXISTS `replcrash` /* pt-table-checksum */
2018-05-08T01:51:32.361538Z        30 Query     USE `replcrash`
2018-05-08T01:51:32.362069Z        30 Query     SHOW TABLES FROM `replcrash` LIKE checksums
2018-05-08T01:51:32.364738Z        30 Query     CREATE TABLE IF NOT EXISTS `replcrash`.`checksums` (
     db             CHAR(64)     NOT NULL,
     tbl            CHAR(64)     NOT NULL,
     chunk          INT          NOT NULL,
     chunk_time     FLOAT            NULL,
     chunk_index    VARCHAR(200)     NULL,
     lower_boundary TEXT             NULL,
     upper_boundary TEXT             NULL,
     this_crc       CHAR(40)     NOT NULL,
     this_cnt       INT          NOT NULL,
     master_crc     CHAR(40)         NULL,
     master_cnt     INT              NULL,
     ts             TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     PRIMARY KEY (db, tbl, chunk),
     INDEX ts_db_tbl (ts, db, tbl)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8
2018-05-08T01:51:32.920625Z        30 Query     SHOW GLOBAL STATUS LIKE Threads_running
2018-05-08T01:51:32.924636Z        30 Query     SELECT CONCAT(@@hostname, @@port)
...
View Code

技術分享圖片
技術分享圖片
如果遇到類似問題,建議開啟general_log,查看處理過程,再核驗到底什麽原因導致檢測不出主從差異數據(?ω?)

pt-table-checksum檢測不出主從差異處理