MySQL-binlog解析工具
阿新 • • 發佈：2021-09-04
解析MySQL系列資料庫binlog日誌工具
mysqlbinlog
MySQL官方原生提供的解析(binlog)二進位制日誌的工具
用法
mysqlbinlog --no-defaults --help Usage: mysqlbinlog [options] log-files -?, --help Display this help and exit. --base64-output=name Determine when the output statements should be base64-encoded BINLOG statements: 'never' disables it and works only for binlogs without row-based events; 'decode-rows' decodes row events into commented pseudo-SQL statements if the --verbose option is also given; 'auto' prints base64 only when necessary (i.e., for row-based events and format description events). If no --base64-output[=name] option is given at all, the default is 'auto'. --bind-address=name IP address to bind to. --character-sets-dir=name Directory for character set files. -d, --database=name List entries for just this database (local log only). --rewrite-db=name Rewrite the row event to point so that it can be applied to a new database -#, --debug[=#] This is a non-debug version. Catch this and exit. --debug-check This is a non-debug version. Catch this and exit. --debug-info This is a non-debug version. Catch this and exit. --default-auth=name Default authentication client-side plugin to use. -D, --disable-log-bin Disable binary log. This is useful, if you enabled --to-last-log and are sending the output to the same MySQL server. This way you could avoid an endless loop. You would also like to use it when restoring after a crash to avoid duplication of the statements you already have. NOTE: you will need a SUPER privilege to use this option. -F, --force-if-open Force if binlog was not closed properly. (Defaults to on; use --skip-force-if-open to disable.) -f, --force-read Force reading unknown binlog events. -H, --hexdump Augment output with hexadecimal and ASCII event dump. -h, --host=name Get the binlog from server. -i, --idempotent Notify the server to use idempotent mode before applying Row Events -l, --local-load=name Prepare local temporary files for LOAD DATA INFILE in the specified directory. -o, --offset=# Skip the first N entries. -p, --password[=name] Password to connect to remote server. --plugin-dir=name Directory for client-side plugins. -P, --port=# Port number to use for connection or 0 for default to, in order of preference, my.cnf, $MYSQL_TCP_PORT, /etc/services, built-in default (3306). --protocol=name The protocol to use for connection (tcp, socket, pipe, memory). -R, --read-from-remote-server Read binary logs from a MySQL server. This is an alias for read-from-remote-master=BINLOG-DUMP-NON-GTIDS. --read-from-remote-master=name Read binary logs from a MySQL server through the COM_BINLOG_DUMP or COM_BINLOG_DUMP_GTID commands by setting the option to either BINLOG-DUMP-NON-GTIDS or BINLOG-DUMP-GTIDS, respectively. If --read-from-remote-master=BINLOG-DUMP-GTIDS is combined with --exclude-gtids, transactions can be filtered out on the master avoiding unnecessary network traffic. --raw Requires -R. Output raw binlog data instead of SQL statements, output is to log files. -r, --result-file=name Direct output to a given file. With --raw this is a prefix for the file names. --secure-auth Refuse client connecting to server if it uses old (pre-4.1.1) protocol. Deprecated. Always TRUE --server-id=# Extract only binlog entries created by the server having the given id. --server-id-bits=# Set number of significant bits in server-id --set-charset=name Add 'SET NAMES character_set' to the output. -s, --short-form Just show regular queries: no extra info and no row-based events. This is for testing only, and should not be used in production systems. If you want to suppress base64-output, consider using --base64-output=never instead. -S, --socket=name The socket file to use for connection. --ssl-mode=name SSL connection mode. --ssl Deprecated. Use --ssl-mode instead. (Defaults to on; use --skip-ssl to disable.) --ssl-verify-server-cert Deprecated. Use --ssl-mode=VERIFY_IDENTITY instead. --ssl-ca=name CA file in PEM format. --ssl-capath=name CA directory. --ssl-cert=name X509 cert in PEM format. --ssl-cipher=name SSL cipher to use. --ssl-key=name X509 key in PEM format. --ssl-crl=name Certificate revocation list. --ssl-crlpath=name Certificate revocation list path. --tls-version=name TLS version to use, permitted values are: TLSv1, TLSv1.1, TLSv1.2 --server-public-key-path=name File path to the server public RSA key in PEM format. --get-server-public-key Get server public key --start-datetime=name Start reading the binlog at first event having a datetime equal or posterior to the argument; the argument must be a date and time in the local time zone, in any format accepted by the MySQL server for DATETIME and TIMESTAMP types, for example: 2004-12-25 11:25:56 (you should probably use quotes for your shell to set it properly). -j, --start-position=# Start reading the binlog at position N. Applies to the first binlog passed on the command line. --stop-datetime=name Stop reading the binlog at first event having a datetime equal or posterior to the argument; the argument must be a date and time in the local time zone, in any format accepted by the MySQL server for DATETIME and TIMESTAMP types, for example: 2004-12-25 11:25:56 (you should probably use quotes for your shell to set it properly). --stop-never Wait for more data from the server instead of stopping at the end of the last log. Implicitly sets --to-last-log but instead of stopping at the end of the last log it continues to wait till the server disconnects. --stop-never-slave-server-id=# The slave server_id used for --read-from-remote-server --stop-never. This option cannot be used together with connection-server-id. --connection-server-id=# The slave server_id used for --read-from-remote-server. This option cannot be used together with stop-never-slave-server-id. --stop-position=# Stop reading the binlog at position N. Applies to the last binlog passed on the command line. -t, --to-last-log Requires -R. Will not stop at the end of the requested binlog but rather continue printing until the end of the last binlog of the MySQL server. If you send the output to the same MySQL server, that may lead to an endless loop. -u, --user=name Connect to the remote server as username. -v, --verbose Reconstruct pseudo-SQL statements out of row events. -v -v adds comments on column data types. -V, --version Print version and exit. --open-files-limit=# Used to reserve file descriptors for use by this program. -c, --verify-binlog-checksum Verify checksum binlog events. --binlog-row-event-max-size=# The maximum size of a row-based binary log event in bytes. Rows will be grouped into events smaller than this size if possible. This value must be a multiple of 256. --skip-gtids Do not preserve Global Transaction Identifiers; instead make the server execute the transactions as if they were new. --include-gtids=name Print events whose Global Transaction Identifiers were provided. --exclude-gtids=name Print all events but those whose Global Transaction Identifiers were provided. Variables (--variable-name=value) and boolean options {FALSE|TRUE} Value (after reading options)
- --no-defaults: 預設charset問題
示例
# 普通許可權的使用者只讀，不能寫資料 set global read_only=1; # 檢視操作記錄資訊 mysql> show binlog events in 'mysql-bin.000002'; # 恢復指定位置的操作 mysqlbinlog --start-position=120 --stop-position=520 --database=demo /var/lib/mysql/mysql-bin.000003 | /usr/bin/mysql -u root -p -v demo # 基於時間點恢復 /usr/bin/mysqlbinlog --start-datetime="2021-06-27 20:57:55" --stop-datetime="2021-06-27 20:58:18" --database=demo /var/lib/mysql/mysql-bin.000009 | /usr/bin/mysql -uroot -p -v demo
MyFlash
美團點評的開源MySQL閃回工具。
安裝
推薦下載原始碼之後，進行動態編譯連結安裝
動態編譯連結
cc -w `pkg-config --cflags --libs glib-2.0` source/binlogParseGlib.c -o binary/flashback
靜態編譯連結
gcc -w -g `pkg-config --cflags glib-2.0` source/binlogParseGlib.c -o binary/flashback /usr/lib64/libglib-2.0.a -lrt
注意：
- 確認目標裝置上 glib庫的版本和位置(ldd --version)
使用
語法
cd binary
./flashback --help
Usage:
flashback [OPTION...]
Help Options:
-?, --help Show help options
Application Options:
--databaseNames databaseName to apply. if multiple, seperate by comma(,)
--tableNames tableName to apply. if multiple, seperate by comma(,)
--start-position start position
--stop-position stop position
--start-datetime start time (format %Y-%m-%d %H:%M:%S)
--stop-datetime stop time (format %Y-%m-%d %H:%M:%S)
--sqlTypes sql type to filter . support INSERT, UPDATE ,DELETE. if multiple, seperate by comma(,)
--maxSplitSize max file size after split, the uint is M
--binlogFileNames binlog files to process. if multiple, seperate by comma(,)
--outBinlogFileNameBase output binlog file name base
--logLevel log level, available option is debug,warning,error
--include-gtids gtids to process
--exclude-gtids gtids to skip
引數說明
-
databaseNames
指定需要回滾的資料庫名。多個數據庫可以用“,”隔開。如果不指定該引數，相當於指定了所有資料庫。
-
tableNames
指定需要回滾的表名。多個表可以用“,”隔開。如果不指定該引數，相當於指定了所有表。
-
start-position
指定回滾開始的位置。如不指定，從檔案的開始處回滾。請指定正確的有效的位置，否則無法回滾
-
stop-position
指定回滾結束的位置。如不指定，回滾到檔案結尾。請指定正確的有效的位置，否則無法回滾
-
start-datetime
指定回滾的開始時間。注意格式必須是 %Y-%m-%d %H:%M:%S。 如不指定，則不限定時間
-
stop-datetime
指定回滾的結束時間。注意格式必須是 %Y-%m-%d %H:%M:%S。 如不指定，則不限定時間
-
sqlTypes
指定需要回滾的sql型別。目前支援的過濾型別是INSERT, UPDATE ,DELETE。多個型別可以用“,”隔開。
-
maxSplitSize
一旦指定該引數，對檔案進行固定尺寸的分割（單位為M），過濾條件有效，但不進行回滾操作。該引數主要用來將大的binlog檔案切割，防止單次應用的binlog尺寸過大，對線上造成壓力
-
binlogFileNames
指定需要回滾的binlog檔案，目前只支援單個檔案，後續會增加多個檔案支援
-
outBinlogFileNameBase
指定輸出的binlog檔案字首，如不指定，則預設為binlog_output_base.flashback
-
logLevel
僅供開發者使用，預設級別為error級別。在生產環境中不要修改這個級別，否則輸出過多
-
include-gtids
指定需要回滾的gtid,支援gtid的單個和範圍兩種形式。
-
exclude-gtids
指定不需要回滾的gtid，用法同include-gtids
示例
回滾整個檔案
# 閃回結果存放到binlog_output_base.flashback中
./flashback --binlogFileNames=haha.000041
# 應用閃回的日誌
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p
回滾該檔案中的所有insert語句
./flashback --sqlTypes='INSERT' --binlogFileNames=haha.000041
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p
回滾大檔案
#回滾
./flashback --binlogFileNames=haha.000042
#切割大檔案
./flashback --maxSplitSize=1 --binlogFileNames=binlog_output_base.flashback
#應用
mysqlbinlog binlog_output_base.flashback.000001 | mysql -h<host> -u<user> -p
...
mysqlbinlog binlog_output_base.flashback.<N> | mysql -h<host> -u<user> -p
綜合測試用例
測試表結構
CREATE TABLE `testFlashback2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`nameShort` varchar(20) DEFAULT NULL,
`nameLong` varchar(260) DEFAULT NULL,
`amount` decimal(19,9) DEFAULT NULL,
`amountFloat` float DEFAULT NULL,
`amountDouble` double DEFAULT NULL,
`createDatetime6` datetime(6) DEFAULT NULL,
`createDatetime` datetime DEFAULT NULL,
`createTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`nameText` text,
`nameBlob` blob,
`nameMedium` mediumtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
插入&回滾
插入資料
flush logs
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
回滾資料
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000048
# 在當前執行目錄下產生binlog_output_base.flashback檔案
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
刪除&回滾
刪除資料
delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
delete from testFlashback2;
回滾資料
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000050
# 在當前執行目錄下產生binlog_output_base.flashback檔案
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
更新&回滾
更新資料
delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.111,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
mysql> checksum table testFlashback2;
+---------------------+-----------+
| Table | Checksum |
+---------------------+-----------+
| test.testFlashback2 | 717087411 |
+---------------------+-----------+
update testFlashback2 set amount=10.222;
mysql> checksum table testFlashback2;
+---------------------+------------+
| Table | Checksum |
+---------------------+------------+
| test.testFlashback2 | 3797190846 |
+---------------------+------------+
回滾資料
/binary/flashback --binlogFileNames=/var/lib/mysql/haha.000052
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
binlog2sql
binlog2sql是一個開源的Python開發的MySQL Binlog解析工具。根據選項不同，可以得到原始SQL、回滾SQL、去除主鍵的INSERT SQL等。
要求
MySQL server必須設定以下引數
[mysqld]
server_id = 1
log_bin = /var/log/mysql/mysql-bin.log
max_binlog_size = 1G
binlog_format = row
binlog_row_image = full
- 引數 binlog_row_image 必須為FULL，暫不支援MINIMAL
使用者所需最小許可權
select, super/replication client, replication slave
-- 建議授權
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO USER_NAME;
說明
- select：需要讀取server端information_schema.COLUMNS表，獲取表結構的元資訊，拼接成視覺化的sql語句
- super/replication client：兩個許可權都可以，需要執行'SHOW MASTER STATUS', 獲取server端的binlog列表
- replication slave：通過BINLOG_DUMP協議獲取binlog內容的許可權
mysql server必須開啟，不支援離線解析
通過 BINLOG_DUMP 協議來獲取 binlog 內容，需要讀取server端 information_schema.COLUMNS 表，來獲取表結構的元資訊，才能拼接成 SQL 語句。因此，必須開啟mysql server。
安裝配置
依賴
- Python 2.7, 3.4+
- MySQL 5.6, 5.7
安裝
git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql
pip install -r requirements.txt
使用
用法
解析出標準SQL
python binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -t test3 test4 --start-file='mysql-bin.000002'
解析回滾SQl
python binlog2sql.py --flashback -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttest3 --start-file='mysql-bin.000002' --start-position=763 --stop-position=1147
選項說明
解析模式
-
--stop-never ：持續解析binlog。可選。預設False，同步至執行命令時最新的binlog位置。
-
-K, --no-primary-key ：對INSERT語句去除主鍵。可選。預設False
-
-B, --flashback ：生成回滾SQL，可解析大檔案，不受記憶體限制。可選。預設False。與stop-never或no-primary-key不能同時新增。
-
--back-interval ：在-B模式下，每列印一千行回滾SQL，加一句SLEEP多少秒，如不想加SLEEP，請設為0。可選。預設1.0。
解析範圍
- --start-file： 起始解析檔案，只需檔名，無需全路徑 。必須。
- --start-position/--start-pos： 起始解析位置。可選。預設為start-file的起始位置。
- --stop-file/--end-file ：終止解析檔案。可選。預設為start-file同一個檔案。若解析模式為stop-never，此選項失效。
- --stop-position/--end-pos ：終止解析位置。可選。預設為stop-file的最末位置；若解析模式為stop-never，此選項失效。
- --start-datetime ：起始解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。預設不過濾。
- --stop-datetime ：終止解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。預設不過濾。
過濾物件
-
-d, --databases ：只解析目標db的sql，多個庫用空格隔開，如-d db1 db2。可選。預設為空。
-
-t, --tables ：只解析目標table的sql，多張表用空格隔開，如-t tbl1 tbl2。可選。預設為空。
-
--only-dml ：只解析dml，忽略ddl。可選。預設False。
-
--sql-type ：只解析指定型別，支援INSERT, UPDATE, DELETE。多個型別用空格隔開，如--sql-type INSERT DELETE。可選。預設為增刪改都解析。用了此引數但沒填任何型別，則三者都不解析。
示例
建立測試資料
--構造測試表
create table tbl
(
id int primary key,
name varchar(30) not null,
birthday date not null
);
-- 插入3條資料
insert into tbl values(1,'小明','1993-01-02');
insert into tbl values(2,'小華','1994-08-15');
insert into tbl values(3,'小麗','1995-07-12');
-- 模擬誤刪除資料
delete from tbl;
恢復資料
檢視binlog日誌
show master status;
解析出標準SQL
python binlog2sql/binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttbl --start-file='mysql-bin.000052' --start-datetime='2016-12-13 20:25:00' --stop-datetime='2016-12-13 20:30:00'
使用flashback模式生成回滾sql
python binlog2sql/binlog2sql.py -h127.0.0.1 -P3306 -uadmin -p'admin' -dtest -ttbl --start-file='mysql-bin.000052' --start-position=3346 --stop-position=3556 -B > rollback.sql | cat
應用回滾資料
mysql -h127.0.0.1 -P3306 -uadmin -p'admin' < rollback.sql
my2sql
go版MySQL binlog解析工具，通過解析MySQL binlog ，可以生成原始SQL、回滾SQL、去除主鍵的INSERT SQL等，也可以生成DML統計資訊。它基於my2fback、binlog_rollback工具二次開發而來。
限制
- 使用回滾/閃回功能時，binlog格式必須為row,且binlog_row_image=full， DML統計以及大事務分析不受影響
- 只能回滾DML， 不能回滾DDL
- 支援指定-tl時區來解釋binlog中time/datetime欄位的內容。開始時間-start-datetime與結束時間-stop-datetime也會使用此指定的時區， 但注意此開始與結束時間針對的是binlog event header中儲存的unix timestamp。結果中的額外的datetime時間資訊都是binlog event header中的unix timestamp
- 此工具是偽裝成從庫拉取binlog，需要連線資料庫的使用者有SELECT, REPLICATION SLAVE, REPLICATION CLIENT許可權
- MySQL8.0版本需要在配置檔案中加入default_authentication_plugin =mysql_native_password，使用者密碼認證必須是mysql_native_password才能解析
安裝配置
編譯
git clone https://github.com/liuhr/my2sql.git
cd my2sql/
go build .
已編譯二進位制檔案
https://github.com/liuhr/my2sql/blob/master/releases/centOS_release_7.x/my2sql
使用
語法
引數說明
-
-U: 優先使用unique key作為where條件，預設false
-
-mode:
- repl: 偽裝成從庫解析binlog檔案, 預設repl
- file: 離線解析binlog檔案
-
-local-binlog-file: 當指定-mode=file 引數時，需要指定-local-binlog-file binlog檔案相對路徑或絕對路徑,可以連續解析多個binlog檔案，只需要指定起始檔名，程式會自動持續解析下個檔案
-
-add-extraInfo: 是否把database/table/datetime/binlogposition...資訊以註釋的方式加入生成的每條sql前，預設false
-
-big-trx-row-limit n: 找出滿足n條sql的事務，預設500條
-
-databases 、 -tables: 庫及表條件過濾, 以逗號分隔
-
-sql: 要解析的sql型別，可選引數insert、update、delete，預設全部解析
-
-doNotAddPrifixDb: 預設生成insert into db1.tb1 (x1, x1) values (y1, y1)類sql，也可以生成不帶庫名的sql
-
-file-per-table: 為每個表生成一個sql檔案
-
-full-columns: 生成的sql是否帶全列資訊，預設false
-
-ignorePrimaryKeyForInsert: 生成的insert語句是否去掉主鍵，預設false
-
-output-dir: 將生成的結果存放到制定目錄
-
-output-toScreen: 將生成的結果列印到螢幕，預設寫到檔案
-
-threads: 執行緒數，預設8個
-
-work-type:
- 2sql：生成原始sql
- rollback：生成回滾sql
- stats：只統計DML、事務資訊
解析出標準SQL
根據時間點解析出標準SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type 2sql -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
根據pos點解析出標準SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
解析出回滾SQL
根據時間點解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type rollback -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type rollback -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir ./tmpdir
根據pos點解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type rollback -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type rollback -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -output-dir ./tmpdir
統計DML以及大事務
統計時間範圍各個表的DML運算元量，統計一個事務大於500條、時間大於300秒的事務
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type stats -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type stats -start-file mysql-bin.011259 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
統計一段pos點範圍各個表的DML運算元量，統計一個事務大於500條、時間大於300秒的事務
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type stats -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode file -local-binlog-file ./mysql-bin.011259 -work-type stats -start-file mysql-bin.011259 -start-pos 4 -stop-file mysql-bin.011259 -stop-pos 583918266 -big-trx-row-limit 500 -long-trx-seconds 300 -output-dir ./tmpdir
從某一個pos點解析出標準SQL，並且持續列印到螢幕
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -mode repl -work-type 2sql -start-file mysql-bin.011259 -start-pos 4 -output-toScreen
示例
誤刪整張表資料，需要緊急回滾
測試資料
CREATE TABLE `tb1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
--`add_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '新增的時間',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
insert into tb1 values(1, 'biu');
insert into tb1 values(2, 'biao');
commit;
-- 檢視測試表校驗值
checksum table tb1;
flush logs;
-- 檢視當前binlog
show master status;
-- 刪除表資料
delete from tb1;
commit;
生成標準SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode repl -work-type 2sql -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode file -local-binlog-file ./mysql-bin.000046 -work-type 2sql -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
解析出回滾SQL
#偽裝成從庫解析binlog
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode repl -work-type rollback -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
#直接讀取binlog檔案解析
./my2sql -user root -password xxxx -host 127.0.0.1 -port 3306 -databases devdb -tables tb1 -mode file -local-binlog-file ./mysql-bin.000046 -work-type rollback -start-file mysql-bin.000046 -start-datetime "2020-07-16 10:20:00" -stop-datetime "2020-07-16 11:00:00" -output-dir /tmp
-
查看回滾SQL
cat rollback.46.sql
應用回滾SQL恢復資料
mysql -uroot -proot -P3306 -h127.0.0.1 devdb < /tmp/rollback.46.sql
Maxwell
Maxwell 是一個讀取 MySQL binlogs 並將修改行欄位的更新寫入 Kafka, Kinesis, RabbitMQ, Google Cloud Pub/Sub 或 Redis (Pub/Sub or LPUSH) 以作為 JSON 的應用程式。
my2fback
my2fback 實現了基於row格式binlog的回滾閃回功能，讓誤刪除或者誤更新資料，可以不停機不使用備份而快速回滾誤操作。也可以解釋binlog（支援非row格式binlog）生成易讀的SQL。
限制
- 使用回滾/閃回功能時，binlog格式必須為row,且binlog_row_image=full， 其它功能支援非row格式binlog
- 只能回滾DML， 不能回滾DDL
- 支援V4格式的binlog， V3格式的沒測試過，測試與使用結果顯示，mysql5.1，mysql5.5, mysql5.6與mysql5.7的binlog均支援
- 支援指定-tl時區來解釋binlog中time/datetime欄位的內容。開始時間-sdt與結束時間-edt也會使用此指定的時區，
- 但注意此開始與結束時間針對的是binlog event header中儲存的unix timestamp。結果中的額外的datetime時間資訊都是binlog event header中的unix timestamp
- decimal欄位使用float64來表示， 但不損失精度
- 所有字元型別欄位內容按golang的utf8(相當於mysql的utf8mb4)來表示
安裝配置
使用GO>=1.11.x版本來編譯
開啟GO111MODULE引數
編譯linux 平臺
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
編譯windows 平臺
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
沒有開啟GO111MODULE引數
編譯linux 平臺
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
編譯windows 平臺
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o releases/my2fback -ldflags "-s -w" main.go
使用
語法
my2fback -h
my2fback V2.0 By WangJiemin.
E_mail: [email protected]
*****************************************************************************************************
* system_command: /usr/local/bin/my2fback *
* system_goos: linux *
* system_arch: amd64 *
* hostname: test_dbs2.yz.babytree-ops.org *
* hostaddress: 10.10.1.221 *
* blog: https://jiemin.wang *
* read binlog from master, work as a fake slave: ./my2fback -m repl opts... *
* read binlog from local filesystem: ./my2fback -m file opts... mysql-bin.000010 *
*****************************************************************************************************
-C works with -w='stats', keep analyzing transations to last binlog for -m=file, and keep analyzing for -m=repl
-H string
master host, DONOT need to specify when -w=stats. if mode is file, it can be slave or other mysql contains same schema and table structure, not only master. default 127.0.0.1 (default "127.0.0.1")
-I for insert statement when -wtype=2sql, ignore primary key
-M string
valid options are: mysql,mariadb. server of binlog, mysql or mariadb, default mysql (default "mysql")
-P uint
master port, default 3306. DONOT need to specify when -w=stats (default 3306)
-S string
mysql socket file
-U prefer to use unique key instead of primary key to build where condition for delete/update sql
-a Works with -w=2sql|rollback. for update sql, include unchanged columns. for update and delete, use all columns to build where condition.
default false, this is, use changed columns to build set part, use primary/unique key to build where condition
-b int
transaction with affected rows greater or equal to this value is considerated as big transaction. Valid values range from 10 to 30000, default 500 (default 500)
-d Works with -w=2sql|rollback. Prefix table name with database name in sql, ex: insert into db1.tb1 (x1, x1) values (y1, y1). Default true (default true)
-dbs string
only parse database which match any of these regular expressions. The regular expression should be in lower case because database name is translated into lower case and then matched against it.
Multi regular expressions is seperated by comma, default parse all databases. Useless when -w=stats
-dj string
dump table structure to this file. default tabSchame.json (default "tabSchame.json")
-e Works with -w=2sql|rollback. Print database/table/datetime/binlogposition...info on the line before sql, default false
-ebin string
binlog file to stop reading
-edt string
Stop reading the binlog at first event having a datetime equal or posterior to the argument, it should be like this: "2004-12-25 11:25:56"
-epos uint
Stop reading the binlog at position
-f Works with -w=2sql|rollback. one file for one table if true, else one file for all tables. default false. Attention, always one file for one binlog
-i int
works with -w='stats', print stats info each PrintInterval. Valid values range from 1 to 600, default 30 (default 30)
-ies string
for sql which is error to parsed and matched by this regular expression, just print error info, skip it and continue parsing, otherwise stop parsing and exit.
The regular expression should be in lower case, because sql is translated into lower case and then matched against it. (default "^create definer.+trigger")
-k Works with -w=2sql|rollback. wrap result statements with 'begin...commit|rollback'
-l int
transaction with duration greater or equal to this value is considerated as long transaction. Valid values range from 1 to 3600, default 300 (default 300)
-m string
valid options are: repl,file. repl: as a slave to get binlogs from master. file: get binlogs from local filesystem. default file (default "file")
-mid uint
works with -m=repl, this program replicates from master as slave to read binlogs. Must set this server id unique from other slaves, default 1113306 (default 1113306)
-o string
result output dir, default current work dir. Attension, result files could be large, set it to a dir with large free space
-oj
Only use table structure from -rj, do not get or merge table struct from mysql
-ors
for mysql>=5.6.2 and binlog_rows_query_log_events=on, if set, output original sql. default false
-p string
mysql user password. DONOT need to specify when -w=stats
-r int
Works with -w=2sql|rollback. rows for each insert sql. Valid values range from 1 to 500, default 30 (default 30)
-rj string
Works with -w=2sql|rollback, read table structure from this file and merge from mysql
-sbin string
binlog file to start reading
-sdt string
Start reading the binlog at first event having a datetime equal or posterior to the argument, it should be like this: "2004-12-25 11:25:56"
-spos uint
start reading the binlog at position
-sql string
valid options are: insert,update,delete. only parse these types of sql, comma seperated, valid types are: insert, update, delete; default is all(insert,update,delete)
-stsql
when -w=2sql, also parse plain sql and write into result file even if binlog_format is not row. default false
-t uint
Works with -w=2sql|rollback. threads to run, default 4 (default 2)
-tbs string
only parse table which match any of these regular expressions.The regular expression should be in lower case because database name is translated into lower case and then matched against it.
Multi regular expressions is seperated by comma, default parse all tables. Useless when -w=stats
-tl string
time location to parse timestamp/datetime column in binlog, such as Asia/Shanghai. default Local (default "Local")
-u string
mysql user. DONOT need to specify when -w=stats
-v print version
-w string
valid options are: tbldef,stats,2sql,rollback. tbldef: only get table definition structure; 2sql: convert binlog to sqls, rollback: generate rollback sqls, stats: analyze transactions. default: stats (default "stats")
常用引數
-m string
valid options are: repl,file. repl: as a slave to get binlogs from master. file: get binlogs from local filesystem. default file (default "file")
relp: 模仿 SLAVE 的IO_THREAD連線到MASTER獲取BINLOG EVENT
file: 解析本地的BINLOG(default: file)
-w string
valid options are: tbldef,stats,2sql,rollback. tbldef: only get table definition structure; 2sql: convert binlog to sqls, rollback: generate rollback sqls, stats: analyze transactions. default: stats (default "stats")
2sql: 解析成SQL語句
rollback: 解析為回滾語句
-M string
valid options are: mysql,mariadb. server of binlog, mysql or mariadb, default mysql (default "mysql")
選擇是MySQL還是Mariadb, 不選擇預設為MySQL
-e Works with -w=2sql|rollback. Print database/table/datetime/binlogposition...info on the line before sql, default false
在sql之前的行上列印database/table/datetime/binlogposition...info，預設為false
-f Works with -w=2sql|rollback. one file for one table if true, else one file for all tables. default false. Attention, always one file for one binlog
如果為true，則為一個表的一個檔案，否則為所有表的一個檔案。預設為false。注意，一個binlog總是一個檔案
-r int
Works with -w=2sql|rollback. rows for each insert sql. Valid values range from 1 to 500, default 30 (default 30)
INSERT SQL 語句每一行包含的values的行數
-t uint
Works with -w=2sql|rollback. threads to run, default 4 (default 2)
開啟幾個thread進行來執行解析2sql|rollback
-o string
result output dir, default current work dir. Attension, result files could be large, set it to a dir with large free space
輸入的目錄
-k Works with -w=2sql|rollback. wrap result statements with 'begin...commit|rollback'
使用-w = 2sql | rollback。使用'begin ... commit | rollback'包裝結果語句
-l int
transaction with duration greater or equal to this value is considerated as long transaction. Valid values range from 1 to 3600, default 300 (default 300)
-b int
transaction with affected rows greater or equal to this value is considerated as big transaction. Valid values range from 10 to 30000, default 500 (default 500)
-dWorks with -w=2sql|rollback. Prefix table name with database name in sql, ex: insert into db1.tb1 (x1, x1) values (y1, y1). Default true (default true)
使用-w=2sql|rollback。在sql中具有資料庫名稱的字首表名
示例
file本地方式解析binlog
./my2fback -m file -w 2sql -M mysql -t 6 -H ***.***.***.*** -u test -p test -dbs babytree -tbs userbaby -e -f -d -r 20 -k -b 100 -l 10 -o /data/bak/20190626/tosql /data/bak/20190626/mysql-bin.002938
binlog_rollback
回滾/閃回， 前滾， DML分析報告， DDL資訊
binlog_inspector
回滾/閃回，前滾， 分析各表DML情況， 找出長事務與大事務