1. 程式人生 > >1197多行事務要求更大的max_binlog_cache_size處理與優化

1197多行事務要求更大的max_binlog_cache_size處理與優化

last tmp 配置 更改 delay clas rep 逗號 cert

1197多語句事務要求更大的max_binlog_cache_size報錯 binlog_cache_size:為每個session 分配的內存,在事務過程中用來存儲二進制日誌的緩存,提高記錄bin-log的效率。沒有什麽大事務,dml也不是很頻繁的情況下可以設置小一點,如果事務大而且多,dml操作也頻繁,則可以適當的調大一點。 max_binlog_cache_size設置的參考標準 Binlog_cache_disk_use表示因為我們binlog_cache_size設計的內存不足導致緩存二進制日誌用到了臨時文件的次數;Binlog_cache_use 表示用binlog_cache_size緩存的次數,當對應的Binlog_cache_disk_use 值比較大的時候 我們可以考慮適當的調高 binlog_cache_size 對應的值
【故障情景】 通過腳本以load的方式導入數據時,出現多行事務需要的max_binlog_cache_size空間不足。該數據文件HAOHUAN.txt只包含以逗號分隔的500萬行左右的數據,每行四列,文件大小為270M
1 [root@172-16-3-190 shells]# bash +x load_data_into.sh 
2                 文件的總數為:1 
3                 文件名為:/tmp/load/HAOHUAN.txt 
4 當前正在處理的文件是:/tmp/load/HAOHUAN.txt
5 load data infile /tmp/load/HAOHUAN.txt
into table practice.temp_baofoo_unbind fields terminated by , lines terminated by \n (merchant_no,bank_code,bank_card,protocol_no) 6 Warning: Using a password on the command line interface can be insecure. 7 ERROR 1197 (HY000) at line 1: Multi-statement transaction required more than max_binlog_cache_size
bytes of storage; increase this mysqld variable and try again
【故障排查】 查看max_binlog_cache_size的大小,發現數據文件的大小確實較max_binlog_cache_size的值要小,如果max_binlog_cache_size的大小不足以存放事務的binlog,那麽會臨時使用磁盤臨時文件來存放binlog,通過查看Binlog_cache_disk_use發現使用臨時文件存放的次數為1。因此增大max_binlog_cache_size的值到300M,再次執行腳本發現還是報相同的錯誤。且使用臨時文件的次數為2,使用臨時文件的存放binlog的總次數也相應由15增加到了16次。
 1 mysql> show global variables like %binlog_cache%;
 2 +-----------------------+-----------+
 3 | Variable_name | Value |
 4 +-----------------------+-----------+
 5 | binlog_cache_size | 16777216 |
 6 | max_binlog_cache_size | 268435456 |
 7 +-----------------------+-----------+
 8 2 rows in set (0.00 sec)
 9 
10 mysql> show global status like %binlog_cache%;
11 +-----------------------+-------+
12 | Variable_name | Value |
13 +-----------------------+-------+
14 | Binlog_cache_disk_use | 1 |
15 | Binlog_cache_use | 15 |
16 +-----------------------+-------+
17 2 rows in set (0.00 sec)
18 
19 mysql> set @@global.max_binlog_cache_size=300000000;
20 Query OK, 0 rows affected, 1 warning (0.00 sec)
21 
22 [root@172-16-3-190 shells]# bash +x load_data_into.sh          
23                 文件的總數為:1 
24                 文件名為:/tmp/load/HAOHUAN.txt 
25 當前正在處理的文件是:/tmp/load/HAOHUAN.txt
26 load data infile /tmp/load/HAOHUAN.txt into table practice.temp_baofoo_unbind fields terminated by , lines terminated by \n (merchant_no,bank_code,bank_card,protocol_no)
27 Warning: Using a password on the command line interface can be insecure.
28 ERROR 1197 (HY000) at line 1: Multi-statement transaction required more than max_binlog_cache_size bytes of storage; increase this mysqld variable and try again
29 
30 mysql> show global status like %binlog_cache%;         
31 +-----------------------+-------+
32 | Variable_name | Value |
33 +-----------------------+-------+
34 | Binlog_cache_disk_use | 2 |
35 | Binlog_cache_use | 16 |
36 +-----------------------+-------+
37 2 rows in set (0.00 sec)

無奈直接增加max_binlog_cache_size的值到500M時問題才解決(後經test實際給到400M也可以load成功),但是slave上的值沒有及時改動,因而SQL同步線程報錯,stop同步線程,同master一樣的更改後,同步才算正常

 1 mysql> set @@global.max_binlog_cache_size=500000000;
 2 Query OK, 0 rows affected, 1 warning (0.00 sec)
 3 
 4 mysql> show slave status \G;
 5 *************************** 1. row ***************************
 6                Slave_IO_State: Waiting for master to send event
 7                   Master_Host: 172.16.3.190
 8                   Master_User: repl
 9                   Master_Port: 3309
10                 Connect_Retry: 30
11               Master_Log_File: binlog.000018
12           Read_Master_Log_Pos: 120
13                Relay_Log_File: relay_bin.000006
14                 Relay_Log_Pos: 6973
15         Relay_Master_Log_File: binlog.000017
16              Slave_IO_Running: Yes
17             Slave_SQL_Running: Yes
18               Replicate_Do_DB: 
19           Replicate_Ignore_DB: 
20            Replicate_Do_Table: 
21        Replicate_Ignore_Table: 
22       Replicate_Wild_Do_Table: 
23   Replicate_Wild_Ignore_Table: 
24                    Last_Errno: 1197
25                    Last_Error: Could not execute Write_rows event on table practice.temp_baofoo_unbind; Multi-statement transaction required more than max_binlog_cache_size bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the events master log binlog.000017, end_log_pos 268602107
26                  Skip_Counter: 0
27           Exec_Master_Log_Pos: 11408
28               Relay_Log_Space: 333526981
29               Until_Condition: None
30                Until_Log_File: 
31                 Until_Log_Pos: 0
32            Master_SSL_Allowed: No
33            Master_SSL_CA_File: 
34            Master_SSL_CA_Path: 
35               Master_SSL_Cert: 
36             Master_SSL_Cipher: 
37                Master_SSL_Key: 
38         Seconds_Behind_Master: 208
39 Master_SSL_Verify_Server_Cert: No
40                 Last_IO_Errno: 0
41                 Last_IO_Error: 
42                Last_SQL_Errno: 1197
43                Last_SQL_Error: Could not execute Write_rows event on table practice.temp_baofoo_unbind; Multi-statement transaction required more than max_binlog_cache_size bytes of storage; increase this mysqld variable and try again, Error_code: 1197; Writing one row to the row-based binary log failed, Error_code: 1534; handler error HA_ERR_RBR_LOGGING_FAILED; the events master log binlog.000017, end_log_pos 268602107
44   Replicate_Ignore_Server_Ids: 
45              Master_Server_Id: 1903309
46                   Master_UUID: 1b589d80-f450-11e7-9150-525400f4ecb2
47              Master_Info_File: /opt/app/mysql_3309/logs/master.info
48                     SQL_Delay: 0
49           SQL_Remaining_Delay: NULL
50       Slave_SQL_Running_State: Reading event from the relay log
51            Master_Retry_Count: 86400
52                   Master_Bind: 
53       Last_IO_Error_Timestamp: 
54      Last_SQL_Error_Timestamp: 180803 17:39:08
55                Master_SSL_Crl: 
56            Master_SSL_Crlpath: 
57            Retrieved_Gtid_Set: 
58             Executed_Gtid_Set: 
59                 Auto_Position: 0
60 1 row in set (0.00 sec)
61 
62 mysql> stop slave;
63 Query OK, 0 rows affected (1 min 10.64 sec)
【故障總結】 max_binlog_cache_size參數時動態參數,該值的設置可以參考binlog_cache_use的大小來相應增加。load導入或者delete數據的大小必須要大於max_binlog_cache_size的值,多行事務才能成功執行。該參數值修改後,註意要與配置文件中的值大小一致。

1197多行事務要求更大的max_binlog_cache_size處理與優化