my16_sql_thread執行慢導致主從延遲高的一個情景
現象:從庫延遲高，查看slave status發現sql_thread執行語句的速度比主庫慢，這樣的延遲會一直高下去，下面是排查的一些過程
1. 檢查了從庫的配置，磁盤的寫入速度的確沒有主庫高
2. iostat -m 1 10查看磁盤寫入 ，從庫為2M/S，主庫也就3M/S，從庫磁盤沒主庫高，但這個檔次的速度應該不影響
3. 將sync_binlog從1設置為，0，3，10，100 無效果，innodb_flush_log_at_trx_commit的值為2
4. 增加slave_parallel_workers的值，無效果
5. 然後又排查了內存相關參數，沒有明顯不合理的地方
6. 最後要從binlog中解析SQL，看看執行的都是哪些SQL，哪些表，在從binlog抽取SQL之前，查看了一下 show full processlist，看到了 System lock
然後就重啟了一下slave，這是個作為備份用的從庫，業務不訪問，當時沒有考慮鎖的問題，應該再排查一下，有沒有其他鎖
解決方法為重啟一下slave;
>show full processlist;
+---------+-------------+-----------+-------------+---------+---------+---------------------------------------------+--------------------------------------------------------------------------------------------+-----------+---------------+
+---------+-------------+-----------+-------------+---------+---------+---------------------------------------------+--------------------------------------------------------------------------------------------+-----------+---------------+
| 2 | system user | | NULL | Connect | 0 | Waiting for dependent transaction to commit | NULL | 0 | 0 |
| 3 | system user | | NULL | Connect | 67873 | System lock | UPDATE......... |
stop slave;
start slave;
>show full processlist;
+---------+-------------+-----------+-------------+---------+-------+---------------------------------------------+-----------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+---------+-------------+-----------+-------------+---------+-------+---------------------------------------------+-----------------------+-----------+---------------+
| 3506710 | root | localhost | ad_dianjing | Query | 0 | starting | show full processlist | 0 | 0 |
| 3508115 | system user | | NULL | Connect | 53 | Waiting for master to send event | NULL | 0 | 0 |
| 3508116 | system user | | NULL | Connect | 0 | Waiting for dependent transaction to commit | NULL | 0 | 0 |
| 3508117 | system user | | NULL | Connect | 56827 | System lock | NULL | 0 | 0 |
| 3508118 | system user | | NULL | Connect | 56828 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508119 | system user | | NULL | Connect | 56828 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508120 | system user | | NULL | Connect | 56977 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508121 | system user | | NULL | Connect | 56981 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508122 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508123 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508124 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508125 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508126 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508127 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508128 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508129 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508130 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508131 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
| 3508132 | system user | | NULL | Connect | 57852 | Waiting for an event from Coordinator | NULL | 0 | 0 |
+---------+-------------+-----------+-------------+---------+-------+---------------------------------------------+-----------------------+-----------+---------------+
19 rows in set (0.00 sec)
Seconds_Behind_Master: 54622
然後就可以看到Seconds_Behind_Master的數值以每秒100的速度往下降
System lock的產生原因分析可以參考下面這篇文章
http://blog.itpub.net/7728585/viewspace-2149659
my16_sql_thread執行慢導致主從延遲高的一個情景