將MySQL去重操作優化到極致的操作方法
•問題提出
源表t_source結構如下:
item_id int,created_time datetime,modified_time datetime,item_name varchar(20),other varchar(20)
要求:
1.源表中有100萬條資料,其中有50萬created_time和item_name重複。
2.要把去重後的50萬資料寫入到目標表。
3.重複created_time和item_name的多條資料,可以保留任意一條,不做規則限制。
•實驗環境
Linux虛機:CentOS release 6.4;8G實體記憶體(MySQL配置4G);100G機械硬碟;雙物理CPU雙核,共四個處理器;MySQL 8.0.16。
•建立測試表和資料
-- 建立源表 create table t_source ( item_id int,other varchar(20) ); -- 建立目標表 create table t_target like t_source; -- 生成100萬測試資料,其中有50萬created_time和item_name重複 delimiter // create procedure sp_generate_data() begin set @i := 1; while @i<=500000 do set @created_time := date_add('2017-01-01',interval @i second); set @modified_time := @created_time; set @item_name := concat('a',@i); insert into t_source values (@i,@created_time,@modified_time,@item_name,'other'); set @i:=@i+1; end while; commit; set @last_insert_id := 500000; insert into t_source select item_id + @last_insert_id,created_time,date_add(modified_time,interval @last_insert_id second),item_name,'other' from t_source; commit; end // delimiter ; call sp_generate_data(); -- 源表沒有主鍵或唯一性約束,有可能存在兩條完全一樣的資料,所以再插入一條記錄模擬這種情況。 insert into t_source select * from t_source where item_id=1; 源表中有1000001條記錄,去重後的目標表應該有500000條記錄。 mysql> select count(*),count(distinct created_time,item_name) from t_source; +----------+----------------------------------------+ | count(*) | count(distinct created_time,item_name) | +----------+----------------------------------------+ | 1000001 | 500000 | +----------+----------------------------------------+ 1 row in set (1.92 sec)
一、巧用索引與變數
1. 無索引對比測試
(1)使用相關子查詢
truncate t_target; insert into t_target select distinct t1.* from t_source t1 where item_id in (select min(item_id) from t_source t2 where t1.created_time=t2.created_time and t1.item_name=t2.item_name);
這個語句很長時間都出不來結果,只看一下執行計劃吧。
mysql> explain select distinct t1.* from t_source t1 where item_id in -> (select min(item_id) from t_source t2 where t1.created_time=t2.created_time and t1.item_name=t2.item_name); +----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+------------------------------+ | 1 | PRIMARY | t1 | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 100.00 | Using where; Using temporary | | 2 | DEPENDENT SUBQUERY | t2 | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 1.00 | Using where | +----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+------------------------------+ 2 rows in set,3 warnings (0.00 sec)
主查詢和相關子查詢都是全表掃描,一共要掃描100萬*100萬資料行,難怪出不來結果。
(2)使用表連線
truncate t_target; insert into t_target select distinct t1.* from t_source t1,(select min(item_id) item_id,item_name from t_source group by created_time,item_name) t2 where t1.item_id = t2.item_id;
這種方法用時14秒,查詢計劃如下:
mysql> explain select distinct t1.* from t_source t1,item_name) t2 where t1.item_id = t2.item_id; +----+-------------+------------+------------+------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------------+------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ | 1 | PRIMARY | t1 | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 100.00 | Using where; Using temporary | | 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 5 | test.t1.item_id | 10 | 100.00 | Distinct | | 2 | DERIVED | t_source | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 100.00 | Using temporary | +----+-------------+------------+------------+------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ 3 rows in set,1 warning (0.00 sec)
•內層查詢掃描t_source表的100萬行,建立臨時表,找出去重後的最小item_id,生成匯出表derived2,此匯出表有50萬行。
•MySQL會在匯出表derived2上自動建立一個item_id欄位的索引auto_key0。
•外層查詢也要掃描t_source表的100萬行資料,在與匯出表做連結時,對t_source表每行的item_id,使用auto_key0索引查詢匯出表中匹配的行,並在此時優化distinct操作,在找到第一個匹配的行後即停止查詢同樣值的動作。
(3)使用變數
set @a:='1000-01-01 00:00:00'; set @b:=' '; set @f:=0; truncate t_target; insert into t_target select item_id,modified_time,other from (select t0.*,if(@a=created_time and @b=item_name,@f:=0,@f:=1) f,@a:=created_time,@b:=item_name from (select * from t_source order by created_time,item_name) t0) t1 where f=1;
這種方法用時13秒,查詢計劃如下:
mysql> explain select item_id,other -> from -> (select t0.*,@b:=item_name -> from -> (select * from t_source order by created_time,item_name) t0) t1 where f=1; +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ | 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | const | 10 | 100.00 | NULL | | 2 | DERIVED | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 100.00 | NULL | | 3 | DERIVED | t_source | NULL | ALL | NULL | NULL | NULL | NULL | 997282 | 100.00 | Using filesort | +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ 3 rows in set,5 warnings (0.00 sec)
•最內層的查詢掃描t_source表的100萬行,並使用檔案排序,生成匯出表derived3。
•第二層查詢要掃描derived3的100萬行,生成匯出表derived2,完成變數的比較和賦值,並自動建立一個匯出列f上的索引auto_key0。
•最外層使用auto_key0索引掃描derived2得到去重的結果行。
與上面方法2比較,總的掃描行數不變,都是200萬行。只存在一點微小的差別,這次自動生成的索引是在常量列 f 上,而表關聯自動生成的索引是在item_id列上,所以查詢時間幾乎相同。
至此,我們還沒有在源表上建立任何索引。無論使用哪種寫法,要查重都需要對created_time和item_name欄位進行排序,因此很自然地想到,如果在這兩個欄位上建立聯合索引,利用索引本身有序的特性消除額外排序,從而提高查詢效能。
2. 建立created_time和item_name上的聯合索引對比測試
-- 建立created_time和item_name欄位的聯合索引 create index idx_sort on t_source(created_time,item_id); analyze table t_source;
(1)使用相關子查詢
truncate t_target; insert into t_target select distinct t1.* from t_source t1 where item_id in (select min(item_id) from t_source t2 where t1.created_time=t2.created_time and t1.item_name=t2.item_name);
本次用時19秒,查詢計劃如下:
mysql> explain select distinct t1.* from t_source t1 where item_id in -> (select min(item_id) from t_source t2 where t1.created_time=t2.created_time and t1.item_name=t2.item_name); +----+--------------------+-------+------------+------+---------------+----------+---------+----------------------------------------+--------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------------+-------+------------+------+---------------+----------+---------+----------------------------------------+--------+----------+------------------------------+ | 1 | PRIMARY | t1 | NULL | ALL | NULL | NULL | NULL | NULL | 997281 | 100.00 | Using where; Using temporary | | 2 | DEPENDENT SUBQUERY | t2 | NULL | ref | idx_sort | idx_sort | 89 | test.t1.created_time,test.t1.item_name | 2 | 100.00 | Using index | +----+--------------------+-------+------------+------+---------------+----------+---------+----------------------------------------+--------+----------+------------------------------+ 2 rows in set,3 warnings (0.00 sec)
•外層查詢的t_source表是驅動表,需要掃描100萬行。
•對於驅動表每行的item_id,通過idx_sort索引查詢出兩行資料。
(2)使用表連線
truncate t_target; insert into t_target select distinct t1.* from t_source t1,item_name) t2 where t1.item_id = t2.item_id;
本次用時13秒,查詢計劃如下:
mysql> explain select distinct t1.* from t_source t1,-> (select min(item_id) item_id,item_name) t2 -> where t1.item_id = t2.item_id; +----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ | 1 | PRIMARY | t1 | NULL | ALL | NULL | NULL | NULL | NULL | 997281 | 100.00 | Using where; Using temporary | | 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 5 | test.t1.item_id | 10 | 100.00 | Distinct | | 2 | DERIVED | t_source | NULL | index | idx_sort | idx_sort | 94 | NULL | 997281 | 100.00 | Using index | +----+-------------+------------+------------+-------+---------------+-------------+---------+-----------------+--------+----------+------------------------------+ 3 rows in set,1 warning (0.00 sec)
和沒有索引相比,子查詢雖然從全表掃描變為了全索引掃描,但還是需要掃描100萬行記錄。因此查詢效能提升並不是明顯。
(3)使用變數
set @a:='1000-01-01 00:00:00'; set @b:=' '; set @f:=0; truncate t_target; insert into t_target select item_id,item_name) t0) t1 where f=1;
本次用時13秒,查詢計劃與沒有索引時的完全相同。可見索引對這種寫法沒有作用。能不能消除巢狀,只用一層查詢出結果呢?
(4)使用變數,並且消除巢狀查詢
set @a:='1000-01-01 00:00:00'; set @b:=' '; truncate t_target; insert into t_target select * from t_source force index (idx_sort) where (@a!=created_time or @b!=item_name) and (@a:=created_time) is not null and (@b:=item_name) is not null order by created_time,item_name;
本次用時12秒,查詢計劃如下:
mysql> explain select * from t_source force index (idx_sort) -> where (@a!=created_time or @b!=item_name) and (@a:=created_time) is not null and (@b:=item_name) is not null -> order by created_time,item_name; +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-------------+ | 1 | SIMPLE | t_source | NULL | index | NULL | idx_sort | 94 | NULL | 997281 | 99.00 | Using where | +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-------------+ 1 row in set,3 warnings (0.00 sec)
該語句具有以下特點:
•消除了巢狀子查詢,只需要對t_source表進行一次全索引掃描,查詢計劃已達最優。
•無需distinct二次查重。
•變數判斷與賦值只出現在where子句中。
•利用索引消除了filesort。
在MySQL 8之前,該語句是單執行緒去重的最佳解決方案。仔細分析這條語句,發現它巧妙地利用了SQL語句的邏輯查詢處理步驟和索引特性。一條SQL查詢的邏輯步驟為:
1.執行笛卡爾乘積(交叉連線)
2.應用ON篩選器(連線條件)
3.新增外部行(outer join)
4.應用where篩選器
5.分組
6.應用cube或rollup
7.應用having篩選器
8.處理select列表
9.應用distinct子句
10.應用order by子句
11.應用limit子句
每條查詢語句的邏輯執行步驟都是這11步的子集。拿這條查詢語句來說,其執行順序為:強制通過索引idx_sort查詢資料行 -> 應用where篩選器 -> 處理select列表 -> 應用order by子句。
為了使變數能夠按照created_time和item_name的排序順序進行賦值和比較,必須按照索引順序查詢資料行。這裡的force index (idx_sort)提示就起到了這個作用,必須這樣寫才能使整條查重語句成立。否則,因為先掃描表才處理排序,因此不能保證變數賦值的順序,也就不能確保查詢結果的正確性。order by子句同樣不可忽略,否則即使有force index提示,MySQL也會使用全表掃描而不是全索引掃描,從而使結果錯誤。索引同時保證了created_time,item_name的順序,避免了檔案排序。force index (idx_sort)
提示和order by子句缺一不可,索引idx_sort在這裡可謂恰到好處、一舉兩得。
查詢語句開始前,先給變數初始化為資料中不可能出現的值,然後進入where子句從左向右判斷。先比較變數和欄位的值,再將本行created_time和item_name的值賦給變數,按created_time、item_name的順序逐行處理。item_name是字串型別,(@b:=item_name)不是有效的布林表示式,因此要寫成(@b:=item_name) is not null。
最後補充一句,這裡忽略了“insert into t_target select * from t_source group by created_time,item_name
;”的寫法,因為它受“sql_mode='ONLY_FULL_GROUP_BY'
”的限制。
二、利用視窗函式
MySQL 8中新增的視窗函式使得原來麻煩的去重操作變得很簡單。
truncate t_target; insert into t_target select item_id,other from (select *,row_number() over(partition by created_time,item_name) as rn from t_source) t where rn=1;
這個語句執行只需要12秒,而且寫法清晰易懂,其查詢計劃如下:
mysql> explain select item_id,other -> from (select *,item_name) as rn -> from t_source) t where rn=1; +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ | 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 8 | const | 10 | 100.00 | NULL | | 2 | DERIVED | t_source | NULL | ALL | NULL | NULL | NULL | NULL | 997281 | 100.00 | Using filesort | +----+-------------+------------+------------+------+---------------+-------------+---------+-------+--------+----------+----------------+ 2 rows in set,2 warnings (0.00 sec)
該查詢對t_source表進行了一次全表掃描,同時用filesort對錶按分割槽欄位created_time、item_name進行了排序。外層查詢從每個分割槽中保留一條資料。因為重複created_time
和item_name
的多條資料中可以保留任意一條,所以oevr中不需要使用order by子句。
從執行計劃看,視窗函式去重語句似乎沒有消除巢狀查詢的變數去重好,但此方法實際執行是最快的。
MySQL視窗函式說明參見“https://dev.mysql.com/doc/refman/8.0/en/window-functions.html”。
三、多執行緒並行執行
前面已經將單條查重語句調整到最優,但還是以單執行緒方式執行。能否利用多處理器,讓去重操作多執行緒並行執行,從而進一步提高速度呢?比如我的實驗環境是4處理器,如果使用4個執行緒同時執行查重SQL,理論上應該接近4倍的效能提升。
1. 資料分片
在生成測試資料時,created_time採用每條記錄加一秒的方式,也就是最大和在最小的時間差為50萬秒,而且資料均勻分佈,因此先把資料平均分成4份。
(1)查詢出4份資料的created_time邊界值
mysql> select date_add('2017-01-01',interval 125000 second) dt1,-> date_add('2017-01-01',interval 2*125000 second) dt2,interval 3*125000 second) dt3,-> max(created_time) dt4 -> from t_source; +---------------------+---------------------+---------------------+---------------------+ | dt1 | dt2 | dt3 | dt4 | +---------------------+---------------------+---------------------+---------------------+ | 2017-01-02 10:43:20 | 2017-01-03 21:26:40 | 2017-01-05 08:10:00 | 2017-01-06 18:53:20 | +---------------------+---------------------+---------------------+---------------------+ 1 row in set (0.00 sec)
(2)檢視每份資料的記錄數,確認資料平均分佈
mysql> select case when created_time >= '2017-01-01' -> and created_time < '2017-01-02 10:43:20' -> then '2017-01-01' -> when created_time >= '2017-01-02 10:43:20' -> and created_time < '2017-01-03 21:26:40' -> then '2017-01-02 10:43:20' -> when created_time >= '2017-01-03 21:26:40' -> and created_time < '2017-01-05 08:10:00' -> then '2017-01-03 21:26:40' -> else '2017-01-05 08:10:00' -> end min_dt,-> case when created_time >= '2017-01-01' -> and created_time < '2017-01-02 10:43:20' -> then '2017-01-02 10:43:20' -> when created_time >= '2017-01-02 10:43:20' -> and created_time < '2017-01-03 21:26:40' -> then '2017-01-03 21:26:40' -> when created_time >= '2017-01-03 21:26:40' -> and created_time < '2017-01-05 08:10:00' -> then '2017-01-05 08:10:00' -> else '2017-01-06 18:53:20' -> end max_dt,-> count(*) -> from t_source -> group by case when created_time >= '2017-01-01' -> and created_time < '2017-01-02 10:43:20' -> then '2017-01-01' -> when created_time >= '2017-01-02 10:43:20' -> and created_time < '2017-01-03 21:26:40' -> then '2017-01-02 10:43:20' -> when created_time >= '2017-01-03 21:26:40' -> and created_time < '2017-01-05 08:10:00' -> then '2017-01-03 21:26:40' -> else '2017-01-05 08:10:00' -> end,-> case when created_time >= '2017-01-01' -> and created_time < '2017-01-02 10:43:20' -> then '2017-01-02 10:43:20' -> when created_time >= '2017-01-02 10:43:20' -> and created_time < '2017-01-03 21:26:40' -> then '2017-01-03 21:26:40' -> when created_time >= '2017-01-03 21:26:40' -> and created_time < '2017-01-05 08:10:00' -> then '2017-01-05 08:10:00' -> else '2017-01-06 18:53:20' -> end; +---------------------+---------------------+----------+ | min_dt | max_dt | count(*) | +---------------------+---------------------+----------+ | 2017-01-01 | 2017-01-02 10:43:20 | 249999 | | 2017-01-02 10:43:20 | 2017-01-03 21:26:40 | 250000 | | 2017-01-03 21:26:40 | 2017-01-05 08:10:00 | 250000 | | 2017-01-05 08:10:00 | 2017-01-06 18:53:20 | 250002 | +---------------------+---------------------+----------+ 4 rows in set (4.86 sec)
4份資料的並集應該覆蓋整個源資料集,並且資料之間是不重複的。也就是說4份資料的created_time要連續且互斥,連續保證處理全部資料,互斥確保了不需要二次查重。實際上這和時間範圍分割槽的概念類似,或許用分割槽表更好些,只是這裡省略了重建表的步驟。
2. 建立查重的儲存過程
有了以上資訊我們就可以寫出4條語句處理全部資料。為了呼叫介面儘量簡單,建立下面的儲存過程。
delimiter // create procedure sp_unique(i smallint) begin set @a:='1000-01-01 00:00:00'; set @b:=' '; if (i<4) then insert into t_target select * from t_source force index (idx_sort) where created_time >= date_add('2017-01-01',interval (i-1)*125000 second) and created_time < date_add('2017-01-01',interval i*125000 second) and (@a!=created_time or @b!=item_name) and (@a:=created_time) is not null and (@b:=item_name) is not null order by created_time,item_name; else insert into t_target select * from t_source force index (idx_sort) where created_time >= date_add('2017-01-01',interval (i-1)*125000 second) and created_time <= date_add('2017-01-01',item_name; end if; end //
查詢語句的執行計劃如下:
mysql> explain select * from t_source force index (idx_sort) -> where created_time >= date_add('2017-01-01',interval (1-1)*125000 second) -> and created_time < date_add('2017-01-01',interval 1*125000 second) -> and (@a!=created_time or @b!=item_name) -> and (@a:=created_time) is not null -> and (@b:=item_name) is not null -> order by created_time,item_name; +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-----------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-----------------------+ | 1 | SIMPLE | t_source | NULL | range | idx_sort | idx_sort | 6 | NULL | 498640 | 100.00 | Using index condition | +----+-------------+----------+------------+-------+---------------+----------+---------+------+--------+----------+-----------------------+ 1 row in set,3 warnings (0.00 sec)
MySQL優化器進行索引範圍掃描,並且使用索引條件下推(ICP)優化查詢。
3. 並行執行
下面分別使用shell後臺程序和MySQL Schedule Event實現並行。
(1)shell後臺程序
•建立duplicate_removal.sh檔案,內容如下:
#!/bin/bash mysql -vvv -u root -p123456 test -e "truncate t_target" &>/dev/null date '+%H:%M:%S' for y in {1..4} do sql="call sp_unique($y)" mysql -vvv -u root -p123456 test -e "$sql" &>par_sql1_$y.log & done wait date '+%H:%M:%S'
•執行指令碼檔案
./duplicate_removal.sh
執行輸出如下:
[mysql@hdp2~]$./duplicate_removal.sh 14:27:30 14:27:35
這種方法用時5秒,並行執行的4個過程呼叫分別用時為4.87秒、4.88秒、4.91秒、4.73秒:
[mysql@hdp2~]$cat par_sql1_1.log | sed '/^$/d' mysql: [Warning] Using a password on the command line interface can be insecure. -------------- call sp_unique(1) -------------- Query OK,124999 rows affected (4.87 sec) Bye [mysql@hdp2~]$cat par_sql1_2.log | sed '/^$/d' mysql: [Warning] Using a password on the command line interface can be insecure. -------------- call sp_unique(2) -------------- Query OK,125000 rows affected (4.88 sec) Bye [mysql@hdp2~]$cat par_sql1_3.log | sed '/^$/d' mysql: [Warning] Using a password on the command line interface can be insecure. -------------- call sp_unique(3) -------------- Query OK,125000 rows affected (4.91 sec) Bye [mysql@hdp2~]$cat par_sql1_4.log | sed '/^$/d' mysql: [Warning] Using a password on the command line interface can be insecure. -------------- call sp_unique(4) -------------- Query OK,125001 rows affected (4.73 sec) Bye [mysql@hdp2~]$
可以看到,每個過程的執行時間均4.85,因為是並行執行,總的過程執行時間為最慢的4.91秒,比單執行緒速度提高了2.5倍。
(2)MySQL Schedule Event
•建立事件歷史日誌表
-- 用於檢視事件執行時間等資訊 create table t_event_history ( dbname varchar(128) not null default '',eventname varchar(128) not null default '',starttime datetime(3) not null default '1000-01-01 00:00:00',endtime datetime(3) default null,issuccess int(11) default null,duration int(11) default null,errormessage varchar(512) default null,randno int(11) default null );
•為每個併發執行緒建立一個事件
delimiter // create event ev1 on schedule at current_timestamp + interval 1 hour on completion preserve disable do begin declare r_code char(5) default '00000'; declare r_msg text; declare v_error integer; declare v_starttime datetime default now(3); declare v_randno integer default floor(rand()*100001); insert into t_event_history (dbname,eventname,starttime,randno) #作業名 values(database(),'ev1',v_starttime,v_randno); begin #異常處理段 declare continue handler for sqlexception begin set v_error = 1; get diagnostics condition 1 r_code = returned_sqlstate,r_msg = message_text; end; #此處為實際呼叫的使用者程式過程 call sp_unique(1); end; update t_event_history set endtime=now(3),issuccess=isnull(v_error),duration=timestampdiff(microsecond,now(3)),errormessage=concat('error=',r_code,',message=',r_msg),randno=null where starttime=v_starttime and randno=v_randno; end // create event ev2 on schedule at current_timestamp + interval 1 hour on completion preserve disable do begin declare r_code char(5) default '00000'; declare r_msg text; declare v_error integer; declare v_starttime datetime default now(3); declare v_randno integer default floor(rand()*100001); insert into t_event_history (dbname,'ev2',r_msg = message_text; end; #此處為實際呼叫的使用者程式過程 call sp_unique(2); end; update t_event_history set endtime=now(3),randno=null where starttime=v_starttime and randno=v_randno; end // create event ev3 on schedule at current_timestamp + interval 1 hour on completion preserve disable do begin declare r_code char(5) default '00000'; declare r_msg text; declare v_error integer; declare v_starttime datetime default now(3); declare v_randno integer default floor(rand()*100001); insert into t_event_history (dbname,'ev3',r_msg = message_text; end; #此處為實際呼叫的使用者程式過程 call sp_unique(3); end; update t_event_history set endtime=now(3),randno=null where starttime=v_starttime and randno=v_randno; end // create event ev4 on schedule at current_timestamp + interval 1 hour on completion preserve disable do begin declare r_code char(5) default '00000'; declare r_msg text; declare v_error integer; declare v_starttime datetime default now(3); declare v_randno integer default floor(rand()*100001); insert into t_event_history (dbname,'ev4',r_msg = message_text; end; #此處為實際呼叫的使用者程式過程 call sp_unique(4); end; update t_event_history set endtime=now(3),randno=null where starttime=v_starttime and randno=v_randno; end //
為了記錄每個事件執行的時間,在事件定義中增加了操作日誌表的邏輯,因為每個事件中只多執行了一條insert,一條update,4個事件總共多執行8條很簡單的語句,對測試的影響可以忽略不計。執行時間精確到毫秒。
•觸發事件執行
mysql -vvv -u root -p123456 test -e "truncate t_target;alter event ev1 on schedule at current_timestamp enable;alter event ev2 on schedule at current_timestamp enable;alter event ev3 on schedule at current_timestamp enable;alter event ev4 on schedule at current_timestamp enable;"
該命令列順序觸發了4個事件,但不會等前一個執行完才執行下一個,而是立即向下執行。這可從命令的輸出可以清除看到:
[mysql@hdp2~]$mysql -vvv -u root -p123456 test -e "truncate t_target;alter event ev1 on schedule at current_timestamp enable;alter event ev2 on schedule at current_timestamp enable;alter event ev3 on schedule at current_timestamp enable;alter event ev4 on schedule at current_timestamp enable;" mysql: [Warning] Using a password on the command line interface can be insecure. -------------- truncate t_target -------------- Query OK,0 rows affected (0.06 sec) -------------- alter event ev1 on schedule at current_timestamp enable -------------- Query OK,0 rows affected (0.02 sec) -------------- alter event ev2 on schedule at current_timestamp enable -------------- Query OK,0 rows affected (0.00 sec) -------------- alter event ev3 on schedule at current_timestamp enable -------------- Query OK,0 rows affected (0.02 sec) -------------- alter event ev4 on schedule at current_timestamp enable -------------- Query OK,0 rows affected (0.00 sec) Bye [mysql@hdp2~]$
•檢視事件執行日誌
mysql> select * from test.t_event_history; +--------+-----------+-------------------------+-------------------------+-----------+----------+--------------+--------+ | dbname | eventname | starttime | endtime | issuccess | duration | errormessage | randno | +--------+-----------+-------------------------+-------------------------+-----------+----------+--------------+--------+ | test | ev1 | 2019-07-31 14:38:04.000 | 2019-07-31 14:38:09.389 | 1 | 5389000 | NULL | NULL | | test | ev2 | 2019-07-31 14:38:04.000 | 2019-07-31 14:38:09.344 | 1 | 5344000 | NULL | NULL | | test | ev3 | 2019-07-31 14:38:05.000 | 2019-07-31 14:38:09.230 | 1 | 4230000 | NULL | NULL | | test | ev4 | 2019-07-31 14:38:05.000 | 2019-07-31 14:38:09.344 | 1 | 4344000 | NULL | NULL | +--------+-----------+-------------------------+-------------------------+-----------+----------+--------------+--------+ 4 rows in set (0.00 sec)
可以看到,每個過程的執行均為4.83秒,又因為是並行執行的,因此總的執行之間為最慢的5.3秒,優化效果和shell後臺程序方式幾乎相同。
總結
以上所述是小編給大家介紹的將MySQL去重操作優化到極致的操作方法,希望對大家有所幫助,如果大家有任何疑問請給我留言,小編會及時回覆大家的。在此也非常感謝大家對我們網站的支援!
如果你覺得本文對你有幫助,歡迎轉載,煩請註明出處,謝謝!