mysql小白系列_13 Online DDL
Online DDL:
一.FIC之前的添加刪除二級索引:
- 1.首先創建臨時表,定義目標新表的選項和索引
- 2.逐行拷貝數據到臨時表
- 3.插入行時更新索引信息
- 4.數據全部被拷貝到新表後,刪除舊表,rename新表
為保證數據一致性,舊表復制數據到新表的全程只允許讀,不允許改
二.5.1 5.5FIC新特性 fast index creation
- 1.創建二級索引給表加上共享S鎖
- 2.創建過程不需要重建表
- 3.刪除二級索引只需要更新內部視圖,並標記索引的空間可用,去除數據庫元數據的關於該索引的定義
依然只能讀,不允許改,但加快了修改索引的速度
因為聚簇索引的特性,主鍵索引修改依然需要通過舊方式copy table過程
三.5.6之後
- 1.之前FIC只對二級索引創建刪除有效,5.6之後將此特性擴展到添加列、刪除列,修改列類型、列重命名、設置默認值
Online DDL選項
索引的copy與inplace創建
一.ALGORITHM=COPY
- 1.新建帶索引(主鍵索引)的臨時表
- 2.將原表鎖定,禁止DML操作,只允許select查詢
- 3.將原表數據拷貝到臨時表
- 4.禁止讀寫,進行rename,升級字典鎖
- 5.完成創建索引操作
需要記錄undo和redo,效率不如inplace,短期占用buffer pool,影響性能
二.ALGORITHM=INPLACE
- 1.創建索引(二級索引,主鍵+普通字段)數據字典
- 2.加表共享鎖S,禁止DML,允許select查詢
- 3.讀取聚簇索引,構造新的索引項,排序並插入新索引
- 5.等待打開當前表的所有只讀事務提交
- 6.創建索引結束
避免重建表帶來的IO和CPU消耗,保證DDL期間的性能和並發
三.ALGORITHM=DEFAULT
- 1.
alter table
後面什麽都不加的時候,默認是這個方式 - 2.如果
old_alter_table
為OFF,默認就是inplace
方式 - 3.inplace不支持則進行copy
四.lock選項
控制是否鎖表,根據不同的DDL操作類型表現不同,mysql原則是盡量不鎖表,但是修改主鍵這樣的昂貴操作不得不鎖表
- 1.
LOCK=NONE
ALGORITHM=COPY
的默認lock級別 - 2.
LOCK=SHARED
,DDL期間表上的寫操作會被阻塞,但是不影響select - 3.
LOCK=DEFAULT
,讓mysql自己判斷lock模式,原則是盡量不鎖表 - 4.
LOCK=EXCLUSIVE
,DDL期間該表不可用,堵塞任何讀寫請求,使用場景:- 最短時間內完成
- 短時間表不可用能剛接受
註意:
- 1.任何模式下,online DDL開始之前都需要一個短時間的排它鎖X來準備環境
- 2.當alter命令發出後,會首先等待該表上的其他操作完成
- 3.alter命令之後的其他請求會出現等待MDL鎖
- 4.alter完成之前,其他DDL也會被阻塞一小段時間?
https://www.cnblogs.com/zengkefu/p/5674945.html
ddl實現細節
Prepare階段
- 1.創建frm文件
- 2.持有exclusive-mdl鎖,禁止讀寫,速度非常快
- 3.根據alter類型,確定執行方式,檢查存儲引擎是否支持inplace,不支持則使用copy---copy,online-rebuild:重新組織表,online-norebuild:改數據字典即可
- 4.更新數據字典的內存對象
- 5.分配row_log對象記錄增量,記錄DDL期間數據修改的日誌,如果日誌量超過參數
innodb_online_alter_log_max_size
的上限,則DDL失敗 - 6.生成臨時ibd文件(rebuild類型)
ddl執行階段
- 1.降級exclusvie-mdl鎖,允許讀寫
- 2.掃描原表的聚簇索引每條記錄
- 3.遍歷新表的聚簇索引和二級索引,逐一處理
- 4.根據記錄構造對應的索引項
- 5.將構造索引項插入sort_buffer塊進行排序
- 6.將sort_buffer塊中的排序結果插入新的索引
- 7.處理ddl執行過程中產生的增量-row-log,rebuild類型需要
commit階段
- 1.升級到exclusive-mdl索引,禁止讀寫
- 2.應用最後row_log中產生的日誌
- 3.更新innodb的數據字典
- 4.提交事務-刷數據的redo日誌
- 5.修改統計信息
- 6.rename臨時ibd文件,frm文件
- 7.變更完成
http://blog.itpub.net/22664653/viewspace-2056953
DDL SQL組合使用
1.全部變更寫到一條sql裏面
alter table t1 add index idx1(c1),add unique index idx2(c2),change c4_old_name c4_new_name integer unsigned;
copy方式下,效率最高,只需要拷貝1次
2.每個變更一條sql
alter table t1 add index idx1(c1);
alter table t1 add unique index idx2(c2);
alter table t1 change c4_old_name c4_new_name integer unsigned not null;
測試
1.顯式開啟一個查詢事務,默認DDL並不好?理論與實際不一樣
- session1
mysql session 1 >use db1;
Database changed
mysql session 1 >begin;
Query OK, 0 rows affected (0.00 sec)
mysql session 1 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (0.00 sec)
mysql session 1 >
開始一個事務,執行簡單查詢
- session2
mysql session 2 >use db1
Database changed
mysql session 2 >alter table t1 add index idx_id(id);
Query OK, 0 rows affected (9 min 47.09 sec)
Records: 0 Duplicates: 0 Warnings: 0
被阻塞9分47.09秒
- session3
mysql session 3 >use db1;
Database changed
mysql session 3 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (9 min 37.60 sec)
被阻塞9分37.60秒
- session4
mysql session 4 >mysql session 4 >show processlist;
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 432908 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 432806 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 67 | root | localhost | db1 | Query | 23 | Waiting for table metadata lock | alter table t1 add index idx_id(id) |
| 68 | root | localhost | db1 | Query | 14 | Waiting for table metadata lock | select * from t1 |
| 69 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 70 | root | localhost | db1 | Sleep | 36 | | NULL |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-------------------------------------+
7 rows in set (0.00 sec)
session2的alter和session3的select被阻塞
都在等待MDL鎖 alter
2.第二個加了算法,時間更長,dml並發更好?理論與實際不一樣
- session1
mysql session 1 >begin;
Query OK, 0 rows affected (0.00 sec)
mysql session 1 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (0.00 sec)
開啟一個事務
- session2
mysql session 2 >use db1
Database changed
mysql session 2 >alter table t1 add index idx_id(id), algorithm=inplace,lock=none;
Query OK, 0 rows affected (9 min 34.48 sec)
Records: 0 Duplicates: 0 Warnings: 0
顯示指定inplace方式增加索引
- session3
mysql session 3 >use db1;
Database changed
mysql session 3 >select * from t1;
+------+------+
| id | name |
+------+------+
| 1 | yzw |
+------+------+
1 row in set (9 min 22.13 sec)
查詢被阻塞
- session4
mysql session 4 >show processlist;
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 500684 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 500582 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 71 | root | localhost | db1 | Sleep | 45 | | NULL |
| 72 | root | localhost | db1 | Query | 20 | Waiting for table metadata lock | alter table t1 add index idx_id(id), algorithm=inplace,lock=none |
| 73 | root | localhost | db1 | Query | 7 | Waiting for table metadata lock | select * from t1 |
| 74 | root | localhost | NULL | Query | 0 | starting | show processlist |
+----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------+
7 rows in set (0.00 sec)
等待MDL鎖
3.原有copy方式DDL
1.構造測試數據
sysbench --test=insert.lua --oltp-table-size=5000000 --mysql-table-engine=innodb --mysql-user=root --mysql-password=123456 --mysql-port=3306 --mysql-host=localhost --mysql-db=test --max-requests=0 --time=60 --oltp-tables-count=2 --report-interval=10 --threads=2 prepare
2.使用原有copy方式進行ddl
mysql session 1 >select version();
+------------+
| version() |
+------------+
| 5.7.16-log |
+------------+
1 row in set (0.00 sec)
mysql session 1 >show create table sbtest1
-> ;
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sbtest1 | CREATE TABLE `sbtest1` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT ‘0‘,
`c` char(120) NOT NULL DEFAULT ‘‘,
`pad` char(60) NOT NULL DEFAULT ‘‘,
PRIMARY KEY (`id`),
KEY `k_1` (`k`)
) ENGINE=InnoDB AUTO_INCREMENT=5000001 DEFAULT CHARSET=utf8 MAX_ROWS=1000000 |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql session 1 >show variables like ‘%old_alter_table%‘;
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.01 sec)
mysql session 1 >set global old_alter_table=on;
Query OK, 0 rows affected (0.00 sec)
mysql session 1 >show variables like ‘old_alter_table‘;
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.00 sec)
mysql session 1 >set old_alter_table=1;
Query OK, 0 rows affected (0.00 sec)
mysql session 1 >show variables like ‘old_alter_table‘;
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | ON |
+-----------------+-------+
1 row in set (0.00 sec)
mysql session 1 >show index from sbtest1;
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| sbtest1 | 0 | PRIMARY | 1 | id | A | 4804854 | NULL | NULL | | BTREE | | |
| sbtest1 | 1 | k_1 | 1 | k | A | 883154 | NULL | NULL | | BTREE | | |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
2 rows in set (0.00 sec)
2.刪除增加索引
會觸發copy操作,並在數據目錄產生臨時frm和ibd文件
其他DML會被阻塞,請求MDL鎖
- session1
mysql session 1 >alter table sbtest1 drop index k_1;
Query OK, 5000000 rows affected (55.42 sec)
Records: 5000000 Duplicates: 0 Warnings: 0
mysql session 1 >alter table sbtest1 add index k_1(k);
Query OK, 5000000 rows affected (1 min 20.26 sec)
Records: 5000000 Duplicates: 0 Warnings: 0
- session2
mysql session 2 >update sbtest1 set k=2496416 where id=1;
Query OK, 1 row affected (14.39 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql session 2 >update sbtest1 set k=2496417 where id=1;
Query OK, 1 row affected (1 min 12.48 sec)
Rows matched: 1 Changed: 1 Warnings: 0
- session3
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 507514 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 507412 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 716 | root | localhost | test | Query | 46 | copy to tmp table | alter table sbtest1 drop index k_1 |
| 834 | root | localhost | test | Query | 5 | Waiting for table metadata lock | update sbtest1 set k=2496416 where id=1 |
| 835 | root | localhost | NULL | Sleep | 35 | | NULL |
| 836 | root | localhost | db1 | Query | 0 | starting | show processlist |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
mysql session 3 >mysql session 3 >show processlist;
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 507617 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 507515 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 716 | root | localhost | test | Query | 27 | copy to tmp table | alter table sbtest1 add index k_1(k) |
| 834 | root | localhost | test | Query | 20 | Waiting for table metadata lock | update sbtest1 set k=2496417 where id=1 |
| 835 | root | localhost | NULL | Sleep | 138 | | NULL |
| 836 | root | localhost | db1 | Query | 0 | starting | show processlist |
+-----+------+--------------------+------+------------------+--------+---------------------------------------------------------------+-----------------------------------------+
7 rows in set (0.00 sec)
- session4
[root@master test]# pwd
/mysqlData/data/test
[root@master test]# ls -ltr
total 2564148
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:25 sbtest2.frm
-rw-r-----. 1 mysql mysql 1224736768 Mar 13 11:26 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:33 sbtest1.frm
-rw-r-----. 1 mysql mysql 1149239296 Mar 13 11:34 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:35 #sql-663e_2cc.frm
-rw-r-----. 1 mysql mysql 251658240 Mar 13 11:35 #sql-663e_2cc.ibd
4.新特性online DDL
1.改成online DDL方式
並不影響DML
- session1
mysql session 1 >set old_alter_table=0;
Query OK, 0 rows affected (0.00 sec)
mysql session 1 >show variables like ‘%old_alter_table%‘;
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| old_alter_table | OFF |
+-----------------+-------+
1 row in set (0.01 sec)
mysql session 1 >alter table sbtest1 drop index k_1;
Query OK, 0 rows affected (0.07 sec)
Records: 0 Duplicates: 0 Warnings: 0
# 索引瞬間被刪掉了
mysql session 1 >alter table sbtest1 add index k_1(k);
Query OK, 0 rows affected (9.45 sec)
Records: 0 Duplicates: 0 Warnings: 0
- session2
mysql session 2 >update sbtest1 set k=2496417 where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
# 添加二級索引的時候,也不會阻塞DML
- session3
mysql session 3 >show processlist;
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 508423 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 508321 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 716 | root | localhost | test | Query | 1 | altering table | alter table sbtest1 add index k_1(k) |
| 1328 | root | localhost | test | Sleep | 0 | | NULL |
| 1329 | root | localhost | db1 | Query | 0 | starting | show processlist |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------+
6 rows in set (0.00 sec)
5.大事務之前的online DDL
alter之前有大事務執行,ddl會被阻塞,且後面的所有其他所有請求都會被阻塞
操作DDL之前最好還是先檢查一下是否有大事務在進行中
- session1
mysql session 1 >select * from sbtest1 where c=‘long select before alter‘;
Empty set (2.49 sec)
- session2
mysql session 2 >alter table sbtest1 add index k_1(k);
Query OK, 0 rows affected (11.46 sec)
Records: 0 Duplicates: 0 Warnings: 0
- session3
mysql session 3 >select * from sbtest1 where c=‘long select after alter execution but not complete‘;
Empty set (3.69 sec)
- session4
mysql session 4 >show processlist;
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 508885 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 508783 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 1328 | root | localhost | test | Query | 1 | Waiting for table metadata lock | alter table sbtest1 add index k_1(k) |
| 1329 | root | localhost | test | Query | 1 | Waiting for table metadata lock | select * from sbtest1 where c=‘long select after alter execution but not complete‘ |
| 1480 | root | localhost | test | Query | 2 | Sending data | select * from sbtest1 where c=‘long select before alter‘ |
| 1619 | root | localhost | NULL | Query | 0 | starting | show processlist |
+------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)
6.添加刪除列,不加inplace算法
允許ddl期間進行並發DML操作,並產生臨時文件
- session1,添加刪除列
mysql session 1 >ALTER TABLE `sbtest2` -> ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0, -> ADD COLUMN `f_new_col2` varchar(32) NULL DEFAULT ‘‘ AFTER `f_new_col1`;
Query OK, 0 rows affected (51.50 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql session 1 >alter table sbtest2 drop column f_new_col1, -> drop column f_new_col2;
Query OK, 0 rows affected (48.64 sec)
Records: 0 Duplicates: 0 Warnings: 0
- session2,進行update
mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘;
Query OK, 0 rows affected (25.67 sec)
Rows matched: 0 Changed: 0 Warnings: 0
mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘;
Query OK, 0 rows affected (6.15 sec)
Rows matched: 0 Changed: 0 Warnings: 0
- session3,查詢
mysql session 3 >select * from sbtest2 where c=‘select when add colomun ddl start‘;
Empty set (23.74 sec)
mysql session 3 >select * from sbtest2 where c=‘select when add colomun ddl start‘;
Empty set (2.24 sec)
- session4
mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 523165 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 523063 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 0 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 9 | altering table | ALTER TABLE `sbtest2`
ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0,
ADD COLUMN `f_new_col2` varc |
| 11110 | root | localhost | test | Query | 6 | updating | update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘ |
| 11113 | root | localhost | test | Query | 5 | Sending data | select * from sbtest2 where c=‘select when add colomun ddl start‘ |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)
mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 523338 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 523236 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 22 | altering table | alter table sbtest2 drop column f_new_col1,
drop column f_new_col2 |
| 11110 | root | localhost | test | Query | 6 | updating | update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘ |
| 11113 | root | localhost | test | Query | 2 | Sending data | select * from sbtest2 where c=‘select when add colomun ddl start‘ |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------+
7 rows in set (0.00 sec)
- session5,產生臨時文件
[root@master test]# ls -ltr
total 4010036
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8632 Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1275068416 Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8716 Mar 13 15:54 sbtest2.frm
-rw-r-----. 1 mysql mysql 1442840576 Mar 13 15:55 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8632 Mar 13 15:57 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 1388314624 Mar 13 15:58 #sql-ib87-1396920660.ibd
- 刪除的時候產生的臨時文件比ibd更大
[root@master test]# ls -ltrh
total 3.9G
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8.5K Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1.2G Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 8.5K Mar 13 16:10 sbtest2.frm
-rw-r-----. 1 mysql mysql 1.3G Mar 13 16:11 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:12 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 1.4G Mar 13 16:13 #sql-ib90-1396920666.ibd
7.修改列,加inplace算法(5.7)
- 1).創建
f_new_col2
的時候長度是32
mysql session 1 >ALTER TABLE `sbtest2` -> ADD COLUMN `f_new_col1` int(11) NULL DEFAULT 0, -> ADD COLUMN `f_new_col2` varchar(32) NULL DEFAULT ‘‘ AFTER `f_new_col1`;
Query OK, 0 rows affected (1 min 4.21 sec)
Records: 0 Duplicates: 0 Warnings: 0
- 2).更改
f_new_col2
長度為50
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(50) NULL DEFAULT ‘‘, algorithm=inplace ;
Query OK, 0 rows affected (0.09 sec)
Records: 0 Duplicates: 0 Warnings: 0
- 3).可以發現長度已經被更改
mysql session 1 >show create table sbtest2;
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sbtest2 | CREATE TABLE `sbtest2` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT ‘0‘,
`c` char(120) NOT NULL DEFAULT ‘‘,
`pad` char(60) NOT NULL DEFAULT ‘‘,
`f_new_col1` int(11) DEFAULT ‘0‘,
`f_new_col2` varchar(50) DEFAULT ‘‘,
PRIMARY KEY (`id`),
KEY `k_2` (`k`)
) ENGINE=InnoDB AUTO_INCREMENT=5000001 DEFAULT CHARSET=utf8 MAX_ROWS=1000000 |
+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
- 4).再更改更長長度時不支持
inplace
,但是改成長度是50又支持,50改短或長都不支持?
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(150) NULL DEFAULT ‘‘, algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(32) NULL DEFAULT ‘‘, algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(50) NULL DEFAULT ‘‘, algorithm=inplace ;
Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(32) NULL DEFAULT ‘‘, algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(88) NULL DEFAULT ‘‘, algorithm=inplace ;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
修改列類型應該要rebuild整張表
8.繼續上面的修改列長度(5.7)
ddl堵塞DML,但是不阻塞select
- 1).session1
mysql session 1 >ALTER TABLE sbtest2 -> CHANGE f_new_col2 f_new_col2 varchar(200) NULL DEFAULT ‘‘;
Query OK, 5000000 rows affected (1 min 28.96 sec)
Records: 5000000 Duplicates: 0 Warnings: 0
- 2).session2
mysql session 2 >update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘;
Query OK, 0 rows affected (1 min 34.01 sec)
Rows matched: 0 Changed: 0 Warnings: 0
- 3).session3
mysql session 3 >select * from sbtest2 where c=‘select when add colomun ddl start‘;
Empty set (2.82 sec)
- 4).session4
mysql session 4 >show processlist;
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
| 47 | repl | 172.16.3.154:42173 | NULL | Binlog Dump GTID | 524786 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 48 | repl | 172.16.3.153:57245 | NULL | Binlog Dump GTID | 524684 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 53 | mha | 172.16.3.152:43681 | NULL | Sleep | 1 | | NULL |
| 11000 | root | localhost | NULL | Query | 0 | starting | show processlist |
| 11021 | root | localhost | test | Query | 3 | copy to tmp table | ALTER TABLE sbtest2
CHANGE f_new_col2 f_new_col2 varchar(200) NULL DEFAULT ‘‘ |
| 12186 | root | localhost | test | Query | 2 | Waiting for table metadata lock | update sbtest2 set c="update when add colomun ddl start" where c=‘33333‘ |
| 12189 | root | localhost | test | Query | 1 | Sending data | select * from sbtest2 where c=‘select when add colomun ddl start‘ |
+-------+------+--------------------+------+------------------+--------+---------------------------------------------------------------+--------------------------------------------------------------------------------+
7 rows in set (0.01 sec)
- 5).session5,copy肯定是產生臨時文件
[root@master test]# ls -ltrh
total 3.2G
-rw-r-----. 1 mysql mysql 61 Mar 13 10:45 db.opt
-rw-r-----. 1 mysql mysql 8.5K Mar 13 11:56 sbtest1.frm
-rw-r-----. 1 mysql mysql 1.2G Mar 13 11:57 sbtest1.ibd
-rw-r-----. 1 mysql mysql 1.4G Mar 13 16:13 sbtest2.ibd
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:15 sbtest2.frm
-rw-r-----. 1 mysql mysql 8.6K Mar 13 16:21 #sql-663e_2b0d.frm
-rw-r-----. 1 mysql mysql 648M Mar 13 16:22 #sql-663e_2b0d.ibd
https://www.cnblogs.com/zengkefu/p/5674945.html
https://dev.mysql.com/doc/refman/5.6/en/upgrading-from-previous-series.html
OAK DDL
5.6之前使用第三方工具進行ddl
1.oak_online_alter_table前提條件
- 1.單列唯一索引,聯合索引和聯合主鍵索引不可以,觸發mysqlbug
- 2.沒有外鍵foreign key
- 3.沒有定義觸發器,有的話也一棍打死
2.oak ddl流程
- 1.確認表符合oak_online_alter_table的執行條件
- 2.如果有觸發器,備份觸發器,刪除觸發器
- 3.預估oak命令執行時間,表有1億,1秒執行3000條,大概執行12個小時,計劃時間執行
- 4.進行新舊表數據校驗,如果DDL改變了表字段類型,可能導致表數據變化
- 5.表切換rename
- 6.刪除原表觸發器
- 7.刪除表
3.check list功能
./run_bat IP onlineddl_check port dbname tabname
4.檢查外鍵和觸發器SQL
- 1.information_schema.key_column_usage
- 2.
select * from information_schema.key_column_usage where table_schema=dbname and table_name=tabname and referenced_table_name is not null
- 3.
select * from information_schema.key_column_usage where referenced_table_schema=dbname and referenced_table_name=tabname;
5.安裝openark-kit工具包
wget -O /etc/yum.repos.d/oak.repo http://download.opensuse.org/repositories/home:LenzGr/CentOS_7/home:LenzGr.repo
install openark-kit
6.操作過程
- 1).構造測試數據
sysbench --test=insert.lua --oltp-table-size=500000 --mysql-table-engine=innodb --mysql-user=root --mysql-password=123456 --mysql-port=3306 --mysql-host=localhost --mysql-db=test --max-requests=0 --time=60 --oltp-tables-count=2 --report-interval=10 --threads=2 prepare
- 2).檢查是否有觸發器
select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema=‘test‘;
- 3).執行online DDL
oak-online-alter-table -u root --ask-pass -S /usr/local/mysql/mysql.sock -d test -t sbtest1 -g new_sbtest1 -a "add last_update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,add key last_update_time(last_update_time)" --sleep=300 --skip-delete-pass
執行ddl的時候,依然可以進行ddl操作
root@master 22:34: [test]> insert into sbtest1 values(10000000,10000,‘AAAAA‘,‘BBBBB‘);
Query OK, 1 row affected (0.01 sec)
- 4).執行校驗
select sum(crc32(concat(ifnull(id,‘NULL‘),ifnull(k,‘NULL‘)))) as sum from sbtest1 union all \
select sum(crc32(concat(ifnull(id,‘NULL‘),ifnull(k,‘NULL‘)))) as sum from new_sbtest1;
# select count(*) from sbtest1 union all select count(*) from new_sbtest1;
root@master 22:34: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500001 |
| 291001 |
+----------+
2 rows in set (0.19 sec)
#
desc sbtest1;
desc new_sbtest1;
- 5).因為在原表增加了觸發器,即使ddl完成之後,新的DML依然會插入到新表
root@master 22:34: [test]> select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema=‘test‘;
+----------------+----------------+---------------------+--------------------+
| trigger_schema | trigger_name | event_object_schema | event_object_table |
+----------------+----------------+---------------------+--------------------+
| test | sbtest1_AI_oak | test | sbtest1 |
| test | sbtest1_AU_oak | test | sbtest1 |
| test | sbtest1_AD_oak | test | sbtest1 |
+----------------+----------------+---------------------+--------------------+
3 rows in set (0.00 sec)
root@master 22:38: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500001 |
| 500001 |
+----------+
2 rows in set (0.16 sec)
root@master 22:38: [test]> insert into sbtest1 values(10000001,10000,‘AAAAA‘,‘BBBBB‘);
Query OK, 1 row affected (0.00 sec)
root@master 22:39: [test]> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 500002 |
| 500002 |
+----------+
2 rows in set (0.16 sec)
- 6).表切換,更改數據字典
use test;
set names utf8;
rename table sbtest1 to old_sbtest1,new_sbtest1 to sbtest1;
- 7).刪除觸發器和原表
drop trigger sbtest1_AI_oak;
drop trigger sbtest1_AU_oak;
drop trigger sbtest1_AD_oak;
drop table old_sbtest1;
root@master 22:40: [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| old_sbtest1 |
| sbtest1 |
| sbtest2 |
+----------------+
3 rows in set (0.00 sec)
root@master 22:41: [test]> select trigger_schema,trigger_name,event_object_schema,event_object_table from information_schema.triggers where event_object_schema=‘test‘;
+----------------+----------------+---------------------+--------------------+
| trigger_schema | trigger_name | event_object_schema | event_object_table |
+----------------+----------------+---------------------+--------------------+
| test | sbtest1_AI_oak | test | old_sbtest1 |
| test | sbtest1_AU_oak | test | old_sbtest1 |
| test | sbtest1_AD_oak | test | old_sbtest1 |
+----------------+----------------+---------------------+--------------------+
3 rows in set (0.01 sec)
root@master 22:41: [test]> drop trigger sbtest1_AI_oak;
Query OK, 0 rows affected (0.00 sec)
root@master 22:41: [test]> drop trigger sbtest1_AU_oak;
Query OK, 0 rows affected (0.01 sec)
root@master 22:41: [test]> drop trigger sbtest1_AD_oak;
Query OK, 0 rows affected (0.01 sec)
root@master 22:41: [test]> drop table old_sbtest1;
Query OK, 0 rows affected (0.23 sec)
root@master 22:41: [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| sbtest1 |
| sbtest2 |
+----------------+
2 rows in set (0.00 sec)
7.用oak做Online DDL,如果rename那一步操作錯了?在從庫上rename了怎麽恢復?(包含舊表刪除和trigger刪除).
- 1)第一種情況加索引的Online DDL
- 2)第二種情況加字段的Online DDL
發現rename錯表了?
再做一次drop ddl
不是從主庫進行ddl,選擇從庫ddl,從庫rename錯了,想恢復?
http://www.cnblogs.com/xuanzhi201111/p/4549250.html
http://www.cnblogs.com/xinysu/p/6732646.html#_lab2_4_3
mysql小白系列_13 Online DDL