Mysql資料按天分割槽,定期刪除
阿新 • • 發佈:2018-12-19
需求:
1.日誌表需要按天分割槽
2.只保留一個月資料
方案:
1.建立兩個事件,一個事件生成未來需要的分割槽,另一個事件定期檢查過期資料(移除分割槽)
2.建立事件每小時執行一次,刪除事件每天執行一次
3.事件開始時需要先建立一個當前所需分割槽
全量方法:
1.先構造儲存過程 create_partition_today :將錶轉化為分割槽表,並將歷史資料歸集到該分割槽,未來資料則按天放置:
#alter table to partition table DELIMITER $$ USE `dc_log`$$ DROP PROCEDUREIF EXISTS `create_partition_today`$$ CREATE PROCEDURE `create_partition_today`(IN_SCHEMANAME VARCHAR(64), IN_TABLENAME VARCHAR(64)) BEGIN DECLARE BEGINTIME TIMESTAMP; DECLARE ENDTIME TIMESTAMP; DECLARE DAYS_ENDTIME INT; DECLARE PARTITIONNAME VARCHAR(16); SET BEGINTIME = NOW();SET ENDTIME = BEGINTIME + INTERVAL 1 DAY; SET PARTITIONNAME = DATE_FORMAT(BEGINTIME, 'p%Y%m%d'); SET DAYS_ENDTIME = TO_DAYS(ENDTIME); SET @SQL = CONCAT('ALTER TABLE `', IN_SCHEMANAME, '`.`', IN_TABLENAME, '`', ' PARTITION BY RANGE (to_days(create_time)) (PARTITION', PARTITIONNAME, ' VALUES LESS THAN (', DAYS_ENDTIME, '))'); PREPARE STMT FROM @SQL; EXECUTE STMT; DEALLOCATE PREPARE STMT; END$$ DELIMITER ;
2.按天構造分割槽的儲存過程create_partition_by_day:
#procedure of build partition of today and next day DELIMITER $$ USE `dc_log`$$ DROP PROCEDURE IF EXISTS `create_partition_by_day`$$ CREATE PROCEDURE `create_partition_by_day`(IN_SCHEMANAME VARCHAR(64), IN_TABLENAME VARCHAR(64)) BEGIN DECLARE ROWS_CNT INT UNSIGNED; DECLARE BEGINTIME TIMESTAMP; DECLARE ENDTIME TIMESTAMP; DECLARE DAYS_ENDTIME INT; DECLARE PARTITIONNAME VARCHAR(16); SET BEGINTIME = NOW() + INTERVAL 1 DAY; SET PARTITIONNAME = DATE_FORMAT(BEGINTIME, 'p%Y%m%d'); SET ENDTIME = BEGINTIME + INTERVAL 1 DAY; SET DAYS_ENDTIME = TO_DAYS(ENDTIME); SELECT COUNT(*) INTO ROWS_CNT FROM information_schema.partitions WHERE table_schema = IN_SCHEMANAME AND table_name = IN_TABLENAME AND partition_name = PARTITIONNAME; IF ROWS_CNT = 0 THEN SET @SQL = CONCAT('ALTER TABLE `', IN_SCHEMANAME, '`.`', IN_TABLENAME, '`', ' ADD PARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', DAYS_ENDTIME, '))'); PREPARE STMT FROM @SQL; EXECUTE STMT; DEALLOCATE PREPARE STMT; ELSE SELECT CONCAT("partition `", PARTITIONNAME, "` for table `", IN_SCHEMANAME, ".", IN_TABLENAME, "` already exists") AS result; END IF; END$$ DELIMITER ;
3.按天清除資料的儲存過程clear_partition_by_day
DELIMITER $$ USE `dc_log`$$ DROP PROCEDURE IF EXISTS `clear_partition_by_day`$$ CREATE PROCEDURE `clear_partition_by_day`(IN_SCHEMANAME VARCHAR(64), IN_TABLENAME VARCHAR(64)) BEGIN DECLARE NOWDAYS INT; DECLARE Done INT; DECLARE part VARCHAR(64); DECLARE descr INT; DECLARE rs CURSOR FOR SELECT partition_name part, partition_description descr FROM information_schema.partitions WHERE table_schema = IN_SCHEMANAME AND table_name = IN_TABLENAME; /* 異常處理 */ DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET Done = 1; OPEN rs; SET NOWDAYS = TO_DAYS(NOW()); FETCH rs into part,descr; while Done is null DO IF NOWDAYS - descr > 30 THEN select descr AS cc; SET @SQL = CONCAT('ALTER TABLE `', IN_SCHEMANAME, '`.`', IN_TABLENAME, '`', ' DROP PARTITION ', part); select descr AS aa; SELECT @SQL AS result; select descr AS bb; PREPARE STMT FROM @SQL; EXECUTE STMT; DEALLOCATE PREPARE STMT; END IF; FETCH rs into part,descr; end WHILE; CLOSE rs; END$$ DELIMITER ;
注意:以上過程有可能報錯,這裡記錄兩個錯誤:
1.分割槽欄位必須包含主鍵:
這裡採用的是事件欄位作為分割槽欄位,當然不可能是主鍵,所以再mysql中會報錯,查閱資料大多說是因為分割槽表中不能保證資料唯一,需要將分割槽健納入主鍵才可,不清楚原理.如果不願意這樣做也可以考慮,刪除已有主鍵,無主鍵的表可以用任意欄位作為分割槽欄位
ALTER TABLE dc_system.service_push DROP PRIMARY KEY;
2.時間欄位不允許作為分割槽欄位:
這是因為 DATETIME 會受到時間區的影響,mysql裡面分割槽的時間欄位需要用CURRENT_TIMESTAMP
ALTER TABLE business_log MODIFY create_time DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP;
4.繼續往下,開啟事件變數,這裡直接修改
SET GLOBAL event_scheduler = 'ON'; #開啟
重啟後會被重置,如果需要永久修改,在my.cnf(mysql配置檔案)中[mysqld]部分中新增下面內容,重啟MYSQL
event_scheduler=ON
5.呼叫儲存過程建立當天分割槽:
#created today's partition CALL create_partition_today('dc_log', 'business_log');
6.建立事件:
DELIMITER $$ USE `dc_log`$$ CREATE EVENT IF NOT EXISTS `e_part_manage` ON SCHEDULE EVERY 1 HOUR #every minute STARTS '2018-05-01 18:27:00' ON COMPLETION PRESERVE ENABLE COMMENT 'Creating partitions' DO BEGIN CALL dc_log.create_partition_by_day('dc_log', 'business_log'); END$$ DELIMITER ; #event of clear data which out of date of 30 days DELIMITER $$ USE `dc_log`$$ CREATE EVENT IF NOT EXISTS `clear_data` ON SCHEDULE EVERY 1 DAY #every minute STARTS '2018-05-01 15:58:51' ON COMPLETION PRESERVE ENABLE COMMENT 'clearing data' DO BEGIN CALL clear_partition_by_day('dc_log', 'business_log'); END$$ DELIMITER ;
7.如果有其他表也需要如此處理,則先執行建立當天分割槽,再修改事件:
#service_push #same steps of table business_log ALTER TABLE dc_system.service_push DROP PRIMARY KEY; CALL create_partition_today('dc_system', 'service_push'); DELIMITER ; DELIMITER $$ ALTER EVENT e_part_manage DO BEGIN CALL dc_log.create_partition_by_day('dc_log', 'business_log'); CALL dc_log.create_partition_by_day('dc_system', 'service_push'); END$$ DELIMITER ; DELIMITER $$ ALTER EVENT clear_data DO BEGIN CALL dc_log.clear_partition_by_day('dc_log', 'business_log') ; CALL dc_log.clear_partition_by_day('dc_system', 'service_push'); END$$ DELIMITER ;
8.查看錶的分割槽情況:
use dc_log; SELECT partition_name part, partition_expression expr, partition_description descr, table_rows FROM INFORMATION_SCHEMA.partitions WHERE TABLE_SCHEMA = SCHEMA() AND TABLE_NAME='service_push' ;
9.手動新增分割槽:
ALTER TABLE dc_log.all_log ADD PARTITION (PARTITION 'PARTITIONNAME' VALUES LESS THAN (TO_DAYS(now())));
10.檢視事件和事件開關
show events; SHOW VARIABLES LIKE 'event_scheduler';
done