Zabbix Server MySQL分割槽分表
阿新 • • 發佈：2020-11-07
本教程只是大概講解zabbix MySQL分割槽分表操作，若讀者想了解更多關於表分割槽的詳細知識，可以訪問以下地址：
MySQL官網說明：https://dev.mysql.com/doc/refman/5.7/en/partitioning.html
Zabbix 官網說明：https://www.zabbix.org/wiki/Docs/howto/mysql_partition
Zabbix關於效能優化有很多種方法，例如調整效能引數（包括zabbix server效能引數，MySQL資料庫效能引數等），增加伺服器效能（CPU/記憶體/硬碟更換SSD固態盤/千兆網路換成萬兆光纖網路等），MySQL資料庫優化（包括讀寫分離/叢集/分割槽分表等），本教程主要針對MySQL資料庫的分割槽分表優化進行講解，MySQL分割槽分表不是適合所有MySQL資料庫使用。
本教程是對zabbix監控伺服器的MySQL進行分割槽分表進行優化，是參考zabbix官網說明操作，如果監控裝置達到1000臺以上或者監控資料量非常大就適合使用zabbix官網推薦的分割槽分表優化MySQL資料庫。MySQL分割槽分表作用減少資料庫表在容量、提高資料庫查詢效能。
zabbix 採用架構（Linux + Tomcat + MySQL + Zabbix ）,使用zabbix官網Yum安裝的環境，以下是已安裝軟體版本:
[root@zabbix ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root@zabbix ~]# uname -r 3.10.0-862.el7.x86_64 [root@zabbix ~]# php -v PHP 5.4.16 (cli) (built: Apr 1 2020 04:07:17) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies [root@zabbix ~]# apachectl -v Server version: Apache/2.4.6 (CentOS) Server built: Apr 2 2020 13:13:23 [root@zabbix ~]# zabbix_server -V zabbix_server (Zabbix) 4.0.25 [root@zabbix ~]# mysql -V mysql Ver 14.14 Distrib 5.7.32, for Linux (x86_64) using EditLine wrapper
分割槽分表預備知識：
1、建議瞭解一下MySQL分割槽分表概念，資料庫空間碎片概念，Innodb資料庫引擎。（不瞭解也可以繼續下面操作）
2、MySQL要開啟獨立表空間（innodb_file_per_table，MySQL5.6包含MySQL 5.6以下的版本預設是沒有開啟）。
3、MySQL需要開啟預設使用Innodb引擎。
4、針對zabbix資料庫表要進行分割槽分表，如果表資料量大執行分割槽分表會非常慢，建議先清空history、trends表中的資料（注意：歷史記錄會全部被清空）,再執行分割槽分表。
5、zabbix 資料庫主要針對以下表（zabbix歷史記錄表和zabbix儲存趨勢表）進行分割槽分表：history，history_uint，history_str，history_test，history_log，trends，trends_uint
6、zabbix 資料庫在沒有使用分割槽分表功能，預設使用Housekeeping（管家功能）進行刪除歷史資料和趨勢歷史記錄，如果zabbix資料庫使用了分割槽分表功能需要把Housekeeping（管理功能）關閉。Housekeeping功能監控資料量少可以使用，但監控資料量多每次執行刪除會降低MySQL資料庫效能，並且還會產生很多空間碎片。（zabbix_server.conf配置檔案兩個引數進行歷史記錄資料刪除：間隔多久刪除一次，預設單位小時HousekeepingFrequency=1，一次刪除多少資料，預設單位行MaxHousekeeperDelete=5000）
一、檢視開啟獨立表空間和、預設使用Innodb引擎、最大連線數
[root@zabbix ~]# mysql -u root -p
Enter password:
mysql> show variables like '%innodb_file_per_table%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| innodb_file_per_table | ON |
+-----------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like '%storage_engine%';
+----------------------------------+--------+
| Variable_name | Value |
+----------------------------------+--------+
| default_storage_engine | InnoDB |
| default_tmp_storage_engine | InnoDB |
| disabled_storage_engines | |
| internal_tmp_disk_storage_engine | InnoDB |
+----------------------------------+--------+
4 rows in set (0.00 sec)
mysql> show variables like '%max_connections%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.00 sec)
[root@zabbix ~]# echo "max_connections=1000" >> /etc/my.cnf
二、清空zabbix 資料庫歷史資料
mysql> select table_name,(data_length+index_length)/1024/1024 as total_mb,table_rows from information_schema.tables where table_schema='zabbix';
+----------------------------+------------+------------+
| history | 0.85937500 | 9443 |
| history_log | 0.03125000 | 0 |
| history_str | 0.03125000 | 19 |
| history_text | 0.03125000 | 0 |
| history_uint | 0.37500000 | 4216 |
| trends | 0.01562500 | 244 |
| trends_uint | 0.01562500 | 137 |
mysql> use zabbix
mysql>truncate table history;
mysql>optimize table history;
mysql>truncate table history_str;
mysql>optimize table history_str;
mysql>truncate table history_log;
mysql>optimize table history_log;
mysql>truncate table history_text;
mysql>optimize table history_text;
mysql>truncate table history_uint;
mysql>optimize table history_uint;
mysql>truncate table trends;
mysql>optimize table trends_uint;
三、zabbix 資料庫匯入儲存過程
###先編輯一個SQL指令碼執行匯入四個儲存過程，此指令碼複製zabbix官網提供的SQL指令碼
[root@zabbix ~]# vim /tmp/history_stored.sql
DELIMITER $$
CREATE PROCEDURE `partition_create`(SCHEMANAME varchar(64), TABLENAME varchar(64), PARTITIONNAME varchar(64), CLOCK int)
BEGIN
/*
SCHEMANAME = The DB schema in which to make changes
TABLENAME = The table with partitions to potentially delete
PARTITIONNAME = The name of the partition to create
*/
/*
Verify that the partition does not already exist
*/
DECLARE RETROWS INT;
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_description >= CLOCK;
IF RETROWS = 0 THEN
/*
1. Print a message indicating that a partition was created.
2. Create the SQL to create the partition.
3. Execute the SQL from #2.
*/
SELECT CONCAT( "partition_create(", SCHEMANAME, ",", TABLENAME, ",", PARTITIONNAME, ",", CLOCK, ")" ) AS msg;
SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME, ' ADD PARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' );
PREPARE STMT FROM @sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_drop`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), DELETE_BELOW_PARTITION_DATE BIGINT)
BEGIN
/*
SCHEMANAME = The DB schema in which to make changes
TABLENAME = The table with partitions to potentially delete
DELETE_BELOW_PARTITION_DATE = Delete any partitions with names that are dates older than this one (yyyy-mm-dd)
*/
DECLARE done INT DEFAULT FALSE;
DECLARE drop_part_name VARCHAR(16);
/*
Get a list of all the partitions that are older than the date
in DELETE_BELOW_PARTITION_DATE. All partitions are prefixed with
a "p", so use SUBSTRING TO get rid of that character.
*/
DECLARE myCursor CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND CAST(SUBSTRING(partition_name FROM 2) AS UNSIGNED) < DELETE_BELOW_PARTITION_DATE;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
/*
Create the basics for when we need to drop the partition. Also, create
@drop_partitions to hold a comma-delimited list of all partitions that
should be deleted.
*/
SET @alter_header = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " DROP PARTITION ");
SET @drop_partitions = "";
/*
Start looping through all the partitions that are too old.
*/
OPEN myCursor;
read_loop: LOOP
FETCH myCursor INTO drop_part_name;
IF done THEN
LEAVE read_loop;
END IF;
SET @drop_partitions = IF(@drop_partitions = "", drop_part_name, CONCAT(@drop_partitions, ",", drop_part_name));
END LOOP;
IF @drop_partitions != "" THEN
/*
1. Build the SQL to drop all the necessary partitions.
2. Run the SQL to drop the partitions.
3. Print out the table partitions that were deleted.
*/
SET @full_sql = CONCAT(@alter_header, @drop_partitions, ";");
PREPARE STMT FROM @full_sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, @drop_partitions AS `partitions_deleted`;
ELSE
/*
No partitions are being deleted, so print out "N/A" (Not applicable) to indicate
that no changes were made.
*/
SELECT CONCAT(SCHEMANAME, ".", TABLENAME) AS `table`, "N/A" AS `partitions_deleted`;
END IF;
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32), KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT)
BEGIN
DECLARE OLDER_THAN_PARTITION_DATE VARCHAR(16);
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE OLD_PARTITION_NAME VARCHAR(16);
DECLARE LESS_THAN_TIMESTAMP INT;
DECLARE CUR_TIME INT;
CALL partition_verify(SCHEMA_NAME, TABLE_NAME, HOURLY_INTERVAL);
SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00'));
SET @__interval = 1;
create_loop: LOOP
IF @__interval > CREATE_NEXT_INTERVALS THEN
LEAVE create_loop;
END IF;
SET LESS_THAN_TIMESTAMP = CUR_TIME + (HOURLY_INTERVAL * @__interval * 3600);
SET PARTITION_NAME = FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL * (@__interval - 1) * 3600, 'p%Y%m%d%H00');
IF(PARTITION_NAME != OLD_PARTITION_NAME) THEN
CALL partition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP);
END IF;
SET @__interval=@__interval+1;
SET OLD_PARTITION_NAME = PARTITION_NAME;
END LOOP;
SET OLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVAL KEEP_DATA_DAYS DAY), '%Y%m%d0000');
CALL partition_drop(SCHEMA_NAME, TABLE_NAME, OLDER_THAN_PARTITION_DATE);
END$$
DELIMITER ;
DELIMITER $$
CREATE PROCEDURE `partition_verify`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64), HOURLYINTERVAL INT(11))
BEGIN
DECLARE PARTITION_NAME VARCHAR(16);
DECLARE RETROWS INT(11);
DECLARE FUTURE_TIMESTAMP TIMESTAMP;
/*
* Check if any partitions exist for the given SCHEMANAME.TABLENAME.
*/
SELECT COUNT(1) INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_name IS NULL;
/*
* If partitions do not exist, go ahead and partition the table
*/
IF RETROWS = 1 THEN
/*
* Take the current date at 00:00:00 and add HOURLYINTERVAL to it. This is the timestamp below which we will store values.
* We begin partitioning based on the beginning of a day. This is because we don't want to generate a random partition
* that won't necessarily fall in line with the desired partition naming (ie: if the hour interval is 24 hours, we could
* end up creating a partition now named "p201403270600" when all other partitions will be like "p201403280000").
*/
SET FUTURE_TIMESTAMP = TIMESTAMPADD(HOUR, HOURLYINTERVAL, CONCAT(CURDATE(), " ", '00:00:00'));
SET PARTITION_NAME = DATE_FORMAT(CURDATE(), 'p%Y%m%d%H00');
-- Create the partitioning query
SET @__PARTITION_SQL = CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, " PARTITION BY RANGE(`clock`)");
SET @__PARTITION_SQL = CONCAT(@__PARTITION_SQL, "(PARTITION ", PARTITION_NAME, " VALUES LESS THAN (", UNIX_TIMESTAMP(FUTURE_TIMESTAMP), "));");
-- Run the partitioning query
PREPARE STMT FROM @__PARTITION_SQL;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END$$
DELIMITER ;
###執行SQL指令碼匯入儲存過程
[root@zabbix ~]# mysql -u root -p
Enter password:
mysql> use zabbix
mysql> source /tmp/history_stored.sql;
Query OK, 0 rows affected (0.03 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
四、zabbix 資料庫使用儲存過程
###建立一個SQL指令碼使用儲存過程
[root@zabbix ~]# vim /tmp/history_stored_used.sql
DELIMITER $$
CREATE PROCEDURE `partition_maintenance_all`(SCHEMA_NAME VARCHAR(32))
BEGIN
CALL partition_maintenance(SCHEMA_NAME, 'history', 28, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_log', 28, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_str', 28, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_text', 28, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'history_uint', 28, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'trends', 730, 24, 14);
CALL partition_maintenance(SCHEMA_NAME, 'trends_uint', 730, 24, 14);
END$$
DELIMITER ;
[root@zabbix ~]# mysql -u root -p
Enter password:
mysql> use zabbix
mysql> source /tmp/history_stored_used.sql;
mysql> call partition_maintenance_all('zabbix');
說明：
CALL partition_maintenance('<zabbix_db_name>', '<table_name>', <days_to_keep_data>, <hourly_interval>, <num_future_intervals_to_create>)
zabbix_db_name：庫名
table_name：表名
days_to_keep_data：儲存多少天的資料
hourly_interval：每隔多久生成一個分割槽
num_future_intervals_to_create：本次一共生成多少個分割槽
例如：
CALL partition_maintenance(SCHEMA_NAME, 'history', 28, 24, 14);
這個例子表示history表最多儲存28天的資料，每隔24小時生成一個分割槽，這次一共生成14個分割槽
五、定時呼叫儲存過程
提示：因為儲存過程會老化，比如上面所說history表最多儲存28天的資料，每隔24小時生成一個分割槽，這次一共生成14個分割槽，當生成第15個分割槽時儲存過程就已經老化了，後面的監控資料就儲存不了，會存在丟失監控資料，所以，需要定時執行呼叫儲存過程。設定定時任務時間是通過每24小時會產生一個分割槽，這次一共生成14個分割槽，也就是說第15天就儲存過程就會老化，那我們就設定每個星期六呼叫儲存過程建立一次（只要在14天以內的時間都允許）。
###通過crontab定時任務呼叫儲存過程
[root@zabbix ~]# crontab –e
###everyweek staurday MySQL create partition at 5:00 AM
00 5 * * 6 /usr/bin/mysql -u zabbix -p'12345' zabbix -e "call partition_maintenance_all('zabbix');" > /dev/null 2>&1
六、關閉zabbix 管家功能（housekeeping）
七、檢查建立的分割槽分表情況
###檢查分割槽情況
[root@zabbix ~]# ls -l /var/lib/mysql/zabbix/ | grep -e history#P#
-rw-r----- 1 mysql mysql 196608 Oct 20 16:39 history#P#p202010200000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010210000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010220000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010230000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010240000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010250000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010260000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010270000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010280000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010290000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010300000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202010310000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202011010000.ibd
-rw-r----- 1 mysql mysql 114688 Oct 20 16:07 history#P#p202011020000.ibd
###檢查分表情況
[root@zabbix ~]# mysql -u root -p
Enter password:
mysql> use zabbix
mysql> show create table history\G;
*************************** 1. row ***************************
Table: history
Create Table: CREATE TABLE `history` (
`itemid` bigint(20) unsigned NOT NULL,
`clock` int(11) NOT NULL DEFAULT '0',
`value` double(16,4) NOT NULL DEFAULT '0.0000',
`ns` int(11) NOT NULL DEFAULT '0',
KEY `history_1` (`itemid`,`clock`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin
/*!50100 PARTITION BY RANGE (`clock`)
(PARTITION p202010200000 VALUES LESS THAN (1603209600) ENGINE = InnoDB,
PARTITION p202010210000 VALUES LESS THAN (1603296000) ENGINE = InnoDB,
PARTITION p202010220000 VALUES LESS THAN (1603382400) ENGINE = InnoDB,
PARTITION p202010230000 VALUES LESS THAN (1603468800) ENGINE = InnoDB,
PARTITION p202010240000 VALUES LESS THAN (1603555200) ENGINE = InnoDB,
PARTITION p202010250000 VALUES LESS THAN (1603641600) ENGINE = InnoDB,
PARTITION p202010260000 VALUES LESS THAN (1603728000) ENGINE = InnoDB,
PARTITION p202010270000 VALUES LESS THAN (1603814400) ENGINE = InnoDB,
PARTITION p202010280000 VALUES LESS THAN (1603900800) ENGINE = InnoDB,
PARTITION p202010290000 VALUES LESS THAN (1603987200) ENGINE = InnoDB,
PARTITION p202010300000 VALUES LESS THAN (1604073600) ENGINE = InnoDB,
PARTITION p202010310000 VALUES LESS THAN (1604160000) ENGINE = InnoDB,
PARTITION p202011010000 VALUES LESS THAN (1604246400) ENGINE = InnoDB,
PARTITION p202011020000 VALUES LESS THAN (1604332800) ENGINE = InnoDB) */
1 row in set (0.00 sec)
ERROR:
No query specified