1. 程式人生 > >將zabbix資料庫從innodb遷移到tokudb並分割槽

將zabbix資料庫從innodb遷移到tokudb並分割槽

下載percona server,並啟用tokudb引擎

tar zxvfPercona-Server-5.6.32-rel78.0-Linux.x86_64.ssl101.tar.gz -C /usr/local/

ln -s Percona-Server-5.6.32-rel78.0-Linux.x86_64.ssl101/ mysql

對資料庫進行配置,使之能夠正常啟動

### TOKUDB ###

plugin-load                     = ha_tokudb.so

tokudb_data_dir                 =/export/mysql/mysql3306/tokudb

tokudb-log-dir                  =/export/mysql/mysql3306/tokudb

tokudb_tmp_dir                  = /export/mysql/mysql3306/tmp

#tokudb_row_format              = tokudb_small

tokudb_cache_size               = 3G

tokudb_commit_sync              = 0

tokudb_directio                 = 1

tokudb_read_block_size          = 128K

tokudb_read_buf_size            = 128K

啟用tokudb引擎

echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag

echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled

echo never >/sys/kernel/mm/transparent_hugepage/enabled

echo never >/sys/kernel/mm/transparent_hugepage/defrag

cd /usr/local/mysql

./bin/ps_tokudb_admin --enable

對資料進行操作,操作前準備

set sql_log_bin=0

set wait_timeout=10000000000000;

select * from zabbix.history intooutfile ‘/tmp/history.txt’;

truncate table zabbix.history;

alter table zabbix.history_unitengine = tokudb;

#alter table zabbix.historypartition by …

此處用的是別人寫的儲存過程對錶進行分割槽,儲存過程見文章最後

CALL partition_maintenance('zabbix','history', 28, 24, 5);

load data infile '/tmp/history.txt’into table history;

說明:

1. 在表非常大的情況下,不建議用mysqldump方式匯出後再匯入,也不建議直接altertable進行修改,因為用這2種方式耗時都比loaddata這種方式慢,而且是幾倍的差距。

2. 本文記錄的只是自己操作的一個方式,可能不是最優的遷移方式。400G的表,將近用了小一天的時間。

3. 不建議直接對zabbix生產庫進行操作,可以先用xtrabackupex進行備份,備份時用流直接將資料傳到測試的機器,可以節省很多時間

innobackupex --defaults-file=/export/mysql/mysql4306/my4306.cnf --user=root--stream=xbstream /tmp/ | ssh [email protected] "xbstream -x -C /export/mysql"

4. 通過備份,還原,建議生產庫的從庫,記錄主庫的binlog位置,待從庫操作完畢後,開啟多執行緒同步去追主,等主從資料一致後可以將zabbix連線到從庫上,再對原生產庫進行操作。

5. 如果zabbix庫中要分割槽的表有id等主鍵,需要將主鍵刪除後才可建議分割槽表,因為分割槽表為了最大限度的利用索引,分割槽列必須包含主鍵。而分割槽又是以時間來分,故需將id等主鍵刪除或改成普通索引。

儲存過程如下:

DELIMITER $$

CREATE PROCEDURE`partition_create`(SCHEMANAME varchar(64), TABLENAME varchar(64), PARTITIONNAMEvarchar(64), CLOCK int)

BEGIN

        /*

           SCHEMANAME = The DB schema in whichto make changes

           TABLENAME = The table withpartitions to potentially delete

           PARTITIONNAME = The name of thepartition to create

        */

        /*

           Verify that the partition does notalready exist

        */

        DECLARE RETROWS INT;

        SELECT COUNT(1) INTO RETROWS

        FROM information_schema.partitions

        WHERE table_schema = SCHEMANAME ANDtable_name = TABLENAME AND partition_description >= CLOCK;

        IF RETROWS = 0 THEN

                /*

                   1. Print a messageindicating that a partition was created.

                   2. Create the SQL to createthe partition.

                   3. Execute the SQL from #2.

                */

                SELECT CONCAT("partition_create(", SCHEMANAME, ",", TABLENAME,",", PARTITIONNAME, ",", CLOCK, ")" ) AS msg;

               SET @sql = CONCAT('ALTER TABLE ', SCHEMANAME, '.', TABLENAME, ' ADD PARTITION (PARTITION ',PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' );

                PREPARE STMT FROM @sql;

                EXECUTE STMT;

                DEALLOCATE PREPARE STMT;

        END IF;

END$$

DELIMITER ;

DELIMITER $$

CREATE PROCEDURE`partition_drop`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64),DELETE_BELOW_PARTITION_DATE BIGINT)

BEGIN

        /*

           SCHEMANAME = The DB schema in whichto make changes

           TABLENAME = The table withpartitions to potentially delete

           DELETE_BELOW_PARTITION_DATE = Deleteany partitions with names that are dates older than this one (yyyy-mm-dd)

        */

        DECLARE done INT DEFAULT FALSE;

        DECLARE drop_part_name VARCHAR(16);

        /*

           Get a list of all the partitionsthat are older than the date

           in DELETE_BELOW_PARTITION_DATE.  All partitions are prefixed with

           a "p", so use SUBSTRING TOget rid of that character.

        */

        DECLARE myCursor CURSOR FOR

                SELECT partition_name

                FROMinformation_schema.partitions

                WHERE table_schema = SCHEMANAMEAND table_name = TABLENAME AND CAST(SUBSTRING(partition_name FROM 2) ASUNSIGNED) < DELETE_BELOW_PARTITION_DATE;

        DECLARE CONTINUE HANDLER FOR NOT FOUNDSET done = TRUE;

        /*

           Create the basics for when we needto drop the partition.  Also, create

           @drop_partitions to hold acomma-delimited list of all partitions that

           should be deleted.

        */

        SET @alter_header = CONCAT("ALTERTABLE ", SCHEMANAME, ".", TABLENAME, " DROP PARTITION");

        SET @drop_partitions = "";

        /*

           Start looping through all thepartitions that are too old.

        */

        OPEN myCursor;

        read_loop: LOOP

                FETCH myCursor INTOdrop_part_name;

                IF done THEN

                        LEAVE read_loop;

                END IF;

                SET @drop_partitions =IF(@drop_partitions = "", drop_part_name, CONCAT(@drop_partitions,",", drop_part_name));

        END LOOP;

        IF @drop_partitions != ""THEN

                /*

                   1. Build the SQL to drop allthe necessary partitions.

                   2. Run the SQL to drop thepartitions.

                   3. Print out the tablepartitions that were deleted.

                */

                SET @full_sql = CONCAT(@alter_header,@drop_partitions, ";");

                PREPARE STMT FROM @full_sql;

                EXECUTE STMT;

                DEALLOCATE PREPARE STMT;

                SELECT CONCAT(SCHEMANAME,".", TABLENAME) AS `table`, @drop_partitions AS `partitions_deleted`;

        ELSE

                /*

                   No partitions are beingdeleted, so print out "N/A" (Not applicable) to indicate

                   that no changes were made.

                */

                SELECT CONCAT(SCHEMANAME,".", TABLENAME) AS `table`, "N/A" AS `partitions_deleted`;

        END IF;

END$$

DELIMITER ;

DELIMITER $$

CREATE PROCEDURE`partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32),KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT)

BEGIN

        DECLARE OLDER_THAN_PARTITION_DATEVARCHAR(16);

        DECLARE PARTITION_NAME VARCHAR(16);

        DECLARE OLD_PARTITION_NAME VARCHAR(16);

        DECLARE LESS_THAN_TIMESTAMP INT;

        DECLARE CUR_TIME INT;

        CALL partition_verify(SCHEMA_NAME,TABLE_NAME, HOURLY_INTERVAL);

        SET CUR_TIME =UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00'));

        SET @__interval = 1;

        create_loop: LOOP

                IF @__interval >CREATE_NEXT_INTERVALS THEN

                        LEAVE create_loop;

                END IF;

                SET LESS_THAN_TIMESTAMP =CUR_TIME + (HOURLY_INTERVAL * @__interval * 3600);

                SET PARTITION_NAME =FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL * (@__interval - 1) * 3600,'p%Y%m%d%H00');

                IF(PARTITION_NAME !=OLD_PARTITION_NAME) THEN

                     CALLpartition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP);

              ENDIF;

                SET @[email protected]__interval+1;

                SET OLD_PARTITION_NAME = PARTITION_NAME;

        END LOOP;

        SETOLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVAL KEEP_DATA_DAYSDAY), '%Y%m%d0000');

        CALL partition_drop(SCHEMA_NAME,TABLE_NAME, OLDER_THAN_PARTITION_DATE);

END$$

DELIMITER ;

DELIMITER $$

CREATE PROCEDURE`partition_verify`(SCHEMANAME VARCHAR(64), TABLENAME VARCHAR(64),HOURLYINTERVAL INT(11))

BEGIN

        DECLARE PARTITION_NAME VARCHAR(16);

        DECLARE RETROWS INT(11);

        DECLARE FUTURE_TIMESTAMP TIMESTAMP;

        /*

         * Check if any partitions exist forthe given SCHEMANAME.TABLENAME.

         */

        SELECT COUNT(1) INTO RETROWS

        FROM information_schema.partitions

        WHERE table_schema = SCHEMANAME ANDtable_name = TABLENAME AND partition_name IS NULL;

        /*

         * If partitions do not exist, go aheadand partition the table

         */

        IF RETROWS = 1 THEN

                /*

                 * Take the current date at00:00:00 and add HOURLYINTERVAL to it. This is the timestamp below which we will store values.

                 * We begin partitioning basedon the beginning of a day.  This isbecause we don't want to generate a random partition

                 * that won't necessarily fallin line with the desired partition naming (ie: if the hour interval is 24hours, we could

                 * end up creating a partitionnow named "p201403270600" when all other partitions will be like"p201403280000").

                 */

                SET FUTURE_TIMESTAMP =TIMESTAMPADD(HOUR, HOURLYINTERVAL, CONCAT(CURDATE(), " ",'00:00:00'));

                SET PARTITION_NAME =DATE_FORMAT(CURDATE(), 'p%Y%m%d%H00');

                -- Create the partitioningquery

                SET @__PARTITION_SQL =CONCAT("ALTER TABLE ", SCHEMANAME, ".", TABLENAME, "PARTITION BY RANGE(`clock`)");

                SET @__PARTITION_SQL =CONCAT(@__PARTITION_SQL, "(PARTITION ", PARTITION_NAME, " VALUESLESS THAN (", UNIX_TIMESTAMP(FUTURE_TIMESTAMP), "));");

                -- Run the partitioning query

                PREPARE STMT [email protected]__PARTITION_SQL;

                EXECUTE STMT;

                DEALLOCATE PREPARE STMT;

        END IF;

END$$

DELIMITER ;

相關推薦

zabbix資料庫innodb遷移tokudb分割槽

下載percona server,並啟用tokudb引擎 tar zxvfPercona-Server-5.6.32-rel78.0-Linux.x86_64.ssl101.tar.gz -C /usr/local/ ln -s Percona-Server-5.6.32

abp .net core 資料庫sqlserver遷移到mysql

先拋個連結https://blog.csdn.net/sD7O95O/article/details/78941648 按照以上部落格中的操作,發現add-migration可以成功,但是update-database的時候報錯,具體報錯資訊不記得了, 大概的意思就是當前mysql的語法不匹

資料庫MYSQL遷移到POSTGRESQL

終於決定把資料庫從MYSQL轉到postgresql了。如何遷移是個問題。手工遷移太麻煩。終於從POSTGRESQL的網站上找到了一個小的遷移工具名字叫mysql2pgsql.perl,這是一個perl編寫的小程式。用法是先把MYSQL裡的資料DUMP下來儲存為mysql.s

mysql資料庫window遷移的linux的方法和注意事項

一般情況下Mysql從window遷移到linux的時候,網上都會有標準的教程如下: 1) 在windows平臺上進入/mysql/bin目錄(假設你的資料庫名字是mydata)       執行mysqldump 命令將你的資料庫匯出,具體命令如下:         

MySQL資料庫windows遷移到linux

前幾天搭建了lamp環境,想把之前寫的小東西遷到linux上執行,涉及到把mysql資料庫的檔案遷移到linux上,直接用fileZilla傳過去應該不行,我試了下,反正沒成功。下面是我採用的方法: (一)用mysqldump命令匯出資料庫檔案: 在windows下cd到M

oracle資料庫9i遷移到11g

0、匯出和匯入之前首先要確認兩個資料庫的字符集是否一致 Export和Import是一對讀寫Oracle資料的工具。Export將Oracle資料庫中的資料輸出到作業系統檔案中, Import把這些檔案中的資料讀到Oracle資料庫中,由於使用exp/imp進行資料遷移時,資料從源資料庫到目標資料庫的過程

文件服務器及域控制器2003遷移至Windows Server 2008 R2

共享文件 f11 ima 新建 png 命名 chm 基礎架構 width (一)背景環境:   當前,多數小企業仍然使用windows server2003 系統做域控制器及文件服務器,由於windows server 2003在多年使用之後變得卡頓,且存在異常的系統錯

啟動伺服器時配置引數資料庫中載入到快取

最近做專案,碰到這樣的需求:在伺服器啟動的時候從資料庫讀取引數,將引數儲存到記憶體快取中 由於使用的是spring的自動注入方式,一開始用@component註解在啟動的時候載入查詢配置引數的bean,由於bean中要用到其他bean來查詢,但此時都為null 查詢相關資料,發現@PostC

hustoj 資料Ubuntu 14.04 成功遷移到 ubuntu 16.04

一、首先需要搭建一個hustoj: 前幾天在Ubuntu16.04上搭建了一個hustoj,剛開始各種錯誤,不過最後還是成功搭建完畢。 搭建方法請參考:hustoj搭建 裡邊介紹了我在搭建中遇到的一些常見問題,最後總結了一個指令碼,直接執行即可安裝成功,中間會有四個檔案無法建

把kafka資料hbase遷移到hdfs,按天載入到hive表(hbase與hadoop為不同叢集)

需求:由於我們用的阿里雲Hbase,按儲存收費,現在需要把kafka的資料直接同步到自己搭建的hadoop叢集上,(kafka和hadoop叢集在同一個區域網),然後對接到hive表中去,表按每天做分割槽 一、首先檢視kafka最小偏移量(offset) /usr/local/kafka/bin/k

C#資料庫中讀取二進位制流生成檔案

下面以圖片檔案為例加以說明。從資料庫表 “圖片儲存”中讀取ID為1的圖片資料並生成圖片檔案。   MySqlConnection conn = new MySqlConnection("Server=localhost;Database=test;charset=utf8;U

無法資料庫SINGLE_USER模式切換回MULTI_USER模式(Error 5064)

今天公司SQL Server資料庫無意間變為SINGLE_USER模式了,而且使用如下語句切換回MULTI_USER失敗: ALTER DATABASE [MyDB] SET MULTI_USER WITH ROLLBACK IMMEDIATE 報錯: Msg 5064, Level

Ubuntu16.04中安裝SqlServer建立連線資料庫、跨平臺遷移教程連結、各種程式語言連線資料庫的教程連結

安裝SQL server 1、匯入公鑰 wget -qO- https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add - 2、註冊mssqlserver Ubuntu儲存庫 sudo

byte[]陣列上傳到資料庫,資料庫還原byte[]陣列為img圖片

 今天碰到的需求是將圖片檔案存入資料庫,而不是儲存一個圖片訪問的URL,不知道有沒有人遇到跟我一樣的需求,查詢過javaAPI後使用ByteArrayOutputStream 和ByteArrayInputStream實現了該功能,測試程式碼如下:  目標檔案為D://11.

windows7下Cygwin加入右鍵選單,當前目錄開啟

原文參考: http://blog.csdn.net/wjr2012/article/details/7964005 很詳細!!! 我在使用的時候遇到的問題是   修改windows登錄檔   1·開始->執行(或者win鍵+R),輸入REGED

資料庫資料放到redis中保持資料一致性方案

1、查詢時先查詢快取(redis),如果查詢不到,查詢資料庫,並將會員唯一標識(會員編號),放到redis佇列中(list) 2、啟動一個job從redis佇列中讀取會員唯一標識,並判斷會員編號作為key是否存在於redis中,如果存在,計數器減一,否則直接查詢會員資訊,放

系統Windows遷移至Linux下的點點滴滴

一、寫在最前   由於本人的技術水平有限,難免會出現錯誤。本文對任何一個人有幫助都是我莫大的榮幸,任何一個大神對我的點撥,我都會感激不盡。 二、技術選型   在2013年8月低的時候,公司中了XXX市場監督局肉品配送車輛監控的專案。整個系統軟體部分需要實現的功能不難,最大的難點就是伺服器的系統要求是Li

powerDesigner 資料庫匯出表結構,生產excel ,有效

將資料庫中的表匯入到PowerDesigner中並轉為excel文件 1、開啟PowerDesigner12,在選單中按照如下方式進行操作    file->Reverse Engineer->DataBase    點選後,彈出 New Physical D

零學springboot——springboot專案打成jar包部署

當我們開發完成一個spring boot專案後,是需要將其打包,並且部署到linux伺服器上的,這裡介紹以jar包的方式部署。 將專案打成jar包 對pom檔案中的maven外掛進行擴充套件 <build> <plugins>

關於平滑遷移Windows下MySQL資料庫一臺電腦到另外一臺電腦

目錄 1、使用場景      因為本人最近需要把以前使用的三星膝上型電腦,替換加入固態硬碟和擴充套件了記憶體的ThinkPad電腦,主要原始識公司開始購買的三星本無法擴充套件記憶體條,搞微服務開發,多開幾個服務;記憶體就直接到95%以上了,原來三星