mysql在rancher上完成大庫的部署

阿新 • • 發佈：2020-12-21

1. 上兩篇部落格*（，）寫了資料庫的部署流程，的，可以有這兩種方法：

第一種方法中，dockerfile需要把每一步的操作打包成要給映象完成操作，有時候公司等其他情況限制，每一層的大小不能超過一定限制，因此此種方式很容易在構建映象的時候失敗。

第二種方法中，對sql語句執行匯入操作，速度過慢，很容易在rancher平臺部署的時候超過初始化的最大時間限制，造成服務啟動失敗。

因此，針對以上兩種情況，採用了一個新的方式：

首先，根據如下命令，完成mysql的某個庫的打包。（如果某個包的大小都大於構建映象時候的一層的大小，那麼幹脆別部署了，找人先把這個限制去掉或者提高下）

第一步：資料庫部署方法：
都是在本地資料庫234上面執行：
匯出資料庫的壓縮檔案方法：
（1）先 mkdir /var/lib/mysql-files/imdb
然後chmod -R 777 /var/lib/mysql-files/imdb
（2）執行mysqldump --no-autocommit --opt -h172.19.57.234 -uroot -p123456 --tab /var/lib/mysql-files/imdb wisekg(資料庫名)
（3）壓縮
cd /var/lib/mysql-files/
tar zcvf wisekg.tar.gz ./imdb
其中wisekg為庫名
若同時部署2個，需要對imdb進行更新名字；重複（1）-（3）步驟生成kb.tar.gz
最後將壓縮檔案放到ftp://ftp3.dc2.fzyun.io/wise/ai-nlp/mysql_server/中。

第二步：修改init.sh以及Dockerfile；
init.sh中主要修改不同資料庫對應的/docker-entrypoint-initdb.d/imdb名字，以及資料庫的名字；
Dockerfile中也是新增對應的資料庫壓縮檔案。

對應的dockerfile和init.sh分別為：

dockerfile:（會自動執行init.sh指令碼）

FROM founder/mysql:5.7

RUN apt-get update && apt-get update ; apt-get install -y parallel && chown -R mysql:mysql /docker-entrypoint-initdb.d

ADD init.sh /docker-entrypoint-initdb.d/
COPY kb.tar.gz /docker-entrypoint-initdb.d/
COPY FounderKB_real.tar.gz /docker-entrypoint-initdb.d/

ADD mysqld.cnf /etc/mysql/mysql.conf.d/

init.sh:

#!/bin/bash
whoami
tar zxvf /docker-entrypoint-initdb.d/kb.tar.gz -C /docker-entrypoint-initdb.d
mkdir /docker-entrypoint-initdb.d/imdbkb/split
cd /docker-entrypoint-initdb.d/imdbkb/split
for i in ../*.txt ; do echo $i ; split -d -a 6 -l 100000 -- $i `basename $i .txt`.; ls|grep `basename $i .txt`|xargs -n1 -i{} mv {} {}.txt; rm -fv $i;  done
for i in `ls *.*|sed 's/^[^.]+.//'|sort -u` ; do
  mkdir ../split-$i
  for j in $i ; do mv $j ../split-$i/$i ; done
done
cd /docker-entrypoint-initdb.d/imdbkb/
 time ( mysql -uroot -p"$MYSQL_ROOT_PASSWORD"  -e 'drop database if exists kb;create database kb;set global FOREIGN_KEY_CHECKS=0;'; (echo "SET FOREIGN_KEY_CHECKS=0;";cat ./*.sql) | mysql -uroot -p"$MYSQL_ROOT_PASSWORD" kb ; parallel -j32 'mysqlimport --use-threads=32 -uroot -p"$MYSQL_ROOT_PASSWORD" kb  /docker-entrypoint-initdb.d/imdbkb/{}/*.txt' ::: split-* )

whoami
tar zxvf /docker-entrypoint-initdb.d/FounderKB_real.tar.gz -C /docker-entrypoint-initdb.d
mkdir /docker-entrypoint-initdb.d/imdbFounderKB_real/split
cd /docker-entrypoint-initdb.d/imdbFounderKB_real/split
for i in ../*.txt ; do echo $i ; split -d -a 6 -l 100000 -- $i `basename $i .txt`.; ls|grep `basename $i .txt`|xargs -n1 -i{} mv {} {}.txt; rm -fv $i;  done
for i in `ls *.*|sed 's/^[^.]+.//'|sort -u` ; do
  mkdir ../split-$i
  for j in $i ; do mv $j ../split-$i/$i ; done
done
cd /docker-entrypoint-initdb.d/imdbFounderKB_real/
 time ( mysql -uroot -p"$MYSQL_ROOT_PASSWORD"  -e 'drop database if exists FounderKB_real;create database FounderKB_real;set global FOREIGN_KEY_CHECKS=0;'; (echo "SET FOREIGN_KEY_CHECKS=0;";cat ./*.sql) | mysql -uroot -p"$MYSQL_ROOT_PASSWORD" FounderKB_real ; parallel -j32 'mysqlimport --use-threads=32 -uroot -p"$MYSQL_ROOT_PASSWORD" FounderKB_real  /docker-entrypoint-initdb.d/imdbFounderKB_real/{}/*.txt' ::: split-* )

rm -r /docker-entrypoint-initdb.d/imdbkb/
rm -r /docker-entrypoint-initdb.d/imdbFounderKB_real/

mysqld.cnf:

#Copyright (c) 2014, 2016, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
datadir        = /var/lib/mysql
#log-error    = /var/log/mysql/error.log
# By default we only accept connections from localhost
#bind-address    = 127.0.0.1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0


innodb_buffer_pool_size = 4G
innodb_write_io_threads = 64
innodb_read_io_threads = 64

innodb_thread_concurrency=0

innodb_flush_method=O_DIRECT
#innodb_io_capacity=2000
#innodb_io_capacity_max=4000
max_allowed_packet=2G
skip-log-bin
sync_binlog=0
secure_file_priv=''

innodb_flush_log_at_trx_commit = 0
innodb_log_buffer_size = 1G
innodb_log_file_size = 3G

innodb_autoinc_lock_mode=2
innodb_doublewrite=0
innodb_old_blocks_pct=5
skip-innodb_adaptive_hash_index

secure_file_priv =