6、Canal實現MySQL到ES實時同步-1
阿新 • • 發佈：2022-03-29
1、準備一臺虛擬機器
1.1 配置靜態ip
可以參考 https://www.cnblogs.com/braveym/p/9096402.html
1.2給hadoop使用者賦予root許可權
切換到root使用者
並輸入命令
visudo
這時會進入/etc/sudoers檔案的編輯頁面，增加如下配置即可
hadoop ALL=(ALL) NOPASSWD: ALL #以下這行一定要註釋掉以免被覆蓋(hadoop使用者屬於wheel組) #%wheel ALL=(ALL) ALL
1.3 配置hostname
永久修改hostname執行如下命令
sudo hostnamectl set-hostname canal01
1.4 配置hosts檔案
sudo vi /etc/hosts
新增以下內容
172.16.25.31 canal01
2、安裝Mysql
配置Mysql 8.0安裝源
sudo rpm -Uvh https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
安裝mysql
sudo yum -y install mysql-community-server
出現以下錯誤
warning: /var/cache/yum/x86_64/7/mysql80-community/packages/mysql-community-client-8.0.28-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 3a79bd29: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql The GPG keys listed for the "MySQL 8.0 Community Server" repository are already installed but they are not correct for this package. Check that the correct key URLs are configured for this repository. Failing package is: mysql-community-client-8.0.28-1.el7.x86_64 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
出現上述提示原因是Mysql的GPG升級了，需要重新獲取
執行以下命令解決：
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
再次進行服務安裝：sudo yum -y install mysql-server就可以了
設定開機啟動mysql:
sudo systemctl enable mysqld
啟動mysql:
sudo systemctl start mysqld
檢視Mysql狀態：
sudo systemctl status mysqld
檢視root的臨時密碼：
sudo grep 'temporary password' /var/log/mysqld.log
我們看到隨機生成的root臨時密碼是(注意每次隨機生成密碼不一樣，不要死記)
修改root密碼，先進入mysql命令列:
mysql -u root -p
這時會要求輸入密碼，我們輸入上述臨時密碼然後回車即可，然後按照如下命令把root使用者的密碼改為root%123
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'Root_12root'; Query OK, 0 rows affected (0.01 sec) mysql> SHOW VARIABLES LIKE 'validate_password%'; +--------------------------------------+--------+ | Variable_name | Value | +--------------------------------------+--------+ | validate_password.check_user_name | ON | | validate_password.dictionary_file | | | validate_password.length | 8 | | validate_password.mixed_case_count | 1 | | validate_password.number_count | 1 | | validate_password.policy | MEDIUM | | validate_password.special_char_count | 1 | +--------------------------------------+--------+ 7 rows in set (0.00 sec) mysql> set global validate_password.policy=0; Query OK, 0 rows affected (0.01 sec) mysql> set global validate_password.length=1; Query OK, 0 rows affected (0.00 sec) mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root%123'; Query OK, 0 rows affected (0.00 sec) mysql> exit Bye
在生產上建議把root密碼設定的更加複雜，我這裡設定的root只能本機連線
3、MySQL特殊處理
在生產上Canal-Admin自己用的MySQL和你要增量採集binlog的MySQL大概率不是同一個，甚至在不 同的機器上。為了簡單，我們這裡採用同一個MySQL，生產上自己要注意。 為Canal-Server採集資料建立使用者
mysql -u root -p
set global validate_password.policy=0; set global validate_password.length=1; CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
exit
為Canal-Admin建立使用者
mysql -u root -p
set global validate_password.policy=0; set global validate_password.length=1; CREATE USER canaladmin IDENTIFIED BY 'canaladmin';revoke all privileges,grant option from 'canaladmin'@'%';
GRANT ALL ON canal_manager.* TO 'canaladmin'@'%';
FLUSH PRIVILEGES;
exit
配置源MySQL的Binlog格式
sudo vi /etc/my.cnf
增加以下配置：
server-id=1 log-bin=mysql-bin binlog-format=ROW binlog-ignore-db=information_schema binlog-ignore-db=mysql binlog-ignore-db=performance_schema binlog-ignore-db=syslog-bin用於指定binlog日誌檔名字首，預設儲存在/var/lib/mysql 目錄下。 server-id用於標識唯一的資料庫，不能和別的伺服器重複，建議使用ip的最後一段，預設值也不可以。 binlog-ignore-db：表示同步的時候忽略的資料庫。 binlog-do-db：指定需要同步的資料庫（如果沒有此項，表示同步所有的庫）。 配置完儲存退出，然後重啟MySQL。
登入mysql
mysql -uroot -p如果能看到上面的圖，則binlog 就成功開啟了。
禁用explicit_defaults_for_timestamp
mysql -uroot -p
SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%'; set persist explicit_defaults_for_timestamp=0; SHOW VARIABLES LIKE '%explicit_defaults_for_timestamp%';
重啟MySQL使上面修改生效：
sudo systemctl status mysqld
安裝MySQL的Java驅動
sudo mkdir -p /usr/share/java
將MySQL的java驅動mysql-connector-java-8.0.18.jar上傳到該目錄下
如果本地沒有改驅動板也可以通過wget下載一個
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.18/mysql-connector-java-8.0.18.jar
4、安裝Canal 下載安裝包
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.admin-1.1.4.tar.gz wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz
安裝Canal-Admin
解壓縮
mkdir -p app/canal-admin tar -zxvf canal.admin-1.1.4.tar.gz -C app/canal-admin cd app/canal-admin
配置環境變數
sudo vim /etc/profile
末尾新增：
#canal-admin export CANAL_ADMIN_HOME=/home/hadoop/app/canal-admin export PATH=${CANAL_ADMIN_HOME}/bin:$PATH
使環境變數生效：
source /etc/profile
修改配置
server: port: 8089 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 spring.datasource: address: canal01:3306 database: canal_manager username: canal password: canal driver-class-name: com.mysql.jdbc.Driver url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false hikari: maximum-pool-size: 30 minimum-idle: 1 canal: adminUser: admin adminPasswd: admin
替換MySQL驅動包 我們安裝的MySQL8，但是Canal-Admin預設的MySQL驅動不支援MySQL8，因此需要替換驅動包：
ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_ADMIN_HOME/lib/mysql-connector-java-8.0.18.jar
rm $CANAL_ADMIN_HOME/lib/mysql-connector-java-5.1.40.jar
初始化元資料庫 登入mysql執行以下命令：
source /home/hadoop/app/canal-admin/conf/canal_manager.sql
[hadoop@canal01 lib]$ mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 8.0.28 MySQL Community Server - GPL Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> source /home/hadoop/app/canal-admin/conf/canal_manager.sql Query OK, 1 row affected, 2 warnings (0.00 sec) Database changed Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected, 1 warning (0.01 sec) Query OK, 0 rows affected, 2 warnings (0.01 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 2 warnings (0.01 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 4 warnings (0.01 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 4 warnings (0.01 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 6 warnings (0.01 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 2 warnings (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) mysql>
啟動canal-admin
sh $CANAL_ADMIN_HOME/bin/startup.sh
檢視日誌：
tail -n 100 admin.log
報錯了！！！
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2022-03-28 15:21:00.496 [main] ERROR org.springframework.boot.SpringApplication - Application run failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in class path resource [com/alibaba/otter/canal/admin/config/EbeanConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) at com.alibaba.otter.canal.admin.CanalAdminApplication.main(CanalAdminApplication.java:19) Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579) ... 16 common frames omitted Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed at io.ebeaninternal.server.core.DefaultContainer.checkDataSource(DefaultContainer.java:323) at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:106) at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:35) at io.ebean.EbeanServerFactory.createInternal(EbeanServerFactory.java:109) at io.ebean.EbeanServerFactory.create(EbeanServerFactory.java:70) at com.alibaba.otter.canal.admin.config.EbeanConfig.ebeanServer(EbeanConfig.java:38) at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f.CGLIB$ebeanServer$0(<generated>) at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f$$FastClassBySpringCGLIB$$d3e49a6c.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361) at com.alibaba.otter.canal.admin.config.EbeanConfig$$EnhancerBySpringCGLIB$$c85da08f.ebeanServer(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 17 common frames omitted Caused by: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:365) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:194) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:460) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:534) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) at io.ebeaninternal.server.core.DefaultContainer.checkDataSource(DefaultContainer.java:316) ... 32 common frames omitted
在mysql的jdbc連線引數新增allowPublicKeyRetrieval=true
修改配置檔案的jdbc連線引數
重新啟動，並檢視日誌
通過瀏覽器開啟頁面 http://canal01:8089/#/login?redirect=%2Fdashboard
預設賬號密碼：admin/123456
其他操作：
sh $CANAL_ADMIN_HOME/bin/stop.sh //停止 sh $CANAL_ADMIN_HOME/bin/restart.sh //重啟
安裝Canal-Server 引入了canal-admin之後，canal-server之前面向命令列的運維方式需要有一些變化，主要的變化在於 配置體系上，每個server節點上不應該再去維護複雜而且冗長的 canal.properties/instance.properties，應該選擇以最小化、無狀態的方式去啟動，因此在canal 1.1.4 上，對於配置做了一些重構來支援canal-admin，同時也相容了原先的命令列運維模式。
解壓縮安裝包：
cd ~; mkdir -p app/canal-server; tar -zxvf canal.deployer-1.1.4.tar.gz -C app/canal-server;配置環境變數：
sudo vim /etc/profile
末尾新增：
#canal-server export CANAL_SERVER_HOME=/home/hadoop/app/canal-server export PATH=${CANAL_SERVER_HOME}/bin:$PATH
使環境變數生效：
source /etc/profile
配置canal-server 採用canal-admin來管理canal-server之後，不再需要按照以前的方式修改大量配置了。
cd $CANAL_SERVER_HOME/conf/;
mv canal.properties canal.properties.bak;
mv canal_local.properties canal.properties;
vim $CANAL_SERVER_HOME/conf/canal.properties;
配置內容如下：
# register ip canal.register.ip = # canal admin config canal.admin.manager = canal01:8089 canal.admin.port = 11110 canal.admin.user = admin canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441 # admin auto register canal.admin.register.auto = true canal.admin.register.cluster =
替換MySQL驅動包
我們安裝的MySQL8，但是Canal-Admin預設的MySQL驅動不支援MySQL8，因此需要替換驅動包ln -s /usr/share/java/mysql-connector-java-8.0.18.jar $CANAL_SERVER_HOME/lib/mysql-connector-java-8.0.18.jar
rm $CANAL_SERVER_HOME/lib/mysql-connector-java-5.1.47.jar
啟停canal-server
$CANAL_SERVER_HOME/bin/startup.sh $CANAL_SERVER_HOME/bin/stop.sh $CANAL_SERVER_HOME/bin/restart.sh