Presto連線MySQL
阿新 • • 發佈：2018-12-17
準備
準備兩臺機器，機器名為 bd1 和 bd2，，並且在這兩個節點上安裝 hadoop，hive，並且準備一個mysql資料庫。
- bd1 - coordinator節點
- bd2 - worker節點
安裝 presto server
wget -c https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.211/presto-server-0.211.tar.gz
tar zxvf presto-server-0.211.tar.gz
cd /opt/presto-server-0.211
mkdir etc
coordinator 節點配置
etc/node.properties
node.environment=production
node.id=ffffffff-ffff-ffff-ffff-fffffffffff1
node.data-dir=/opt/presto-server-0.211/data
etc/config.properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
query.max-memory=8GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node= etc/jvm.config
-server
-Xmx8G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
worker 節點配置
etc/node.properties
node.environment=production
node.id= etc/config.properties
coordinator=false
http-server.http.port=8080
query.max-memory=8GB
query.max-memory-per-node=1GB
query.max-total-memory-per-node=2GB
discovery.uri=http://bd1:8080
etc/jvm.config
-server
-Xmx8G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
執行Presto
分別在兩個節點上執行下面的啟動命令
# 啟動 Presto
bin/launcher start
# 停止 Presto
bin/launcher stop
# 前臺執行 Presto，建議剛開始的時候使用這種方式，如果配置有錯誤，可以立刻在console上看到錯誤資訊，方便除錯。
bin/launcher run
服務啟動後可以通過下面的地址來檢視服務狀態
- http://<coordinator_ip>:8080/ui/
設定 MySQL Connector
建立資料表
由於當前版本的 MySQL Connector 不支援建立資料表的操作，因此我們需要先通過MySQL的終端來建立資料表。
$ mysql -u root -p
MariaDB [(none)]> use test;
MariaDB [test]> create table user(id int not null, username varchar(32) not null, password varchar(32) not null);
MariaDB [test]> insert into user values(1,'user1','password1');
MariaDB [test]> insert into user values(2,'user2','password2');
MariaDB [test]> insert into user values(3,'user3','password3');
MariaDB [test]> select * from user;
+----+----------+-----------+
| id | username | password |
+----+----------+-----------+
| 1 | user1 | password1 |
| 2 | user2 | password2 |
| 3 | user3 | password3 |
+----+----------+-----------+
MySQL Connector 配置
在etc目錄下建立catalog目錄，然後建立 mysql.properties 檔案,編輯儲存後將此檔案複製的叢集中的其它節點相同目錄下。
etc/catalog/mysql.properties
connector.name=mysql
connection-url=jdbc:mysql://bd1:3306
connection-user=root
connection-password=<password>
獲取 Presto 命令列工具
# 下載
wget -c https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.211/presto-cli-0.211-executable.jar
# 這個jar檔案是一個自執行的jar包，因此為了使用方便，我們可以將其改名為presto
mv presto-cli-0.211-executable.jar presto
通過 Presto 操作 MySQL 資料庫
./presto --server bd1:8080 --catalog mysql --schema test
presto:test> show schemas from mysql;
Schema
--------------------
information_schema
performance_schema
test
(4 rows)
Query 20181024_011707_00040_szyec, FINISHED, 2 nodes
Splits: 19 total, 19 done (100.00%)
0:00 [4 rows, 64B] [9 rows/s, 147B/s]
presto:test> show tables from mysql.test;
Table
-------
user
(1 row)
Query 20181024_011716_00041_szyec, FINISHED, 2 nodes
Splits: 19 total, 19 done (100.00%)
0:01 [1 rows, 18B] [0 rows/s, 16B/s]
presto:test> select * from mysql.test.user;
id | username | password
----+----------+-----------
1 | user1 | password1
2 | user2 | password2
3 | user3 | password3
(3 rows)
Query 20181024_011724_00042_szyec, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [3 rows, 0B] [7 rows/s, 0B/s]
presto:test> insert into mysql.test.user values(4,'user4','password4');
INSERT: 1 row
Query 20181024_011938_00046_szyec, FINISHED, 2 nodes
Splits: 35 total, 35 done (100.00%)
0:03 [0 rows, 0B] [0 rows/s, 0B/s]
presto:test> select * from mysql.test.user;
id | username | password
----+----------+-----------
1 | user1 | password1
2 | user2 | password2
3 | user3 | password3
4 | user4 | password4
(4 rows)
Query 20181024_011943_00047_szyec, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:01 [4 rows, 0B] [6 rows/s, 0B/s]