大資料之Hive安裝配置
hive預設將元資料儲存到本地內嵌的Derby資料庫中,但是Derby不支援多會話連結,因此我們使用mysql資料庫來儲存hive的元資料。配置完成hiveSQL的元資料庫之後再開始安裝、配置hive。
1、線上安裝mysql
安裝命令為
sudo apt-get install mysql-server
當出現下面提示選擇的時候輸入y 同意繼續安裝
hduser@master:~$ sudo apt-getinstallmysql-server
Readingpackagelists... Done
Building dependency tree
Reading state information... Done
Thefollowingpackages were automatically installedandarenolongerrequired:
linux-headers-4.13.0-36 linux-headers-4.13.0-36-generic
linux-headers-4.13.0-41 linux-headers-4.13.0-41-generic
linux-image-4.13.0-36-generic linux-image-4.13.0-41-generic
linux-image-extra-4.13.0-36-generic linux-image-extra-4.13.0-41-generic
Use'sudo apt autoremove'toremove them.
Thefollowingadditional packages will be installed:
libaio1 libevent-core-2.0-5 libhtml-template-perl mysql-client-5.7
mysql-client-core-5.7 mysql-common mysql-server-5.7 mysql-server-core-5.7
Suggested packages:
libipc-sharedcache-perl mailx tinyca
Thefollowing
libaio1 libevent-core-2.0-5 libhtml-template-perl mysql-client-5.7
mysql-client-core-5.7 mysql-common mysql-servermysql-server-5.7
mysql-server-core-5.7
0 upgraded, 9 newly installed, 0toremoveand101notupgraded.
Needtoget18.1 MBofarchives.
Afterthis operation, 161 MBofadditional diskspacewill be used.
Doyou wanttocontinue? [Y/n] y
然後安裝繼續執行,輸出如下內容,完成安裝。中間會有讓設定root密碼提示,輸入密碼按向下的方向鍵確認,然後再輸入一次確認密碼
Get:1http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-commonall 5.7.22-0ubuntu0.16.04.1[15.4 kB]
Get:2http://mirrors.aliyun.com/ubuntuxenial/main amd64 libaio1 amd64 0.3.110-2 [6,356 B]
Get:3http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-client-core-5.7 amd64 5.7.22-0ubuntu0.16.04.1[6,425 kB]
Get:4http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-client-5.7 amd64 5.7.22-0ubuntu0.16.04.1[1,673 kB]
Get:5http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-server-core-5.7 amd64 5.7.22-0ubuntu0.16.04.1[7,268 kB]
Get:6http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 libevent-core-2.0-5 amd64 2.0.21-stable-2ubuntu0.16.04.1[70.6 kB]
Get:7http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-server-5.7 amd64 5.7.22-0ubuntu0.16.04.1[2,583 kB]
Get:8http://mirrors.aliyun.com/ubuntuxenial/main amd64 libhtml-template-perl all 2.95-2 [60.4 kB]
Get:9http://mirrors.aliyun.com/ubuntuxenial-updates/main amd64 mysql-server all 5.7.22-0ubuntu0.16.04.1[10.8 kB]
Fetched 18.1 MBin1min 56s (155 kB/s)
Preconfiguring packages ...
Selecting previously unselected package mysql-common.
(Reading database ... 285825 filesanddirectories currently installed.)
Preparing to unpack .../mysql-common_5.7.22-0ubuntu0.16.04.1_all.deb ...
Unpacking mysql-common(5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package libaio1:amd64.
Preparing to unpack .../libaio1_0.3.110-2_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-2) ...
Selecting previously unselected package mysql-client-core-5.7.
Preparing to unpack .../mysql-client-core-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-client-5.7.
Preparing to unpack .../mysql-client-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-server-core-5.7.
Preparing to unpack .../mysql-server-core-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-server-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package libevent-core-2.0-5:amd64.
Preparing to unpack .../libevent-core-2.0-5_2.0.21-stable-2ubuntu0.16.04.1_amd64.deb ...
Unpacking libevent-core-2.0-5:amd64 (2.0.21-stable-2ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for man-db (2.7.5-1) ...
Settingupmysql-common(5.7.22-0ubuntu0.16.04.1) ...
update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf)inauto mode
Selecting previously unselected package mysql-server-5.7.
(Reading database ... 285993 filesanddirectories currently installed.)
Preparing to unpack .../mysql-server-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-server-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package libhtml-template-perl.
Preparing to unpack .../libhtml-template-perl_2.95-2_all.deb ...
Unpacking libhtml-template-perl (2.95-2) ...
Selecting previously unselected package mysql-server.
Preparing to unpack .../mysql-server_5.7.22-0ubuntu0.16.04.1_all.deb ...
Unpacking mysql-server (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Settinguplibaio1:amd64 (0.3.110-2) ...
Settingupmysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Settingupmysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Settingupmysql-server-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Settinguplibevent-core-2.0-5:amd64 (2.0.21-stable-2ubuntu0.16.04.1) ...
Settingupmysql-server-5.7 (5.7.22-0ubuntu0.16.04.1) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf)inauto mode
Renaming removed key_bufferandmyisam-recover options (if present)
Settinguplibhtml-template-perl (2.95-2) ...
Settingupmysql-server (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
使用如下命令登入MySQL伺服器。在提示輸入密碼的地方輸入剛才設定的root密碼
hduser@master:/etc/network$ mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.22-0ubuntu0.16.04.1 (Ubuntu)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql>
2、在Mysql中建立hive元資料資料庫
建立hive資料庫
mysql>createdatabasehive;
將hive資料庫的所有許可權賦給hive使用者
mysql>create user 'hive'@'%' identified by 'hive';
mysql>grant all privileges on *.* to 'hive'@'%' with grant option;
mysql>flush privileges;
mysql>use hive;
mysql>select user,host,password from mysql.user;
注:在ubuntu中,預設情況下富貴地圖MySQL只允許本底登陸,所以需要修改配置檔案將地址繫結註釋掉
sudo vim /etc/mysql/my.cnf
#bind-address = 127.0.0.1 #註釋掉
3、安裝hive
下載最新穩定版hive
https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.3/apache-hive-2.3.3-bin.tar.gz
上傳到master節點的/home/hduser/Downloads目錄下
解壓
tar -zxvf Downloads/apache-hive-2.3.3-bin.tar.gz
更改安裝檔名稱
hduser@master:~$ mv apache-hive-2.3.3-bin/ ./hive
hduser@master:~$
配置使用者環境變數
vim ~/.bashrc
在檔案末尾新增如下內容
#HIVE
export HIVE_HOME=/home/hduser/hive
export PATH=$PATH:$HIVE_HOME/bin
最後重新匯入環境變數
source ~/.bashrc
注:通過source方法只是臨時重置了系統環境變數,若想系統環境變數永久生效重啟是不錯的方法
4、配置hive引數
配置hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!--指定元資料庫連線資訊-->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<!--指定元資料庫連線驅動程式主類-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<!--指定元資料庫使用者名稱-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<!--指定元資料庫使用者密碼-->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<description>password to use against metastore database</description>
</property>
</configuration>
我們也可指定某個資料夾來存放元資料
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/home/hduser/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
下載mysql-connector-java-5.1.46.tar.gz,下載連結為
https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
將其解壓後在資料夾中選擇mysql-connector-java-5.1.46-bin.jar上傳到master節點的/home/hduser/Downloads目錄下。最後將JDBC檔案放到hive的lib資料夾內即可。
5、啟動hive
Hive資料儲存到HDFS上,因此要先啟動hadoop
start-all.sh
格式化hive
schematool -dbType mysql -initSchema
啟動hive命令列
hive
HiveSQL的優點是學習成本低,可以通過類似SQL語句實現快速MapReduce統計,使MapReduce變得更加簡單,而不必開發專門的MapReduce應用程式。hive是十分適合資料倉庫的統計分析和Windows登錄檔檔案。