1. 程式人生 > >基於MySQL元數據的Hive的安裝和簡單測試

基於MySQL元數據的Hive的安裝和簡單測試

信息 rop msu time trying 功能 條件 ans ack

引言: Hive是一種強大的數據倉庫查詢語言,類似SQL,本文將介紹怎樣搭建Hive的開發測試環境。

1. 什麽是Hive?

hive是基於Hadoop的一個數據倉庫工具,能夠將結構化的數據文件映射為一張數據庫表,並提供簡單的sql查詢功能,能夠將sql語句轉換為MapReduce任務進行執行。 其長處是學習成本低,能夠通過類SQL語句高速實現簡單的MapReduce統計。不必開發專門的MapReduce應用,十分適合數據倉庫的統計分析。

2. 依照Hive的準備條件

2.1 Hadoop集群環境已經安裝完成

 2.2 本文使用Ubuntu做為開發環境(14.04)

3. 安裝步驟

3.1 下載Hive包:apache-hive-0.13.1-bin.tar.gz

3.2 將其解壓到/opt文件夾下

   tar xzvf apache-hive-0.13.1-bin.tar.gz

3.3 環境變量設置

export HIVE_HOME=/opt/apache-hive-0.13

export PATH=$PATH:$HIVE_HOME/bin

export CLASSPATH=$CLASSPATH:$HIVE_HOME/bin

3.4. 改動hive-env.xml,復制hive-env.xml.template.

# Set HADOOP_HOME to point to a specific hadoop install directory

HADOOP_HOME=/opt/hadoop-1.2.1

# Hive Configuration Directory can be controlled by:

export HIVE_CONF_DIR=/opt/apache-hive-0.13/conf

3.5 改動hive-site.xml,主要改動數據庫的連接信息.
 <property>
  <name>hive.metastore.uris</name>
  <value>thrift://127.0.0.1:9083</value>
  <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://BladeStone-Laptop:3306/hive?createDatabaseIfNotExist=true</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
  <description>username to use against metastore database</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>123456</value>
  <description>password to use against metastore database</description>
</property>
3.6 安裝mysql數據庫(Ubuntu系統)

sudo apt-get install mysql-server

技術分享
3.7 創建mysql用戶hive

技術分享

3.8 在mysql中創建hive數據庫

技術分享

3.9 下載mysql驅動,並將驅動拷貝到hive_homelib類庫

  mysql-connector-java-5.1.31-bin.jar

3.10 啟動Hive

技術分享

3.11 在Hive中創建表

技術分享

3.12 登錄mysql。訪問hive數據庫

技術分享

技術分享

3.13 刪除Hive中的表

技術分享

3.14 登錄mysql,查詢TBLS中的數據

技術分享

4. 總結

 通過以上的步驟,我們完整的安裝了一個Hive,並通過加入和刪除一張數據庫表的演示,來展示了Hive和Mysql元數據庫之間的關系操作。

5. 常見問題

 5.1 直接通過hive命令來啟動,則會出現一下錯誤信息:   

Logging initialized using configuration in jar:file:/opt/apache-hive-0.13/lib/hive-common-0.13.1.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
	... 7 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
	... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
	at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:336)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
	... 19 more
)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:382)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
	... 17 more
  解決的方法: hive --service metastore , 用這條命令來啟動hive。

  5.2 配置 hive.metastore.uris
a.不啟動metastore和hiveserver服務
直接命令行hive進入hive shell環境。然後運行show databases 報錯例如以下:
ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
 b.另外一種種情況
1.配置 hive.metastore.uris
2.啟動metastore服務 hive --service metastore
然後直接命令行hive進入hive shell環境,然後運行show databases

c:第三種情況
1.凝視配置項 hive.metastore.uris
2.不啟動metastore服務
然後直接命令行hive進入hive shell環境。然後運行show databases
  

基於MySQL元數據的Hive的安裝和簡單測試