1. 程式人生 > >hadoop+hive本地模式配置

hadoop+hive本地模式配置

一、環境準備:

作業系統:Ubuntu 14.04.1 x86_64

hadoop:hadoop-2.7.6

下載連結:https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.6/

hive:apache-hive-2.3.3

下載連結:https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.3/

使用賬戶:learn_hadoop

二、hadoop配置

在/home/learn_hadoop下建立目錄/home/learn_hadoop/hadoop,將下載的安裝包hadoop-2.7.6.tar.gz儲存在該目錄下,執行如下操作解壓安裝包:

[email protected]:~$ cd ~/hadoop/

[email protected]:~/hadoop$ tar xvf hadoop-2.7.6

配置hadoop環境變數,vim ~/.bashrc,在檔案末尾新增如下內容並儲存:

export HADOOP_HOME=/home/learn_hadoop/hadoop/hadoop-2.7.6
export PATH=$PATH:/home/learn_hadoop/hadoop/hadoop-2.7.6/bin
export CLASSPATH=$CLASSPATH:/home/learn_hadoop/hadoop/hadoop-2.7.6/lib/*:.

export CLASSPATH=$CLASSPATH:/home/learn_hadoop/apache-hive-2.3.3-bin/hive/lib/*:.

由於本文介紹的是本地模式,所以hadoop暫時不需要修改配置檔案。

三、hive配置

在/home/learn_hadoop下建立目錄/home/learn_hadoop/hive,將下載的安裝包apache-hive-2.3.3-bin.tar.gz儲存在該目錄下,執行如下操作解壓安裝包:

[email protected]:~$ cd ~/hive/

[email protected]:~/hadoop$ tar xvf apache-hive-2.3.3-bin.tar.gz

配置hadoop環境變數,vim ~/.bashrc,在檔案末尾新增如下內容並儲存:

export HIVE_HOME=/home/learn_hadoop/hive/apache-hive-2.3.3-bin

export PATH=$PATH:/home/learn_hadoop/hive/apache-hive-2.3.3-bin/bin

設定好環境變數之後,使用source ~/.bashrc使環境變數生效。

接下來,需要修改hive的配置檔案:

[email protected]:~$ cd ~/hive/apache-hive-2.3.3-bin/conf/

[email protected]:~/hive/apache-hive-2.3.3-bin/conf$ cp hive-default.xml.template hive-site.xml

修改hive-site.xml:vim hive-site.xml,修改如下內容並儲存:

   <property>
     <name>hive.exec.local.scratchdir</name>
-    <value>${system:java.io.tmpdir}/${system:user.name}</value>
+    <value>/home/learn_hadoop/hive/iotmp</value>
     <description>Local scratch space for Hive jobs</description>
   </property>
   <property>
     <name>hive.downloaded.resources.dir</name>
-    <value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>
+    <value>/home/learn_hadoop/hive/iotmp</value>
     <description>Temporary local directory for added resources in the remote file system.</description>
   </property>
   <property>
@@ -363,7 +363,7 @@
   </property>
   <property>
     <name>hive.metastore.warehouse.dir</name>
-    <value>/user/hive/warehouse</value>
+    <value>/home/learn_hadoop/hive/warehouse</value>
     <description>location of default database for the warehouse</description>
   </property>
   <property>
@@ -542,7 +542,7 @@
   </property>
   <property>
     <name>javax.jdo.option.ConnectionURL</name>
-    <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
+    <value>jdbc:derby:;databaseName=/home/learn_hadoop/hive/metastore_db;create=true</value>
     <description>
       JDBC connect string for a JDBC metastore.
       To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
@@ -1682,7 +1682,7 @@
   </property>
   <property>
     <name>hive.querylog.location</name>
-    <value>${system:java.io.tmpdir}/${system:user.name}</value>
+    <value>/home/learn_hadoop/hive/iotmp</value>
     <description>Location of Hive run time structured log file</description>
   </property>
   <property>
@@ -3973,7 +3973,7 @@
   </property>
   <property>
     <name>hive.server2.logging.operation.log.location</name>
-    <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
+    <value>/home/learn_hadoop/hive/iotmp/operation_logs</value>
     <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
   </property>

   <property>

四、初始化derby資料庫

hive啟動需要指定資料庫,在第三節中配置了資料庫為derby,所以需要先完成對derby的初始化:

[email protected]:~$ ~/hive/apache-hive-2.3.3-bin/bin/schematool -initSchema -dbType derby

五、啟動hive

由於我們在第三節在~/.bashrc中添加了hive的bin目錄至PATH,所以直接執行hive即可:

[email protected]:~$ hive

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/yang/hive/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/yang/hadoop/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]


Logging initialized using configuration in jar:file:/home/yang/hive/apache-hive-2.3.3-bin/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive> 

六、

看到上述hive>介面之後,即可進行hive的測試練習。如下簡單的建立一個包含一個INT行資料a的TABLE x,並使用SELECT和DROP等對該TABLE進行操作:

hive> CREATE TABLE x(a INT);
OK
Time taken: 6.411 seconds
hive> SELECT * FROM a;
OK
Time taken: 1.672 seconds
hive> DROP TABLE a;
OK
Time taken: 1.742 seconds