HIVE的安裝
阿新 • • 發佈：2018-07-30
jar包沖突 粘貼 count argument tab hadoop集群 mkdir affect actual
第一部分 安裝前準備
環境：workstation11 + centos 7 + hadoop-2.7.7 + mysql 5.6.40 + hive 2.3.3
首先安裝mysql，步驟看這裏。
接著安裝HIVE。
第二部分 正式安裝HIVE
1.開啟hadoop集群，並關閉hadoop的安全模式
[root@hadoop ~]# start-all.sh ... [root@hadoop ~]# hdfs dfsadmin -safemode leave Safe mode is OFF
2.上傳hive安裝包，解壓縮，重命名，修改環境變量
[root@hadoop ~]# cd /usr/local/ [root@hadoop local]# tar xzvf apache-hive-2.3.3-bin.tar.gz #解壓縮 [root@hadoop local]# mv apache-hive-2.3.3-bin hive #重命名 [root@hadoop local]# vi /etc/profile #配置環境變量 添加變量 export HIVE_HOME=/usr/local/hive PATH變量後面添加 :$HIVE_HOME/bin [root@hadoop local]# source /etc/profile #生成環境變量
註意：
如果有hadoop用戶的請進入到hadoop用戶下修改hadoop用戶下的環境變量vi ~/.bash_profile
如果有root用戶的請進入到root用戶下修改環境變量vi /etc/profile
配置完成之後，可查看hive版本信息
[root@hadoop local]# hive --version Hive 2.3.3 Git git://daijymacpro-2.local/Users/daijy/commit/hive -r 8a511e3f79b43d4be41cd231cf5c99e43b248383 Compiled by daijy on Wed Mar 28 16:58:33 PDT 2018 From source with checksum 8873bba6c55a058614e74c0e628ab022
3.進入/usr/local/hive/conf目錄，修改HIVE的4個配置文件
第一個：hive-env.sh
[root@hadoop ~]# cd /usr/local/hive/conf [root@hadoop conf]# cp hive-env.sh.template hive-env.sh [root@hadoop conf]# vi hive-env.sh #添加環境變量 export JAVA_HOME=/usr/java export HADOOP_HOME=/usr/local/hadoop export HIVE_HOME=/usr/local/hive export HIVE_CONF_DIR=/usr/local/hive/conf
第二個：hive-site.xml
首先修改mysql數據庫連接信息
[root@hadoop conf]# cp hive-default.xml.template hive-site.xml #註意命名改變 [root@hadoop conf]# vi hive-site.xml #修改mysql數據庫連接，註意是修改不是添加!!! <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value> </property>
然後創建相應目錄並在文件中修改
[root@hadoop conf]# mkdir -p /tmp/hive/local /tmp/hive/resources /tmp/hive/querylog /tmp/hive/operation_logs #創建目錄 [root@hadoop conf]# vi hive-site.xml #修改配置文件，註意是修改不是添加！！！ <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive/local</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/tmp/hive/resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/tmp/hive/querylog</value> <description>Location of Hive run time structured log file</description> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/tmp/hive/operation_logs</value> <description>Top level directory where operation logs are stored if logging functionality is enabled</description> </property>
第三個：hive-log4j2.properties
[root@hadoop conf]# cp hive-log4j2.properties.template hive-log4j2.properties
第四個：hive-exec-log4j2.properties
[root@hadoop conf]# cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
4.初始化hive數據庫
先復制mysql的驅動程序到hive/lib下面（我這裏是上傳mysql-connector-java-5.1.46-bin.jar到/usr/local/hive/lib）然後
[root@hadoop conf]# mysql -uroot -proot #登錄mysql
mysql> create user ‘hive‘ identified by ‘hive‘; #這句話是Oracle用法，其實不必執行
mysql> grant all privileges on *.* to hive@"%" identified by "hive" with grant option; #創建hive用戶同時設置遠程連接（因為hive-site.xml中配置的是hive用戶） Query OK, 0 rows affected (0.04 sec) mysql> flush privileges; #刷新權限 Query OK, 0 rows affected (0.09 sec) mysql> exit #退出 Bye [root@hadoop conf]# cd .. [root@hadoop hive]# bin/schematool -initSchema -dbType mysql #初始化mysql Metastore connection URL: jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive Starting metastore schema initialization to 2.3.0 Initialization script hive-schema-2.3.0.mysql.sql Initialization script completed schemaTool completed
5.啟動hive，成功！！！
[root@hadoop hive]# hive Logging initialized using configuration in file:/usr/local/hive/conf/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> exit; [root@hadoop hive]#
6.為hive建立hdfs使用目錄
[root@hadoop hive]# hdfs dfs -mkdir /usr [root@hadoop hive]# hdfs dfs -mkdir /usr/hive/ [root@hadoop hive]# hdfs dfs -mkdir -p /usr/hive/warehouse [root@hadoop hive]# hdfs dfs -chmod g+w /usr/hive/warehouse [root@hadoop hive]# hdfs dfs -mkdir /tmp [root@hadoop hive]# hdfs dfs -chmod g+w /tmp [root@hadoop hive]# hdfs dfs -ls -R / #查看 drwxrwxr-x - root supergroup 0 2018-07-27 14:49 /tmp drwxr-xr-x - root supergroup 0 2018-07-27 14:44 /usr drwxr-xr-x - root supergroup 0 2018-07-27 14:45 /usr/hive drwxrwxr-x - root supergroup 0 2018-07-27 14:45 /usr/hive/warehouse
第三部分 遇到錯誤
1.hive數據庫初始化報錯1
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
原因：發生jar包沖突
解決方案：移除其中一個jar包即可
[root@hadoop hive]# mv /usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar /usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar_bak
2.hive數據庫初始化報錯2
Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore connection User: APP Starting metastore schema initialization to 2.3.0 Initialization script hive-schema-2.3.0.mysql.sql Error: Syntax error: Encountered "<EOF>" at line 1, column 64. (state=42X01,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 Use --verbose for detailed stacktrace. *** schemaTool failed ***
原因：先確定hive-site.xml文件名是否正確，如果不對則必須改為hive-site.xml否則不生效。然後查看其中的mysql數據連接信息是否正確修改。我這裏犯了個錯誤就是直接從網上拷貝後粘貼到文件的上方了，後來檢查文件時發現文件中其實是有這四個標簽的並且都有默認值，估計執行時後面標簽的內容把我添加到前面的標簽內容給覆蓋掉了所以才沒有生效。
解決方法：到文件中分別找到這4個標簽並一一修改其中的內容，不要直接復制粘貼。這個文件比較大，可以下載到本地用NotPad++打開修改後再重新上傳覆蓋。
3.啟動hive報錯
[root@hadoop local]# hive Logging initialized using configuration in file:/usr/local/hive/conf/hive-log4j2.properties Async: true Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java:205) at org.apache.hadoop.fs.Path.<init>(Path.java:171) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:659) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:549) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:750) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at java.net.URI.checkPath(URI.java:1823) at java.net.URI.<init>(URI.java:745) at org.apache.hadoop.fs.Path.initialize(Path.java:202) ... 12 more
解決方法：建立相應目錄並在hive-site.xml文件中修改
[root@hadoop conf]# mkdir -p /tmp/hive/local /tmp/hive/resources /tmp/hive/querylog /tmp/hive/operation_logs #創建目錄 [root@hadoop conf]# vi hive-site.xml #修改配置文件，註意是修改不是添加！！！ <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive/local</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/tmp/hive/resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/tmp/hive/querylog</value> <description>Location of Hive run time structured log file</description> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/tmp/hive/operation_logs</value> <description>Top level directory where operation logs are stored if logging functionality is enabled</description> </property>View Code
HIVE的安裝