apache-hive-1.2.1-bin 安裝

更多資源:https://github.com/opensourceteams

技能標籤

• 下載apache hive 安裝包
• 進行apache-hive-1.2.1-bin.tar.gz安裝
• 配置mysql儲存元資料
• 配置HDFS儲存 /user/hive/warehouse
• 該Hive版本1.2.1預設整合在(Spark1.6 到 Spark2.4)

下載

• 下載安裝包apache-hive-1.2.1-bin.tar.gz: https://archive.apache.org/dist/hive/hive-1.2.1
• 由於spark-2.4.0-bin-hadoop2.7 預編譯好的捆綁hive版本為hive-1.2.1,為了方便所以選用版本apache-hive-1.2.1-bin
• 如果spark支援其它版本的hive,需要重新手動編譯spark原始碼也可以實現

安裝

解壓安裝包

tar -zxvf apache-hive-1.2.1-bin.tar.gz -C /opt/module/bigdata/

配置環境變數

export HIVE_HOME=/opt/module/bigdata/apache-hive-1.2.1-bin
export PATH=$HIVE_HOME/lib:#PATH

 

HDFS新建目錄

bin/hadoop fs -mkdir -p  /user/hive/warehouse #建立目錄
bin/hadoop fs -chmod -R 777 /user/hive/warehouse #新建的目錄賦予讀寫許可權
bin/hadoop fs -mkdir -p /tmp/hive/#新建/tmp/hive/目錄
bin/hadoop fs -chmod -R 777 /tmp/hive #目錄賦予讀寫許可權
#用以下命令檢查目錄是否建立成功
bin/hadoop fs -ls /user/hive
bin/hadoop fs -ls /tmp/hive

連線mysql資料庫

• 複製jdbc連線驅動

cp mysql-connector-java-8.0.13.jar  $HIVE_HOME/lib/

新建hive-site.xml資料庫相關的配置

• touch hive-site.xml檔案

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
	 <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>  
  </property>
  
  <property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>  
  </property>
  
  <property>
    <name>javax.jdo.option.ConnectionURL</name> 
    <value>jdbc:mysql://macbookmysql.com:3306/hive?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>admin</value>
    <description>username to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>000000</value>
    <description>password to use against metastore database</description>
  </property>


</configuration>	

新建hive-env.sh

• cp hive-env.sh.template

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hive and Hadoop environment variables here. These variables can be used
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
#
# The hive service being invoked (CLI/HWI etc.) is available via the environment
# variable SERVICE


# Hive Client memory usage can be an issue if a large number of clients
# are running at the same time. The flags below have been useful in 
# reducing memory usage:
#
# if [ "$SERVICE" = "cli" ]; then
#   if [ -z "$DEBUG" ]; then
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
#   else
#     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
#   fi
# fi

# The heap size of the jvm stared by hive shell script can be controlled via:
#
# export HADOOP_HEAPSIZE=1024
#
# Larger heap size may be required when running queries over large number of files or partitions. 
# By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be 
# appropriate for hive server (hwi etc).


# Set HADOOP_HOME to point to a specific hadoop install directory
# HADOOP_HOME=${bin}/../../hadoop

# Hive Configuration Directory can be controlled by:
# export HIVE_CONF_DIR=

# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
# export HIVE_AUX_JARS_PATH=




HADOOP_HOME=/opt/module/bigdata/hadoop-2.9.2
export HIVE_CONF_DIR=/opt/module/bigdata/apache-hive-1.2.1-bin/conf
export HIVE_AUX_JARS_PATH=/opt/module/bigdata/apache-hive-1.2.1-bin/lib

初使化元資料資料庫

schematool -initSchema -dbType mysql

啟動hive

hive

• hive啟動說明

啟動hive metastore服務

• 啟動該服務會開一個 9083埠
• netstat -ntlup 可以檢視服務端佔用的埠

hive --service metastore & 

停止hive metastore服務

ps -ef|grep hive

kill -9 pid

啟動hiveserver服務 (該服務可以不啟動)

hive --service hiveserver2 & 

操作

啟動 hive

hive

退出 hive命令

exit;

建立database

create database;

顯示所有的database

show databases;

顯示所有的表

show tables;

建立表

CREATE TABLE IF NOT EXISTS employee (  name String, salary String) COMMENT 'Employee details' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE;

插入資料

• 資料檔案 employee.txt

小明    100000
小王    50000
小李    60000

• 插入資料檔案到hive上

LOAD DATA LOCAL INPATH '/home/liuwen/temp/employee.txt' OVERWRITE INTO TABLE employee;

查看錶資料

 select * from employee;

問題處理

無法訪問spark-assembly-*.jar

[[email protected] lib]$ schematool -initSchema -dbType mysql
ls: 無法訪問/opt/module/bigdata/spark-2.4.0-bin-without-hadoop/lib/spark-assembly-*.jar: 沒有那個檔案或目錄
Metastore connection URL:        jdbc:mysql://macbookmysql.com:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.cj.jdbc.Driver
Metastore connection User:       admin
Starting metastore schema initialization to 1.2.0
Initialization script hive-schema-1.2.0.mysql.sql
Error: Duplicate key name 'PCS_STATS_IDX' (state=42000,code=1061)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***

• 處理方法
• 處理$HIVE_HOME/bin/hive

  #sparkAssemblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`
  sparkAssemblyPath=`ls ${SPARK_HOME}/jars/*.jar`

end