Hive/Hbase/Sqoop的安裝教程
Hive/Hbase/Sqoop的安裝教程
HIVE INSTALL
1.下載安裝包：https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.3/
2.上傳到Linux指定目錄，解壓：
mkdir hive mv apache-hive-2.3.3-bin.tar.gz hive tar -zxvf apache-hive-2.3.3-bin.tar.gz mv apache-hive-2.3.3-bin apache-hive-2.3.3
### 安裝目錄為：/app/hive/apache-hive-2.3.3
3.配置環境變量：
sudo vi /etc/profile
添加環境變量：
export HIVE_HOME=/app/hive/apache-hive-2.3.3 export PATH=$PATH:$HIVE_HOME/bin
:wq #保存退出
4.修改HIVE配置文件：
配置文件hive-env.sh (在原有的基礎上修改，沒有的項就添加)：
cd /app/hive/apache-hive-2.3.3/conf cp hive-env.sh.template hive-env.sh ###在文件中添加如下內容-- 去掉#，並把目錄改為自己設定的目錄 export HADOOP_HEAPSIZE=1024 export HADOOP_HOME=/app/hadoop/hadoop-2.7.7 #hadoop的安裝目錄 export HIVE_CONF_DIR=/app/hive/apache-hive-2.3.3/conf export HIVE_HOME=/app/hive/apache-hive-2.3.3 export HIVE_AUX_JARS_PATH=/app/hive/apache-hive-2.3.3/lib export JAVA_HOME=/app/lib/jdk
創建hdfs文件目錄：
cd /app/hive/apache-hive-2.3.3 mkdir hive_site_dir cd hive_site_dir hdfs dfs -mkdir -p warehouse #使用這條命令的前提是hadoop已經安裝好了 hdfs dfs -mkdir -p tmp hdfs dfs -mkdir -p log hdfs dfs -chmod -R 777 warehouse hdfs dfs -chmod -R 777 tmp hdfs dfs -chmod -R 777 log 創建臨時文件夾： cd /app/hive/apache-hive-2.3.3 mkdir tmp
配置文件hive-site.xml (在原有的基礎上修改)：
cp hive-default.xml.template hive-site.xml
vi hive-site.xml
>>配置一些數據庫的信息 ConnectionURL/ConnectionUserName/ConnectionPassword/ConnectionDriverName
<!--mysql database connection setting --> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://10.28.85.149:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>szprd</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>szprd</value> </property>
>>配置hdfs文件目錄
<property>
<name>hive.exec.scratchdir</name>
<!--<value>/tmp/hive</value>-->
<value>/app/hive/apache-hive-2.3.3/hive_site_dir/tmp</value>
<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/app/hive/apache-hive-2.3.3/hive_site_dir/warehouse</value>
</property>
<property>
<name>hive.exec.local.scratchdir</name>
<!--<value>${system:java.io.tmpdir}/${system:user.name}</value> -->
<value>/app/hive/apache-hive-2.3.3/tmp/${system:user.name}</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<!--<value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>-->
<value>/app/hive/apache-hive-2.3.3/tmp/${hive.session.id}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<!--<value>${system:java.io.tmpdir}/${system:user.name}</value>-->
<value>/app/hive/apache-hive-2.3.3/hive_site_dir/log/${system:user.name}</valu
<description>Location of Hive run time structured log file</description>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic
schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
proper metastore schema migration. (Default)
False: Warn if the version information stored in metastore doesn‘t match with one from in Hive jars.
</description>
</property>
修改完配置文件後，:wq 保存退出
5.下載合適版本的mysql驅動包，復制到HIVE安裝目錄的 lib目錄下
https://dev.mysql.com/downloads/connector/j/
6.初始化數據庫（在啟動hive前一定要先執行這個命令哦，如果失敗了，請查看數據庫配置信息是否準確~ ）
cd /app/hive/apache-hive-2.3.3/bin ./schematool -initSchema -dbType mysql
7.啟動hive
hive #這裏配置了環境變量後，可以在任意目錄下執行 （/etc/profile）
8.實時查看日誌啟動hive命令（在hive安裝目錄的bin目錄下執行）：
./hive -hiveconf hive.root.logger=DEBUG,console
HBASE INSTALL
1.下載hbase安裝包: http://hbase.apache.org/downloads.html
2.解壓： tar -zxvf hbase-1.2.6.1-bin.tar.gz
3.配置環境變量： (加在最後面)
vi /etc/profile
#HBase Setting export HBASE_HOME=/app/hbase/hbase-1.2.6.1 export PATH=$PATH:$HBASE_HOME/bin
4.編輯配置文件： hbase-env.sh
export HBASE_MANAGES_ZK=false export HBASE_PID_DIR=/app/hadoop/hadoop-2.7.7/pids #如果該目錄不存在，則先創建 export JAVA_HOME=/app/lib/jdk #指定JDK的安裝目錄
編輯配置文件： hbase-site.xml
在configuration節點添加如下配置：
<property> <name>hbase.rootdir</name> <value>hdfs://192.168.1.202:9000/hbase</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/vc/dev/MQ/ZK/zookeeper-3.4.12</value> </property> <property> <name>zookeeper.znode.parent</name> <value>/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.unsafe.stream.capability.enforce</name> <value>false</value> <description> Controls whether HBase will check for stream capabilities (hflush/hsync). Disable this if you intend to run on LocalFileSystem, denoted by arootdir with the ‘file://‘ scheme, but be mindful of the NOTE below. WARNING: Setting this to false blinds you to potential data loss and inconsistent system state in the event of process and/or node failures.If HBase is complaining of an inability to use hsync or hflush it‘s most likely not a false positive. </description> </property>
5.啟動zookeeper
進入zookeeper的安裝目錄下的bin目錄，執行 ./zkServer.sh
然後啟動客戶端： ./zkCli.sh
啟動成功後，輸入： create /hbase hbase
6.啟動hbase
進入Hbase的bin目錄： ./start-hbase.sh
./hbase shell #這裏啟動成功後就可以開始執行hbase相關命令了
list #沒有報錯表示成功
7.web訪問HBASE: http://10.28.85.149:16010/master-status #ip為當前服務器的ip,端口為16010
#Sqoop install
1.下載安裝包： https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/
2.解壓： tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
更改文件名： mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7_hadoop-2.6.0
3. 配置環境變量：/etc/profile
#Sqoop Setting export SQOOP_HOME=/app/sqoop/sqoop-1.4.7_hadoop-2.6.0 export PATH=$PATH:$SQOOP_HOME/bin
4.將mysql的驅動包復制到 Sqoop安裝目錄的lib目錄下
https://dev.mysql.com/downloads/connector/j/
5.編輯配置文件： sqoop的安裝目錄下的 conf下
vi sqoop-env.sh
#Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/app/hadoop/hadoop-2.7.7 #Set path to where hadoop-*-core.jar is available export HADOOP_MAPRED_HOME=/app/hadoop/hadoop-2.7.7 #set the path to where bin/hbase is available export HBASE_HOME=/app/hbase/hbase-1.2.6.1 #Set the path to where bin/hive is available export HIVE_HOME=/app/hive/apache-hive-2.3.3 #Set the path for where zookeper config dir is export ZOOCFGDIR=/app/zookeeper/zookeeper-3.4.12
6,輸入命令：
sqoop help #查看相關的sqoop命令
sqoop version #查看sqoop的版本
ps:
關於停止hbase的命令： stop-hbase.sh ,出現關於pid的錯誤提示時，請參考這篇博文：https://blog.csdn.net/xiao_jun_0820/article/details/35222699
hadoop的安裝教程：http://note.youdao.com/noteshare?id=0cae2da671de0f7175376abb8e705406
zookeeper的安裝教程：http://note.youdao.com/noteshare?id=33e37b0967da40660920f755ba2c03f0
Hive/Hbase/Sqoop的安裝教程