hive部署安裝(apache-hive-1.1.0)
阿新 • • 發佈:2019-01-11
http://blog.csdn.net/an342647823/article/details/46048403
以下操作在hdpsrc3 節點上操作 一,下載安裝包 1,下載hive http://mirrors.hust.edu.cn/apache/ 得到apache-hive-1.1.0.tar.gz ,放到該目錄下 /home/hdpsrc/ mysql-client-5.5.39-2.linux2.6.x86_64.rpm mysql-devel-5.5.39-2.linux2.6.x86_64.rpm mysql-server-5.5.39-2.linux2.6.x86_64.rpm mysql-shared-5.5.39-2.linux2.6.x86_64.rpm mysql-shared-compat-5.5.39-2.linux2.6.x86_64.rpm 拷貝到該目錄下 /home/hdpsrc/Desktop/mysql/ 二,安裝mysql 1,解除安裝系統自帶的mysql相關安裝包,僅解除安裝 mysql 開頭的包
localhost","hive",password("hive"));
create database hive;
grant all on hive.* to [email protected]'%' identified by 'hive';
grant all on hive.* to [email protected]'localhost' identified by 'hive';
flush privileges; 退出mysql exit 驗證hive使用者 mysql -uhive -phive show databases; 看到如下反饋資訊,則說明建立成功 mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | hive | | test | +--------------------+ 3 rows in set (0.00 sec) 退出mysql exit 三,安裝hive 1,解壓安裝包 cd ~ tar -zxvf apache-hive-1.1.0-bin.tar.gz 2,建立軟連線 ln -s apache-hive-1.1.0-bin hive
3,新增環境變數
vi .bash_profile
匯入下面的環境變數
export HIVE_HOME=/home/hdpsrc/hive
export PATH=$PATH:$HIVE_HOME/bin
使其有效
source .bash_profile
4,修改hive-site.xml
主要修改以下引數
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value> </property> <property> <name>hive.hwi.listen.port</name> <value>9999</value> <description>This is the port the Hive Web Interface will listen on</description> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>true</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value>false</value> </property> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>Username to use against metastore database</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Location of Hive run time structured log file</description> </property> cp hive/conf/hive-default.xml.template hive/conf/hive-site.xml 編輯hive-site.xml 5,拷貝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面 mv /home/hdpsrc/Desktop/mysql-connector-java-5.1.6-bin.jar /home/hdpsrc/hive/lib/ 6,把jline-2.12.jar拷貝到hadoop相應的目錄下,替代jline-0.9.94.jar,否則啟動會報錯 cp /home/hdpsrc/hive/lib/jline-2.12.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/ mv /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak / 7,穿件hive臨時資料夾 mkdir /home/hdpsrc/hive/iotmp 四,啟動測試hive 啟動hadoop後,執行hive命令 hive 測試輸入 show database; hive> show databases; OK default Time taken: 0.907 seconds, Fetched: 1 row(s) 遇到問題總結希望可以幫助遇到此類問題的人。 建議先建元資料庫,設定編碼latin1。否則建好元資料相關可能會出問題,如drop table 卡死, create table too long等等 hive對utf-8支援不好。設定完編碼latin1,發現table 欄位描述無法顯示中文。修改元資料庫表的字元 (1)修改表字段註解和表註解
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8
(2) 修改分割槽欄位註解:
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 ;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
(3)修改索引註解:
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
連線元資料設定 dbc:mysql://192.168.209.1:3306/metastore_hive_db?createDatabaseIfNotExist=true&characterEncoding=UTF-8
對於已經建好的表,不起作用。 最好安裝的時候就修改編碼格式。 元資料mysql遠端模式配置 <property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.223.129:9083</value>
<description>執行hive的主機地址及埠(特別重要ip不要弄錯)</description>
</property>
啟動元資料
bin/hive --service metastore &
到此hive已經安裝完成 備註:
以下操作在hdpsrc3 節點上操作 一,下載安裝包 1,下載hive http://mirrors.hust.edu.cn/apache/ 得到apache-hive-1.1.0.tar.gz ,放到該目錄下 /home/hdpsrc/ mysql-client-5.5.39-2.linux2.6.x86_64.rpm mysql-devel-5.5.39-2.linux2.6.x86_64.rpm mysql-server-5.5.39-2.linux2.6.x86_64.rpm mysql-shared-5.5.39-2.linux2.6.x86_64.rpm mysql-shared-compat-5.5.39-2.linux2.6.x86_64.rpm 拷貝到該目錄下 /home/hdpsrc/Desktop/mysql/ 二,安裝mysql 1,解除安裝系統自帶的mysql相關安裝包,僅解除安裝
rpm -qa|grep mysql
sudo rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
2,安裝
cd /home/hdpsrc/Desktop/mysql/
sudo rpm -ivh mysql-*
sudo cp /usr/share/mysql/my-large.cnf /etc/my.cnf
3,啟動設定mysql 啟動mysql服務 sudo service mysql start 設定為開機自啟動 sudo chkconfig mysql on 設定root使用者登入密碼 sudo/usr/bin/mysqladmin -u root password 'wu123' 登入mysql 以root使用者身份登入 mysql -uroot -pwu123 建立hive使用者,資料庫等 insert into mysql.user(Host,User,Password) values("create database hive;
grant all on hive.* to [email protected]'%' identified by 'hive';
grant all on hive.* to [email protected]'localhost' identified by 'hive';
flush privileges; 退出mysql exit 驗證hive使用者 mysql -uhive -phive show databases; 看到如下反饋資訊,則說明建立成功 mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | hive | | test | +--------------------+ 3 rows in set (0.00 sec) 退出mysql exit 三,安裝hive 1,解壓安裝包 cd ~ tar -zxvf apache-hive-1.1.0-bin.tar.gz 2,建立軟連線 ln -s apache-hive-1.1.0-bin hive
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value> </property> <property> <name>hive.hwi.listen.port</name> <value>9999</value> <description>This is the port the Hive Web Interface will listen on</description> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>true</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value>false</value> </property> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>Username to use against metastore database</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Location of Hive run time structured log file</description> </property> cp hive/conf/hive-default.xml.template hive/conf/hive-site.xml 編輯hive-site.xml 5,拷貝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面 mv /home/hdpsrc/Desktop/mysql-connector-java-5.1.6-bin.jar /home/hdpsrc/hive/lib/ 6,把jline-2.12.jar拷貝到hadoop相應的目錄下,替代jline-0.9.94.jar,否則啟動會報錯 cp /home/hdpsrc/hive/lib/jline-2.12.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/ mv /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak / 7,穿件hive臨時資料夾 mkdir /home/hdpsrc/hive/iotmp 四,啟動測試hive 啟動hadoop後,執行hive命令 hive 測試輸入 show database; hive> show databases; OK default Time taken: 0.907 seconds, Fetched: 1 row(s) 遇到問題總結希望可以幫助遇到此類問題的人。 建議先建元資料庫,設定編碼latin1。否則建好元資料相關可能會出問題,如drop table 卡死, create table too long等等 hive對utf-8支援不好。設定完編碼latin1,發現table 欄位描述無法顯示中文。修改元資料庫表的字元 (1)修改表字段註解和表註解
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8
(2) 修改分割槽欄位註解:
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 ;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
(3)修改索引註解:
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
連線元資料設定 dbc:mysql://192.168.209.1:3306/metastore_hive_db?createDatabaseIfNotExist=true&characterEncoding=UTF-8
對於已經建好的表,不起作用。 最好安裝的時候就修改編碼格式。 元資料mysql遠端模式配置 <property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.223.129:9083</value>
<description>執行hive的主機地址及埠(特別重要ip不要弄錯)</description>
</property>
啟動元資料
bin/hive --service metastore &
到此hive已經安裝完成 備註: