hive計算示例

先將資料通過sqoop從mysql匯入hive，在hive執行mysql的查詢語句，得到與mysql一樣的執行結果

步驟：

1. mysql資料準備

• account賬號表
• detail收支資料表

CREATE TABLE `account` (
  `id` int(11) PRIMARY KEY AUTO_INCREMENT,
  `account` varchar(20),
  `name` varchar(5),
  `age` int(3)
);

insert into account(account, name,
 
 age) values("[email protected]", "Tom", 23);
insert into account(account, name, age) values("[email protected]", "Jack", 20);
insert into account(account, name, age) values("[email protected]", "Jone", 22);
insert into account(account, name, age) values("[email protected]", "Jimi", 25
 
);
insert into account(account, name, age) values("[email protected]", "Black", 24);
select * from account;



CREATE TABLE `detail` (
  `id` int(11) PRIMARY KEY AUTO_INCREMENT,
  `account` varchar(20),
  `income` double,
  `expenses` double,
  `time` varchar(10)
);

insert into detail(account, income,
 
 expenses, time) values("[email protected]", 10, 20, 2018-12-1);
insert into detail(account, income, expenses, time) values("[email protected]", 10, 30, 2018-12-4);
insert into detail(account, income, expenses, time) values("[email protected]", 13, 22, 2018-12-3);
insert into detail(account, income, expenses, time) values("[email protected]", 45, 25, 2018-12-2);
insert into detail(account, income, expenses, time) values("[email protected]", 34, 24, 2018-12-1);
insert into detail(account, income, expenses, time) values("[email protected]", 50, 20, 2018-12-1);
select * from detail;

2. 建立hive表

create table account (
	id int, 
	account string, 
	name string, 
	age int
) row format delimited fields terminated by '\t';

create table detail (
	id int, 
	account string, 
	income double, 
	expenses double, 
	time string
) row format delimited fields terminated by '\t';

3. 通過sqoop將mysq當中的資料直接匯入到hive當中

sqoop import --connect jdbc:mysql://localhost:3306/mydata --username root --password 123456 --table account --hive-import --hive-overwrite --hive-table account --fields-terminated-by '\t'

sqoop import --connect jdbc:mysql://localhost:3306/mydata --username root --password 123456 --table detail --hive-import --hive-overwrite --hive-table detail --fields-terminated-by '\t'

4. 計算結果，mysql和hive中計算結果一致

select a.account, a.name, d.total 
from account as a 
join(
	select account, sum(income - expenses) as total 
	from detail group by account
) as d 
on a.account=d.account;

mysql計算結果

+--------------+-------+-------+
| account      | name  | total |
+--------------+-------+-------+
| black@qq.com | Black |    10 |
| jack@qq.com  | Jack  |   -20 |
| jimi@qq.com  | Jimi  |    20 |
| jone@qq.com  | Jone  |    -9 |
| tom@qq.com   | Tom   |    20 |
+--------------+-------+-------+

hive計算結果

black@qq.com	Black	10.0
jack@qq.com	Jack	-20.0
jimi@qq.com	Jimi	20.0
jone@qq.com	Jone	-9.0
tom@qq.com 	Tom	 20.0

報錯及解決

報錯：

/tmp/hive on HDFS should be writable.

解決

> hadoop fs -chmod -R 777 /tmp

參考
hive啟動出現許可權錯誤 /tmp/hive on HDFS should be writable.

報錯：

Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR

解決：
往/etc/profile最後加入

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

然後重新整理配置，source /etc/profile

參考：
ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.