Hive與HBase整合(例項)
阿新 • • 發佈:2019-01-11
例項1
1.先在Hbase中建立表(三列族):
create 'ceshi7', {NAME=>'TIME',VERSIONS=>1,BLOCKCACHE=>true,BLOOMFILTER=>'ROW',COMPRESSION=>'SNAPPY', DATA_BLOCK_ENCODING => 'PREFIX_TREE', BLOCKSIZE => '65536'}, {SPLITS => ['1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}, {NAME=>'ORDERITEM',VERSIONS=>1,BLOCKCACHE=>true,BLOOMFILTER=>'ROW',COMPRESSION=>'SNAPPY', DATA_BLOCK_ENCODING => 'PREFIX_TREE', BLOCKSIZE => '65536'}, {SPLITS => ['1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}, {NAME=>'ORDERTYPE',VERSIONS=>1,BLOCKCACHE=>true,BLOOMFILTER=>'ROW',COMPRESSION=>'SNAPPY', DATA_BLOCK_ENCODING => 'PREFIX_TREE', BLOCKSIZE => '65536'}, {SPLITS => ['1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}
2.之後在Hive中建立外部表:
create external table lqioc_ioc_mid.ceshi7( rowid string, ordertypeno string, ordertypename string, ordertypecost string, yearid string, monthid string, orderitemname string, orderitemnum string)STORED BY'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES("hbase.columns.mapping" =":key,ORDERTYPE:ORDERTYPENO,ORDERTYPE:ORDERTYPENAME,ORDERTYPE:ORDERTYPECOST, TIME:YEARID,TIME:MONTHID, ORDERITEM:ORDERITEMNAME,ORDERITEM:ORDERITEMNUM") TBLPROPERTIES("hbase.table.name" = "ceshi7","hbase.mapred.output.outputtable" = "ceshi7");
3.報錯資訊:
Error: Error while processing statement:
FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:MetaException(message:
Cannot CREATE EXTERNAL TABLE when
hive.server2.enable.doAs is set to false.)
hive.server2.enable.doAs=false,則yarn作業獲取到的hiveserver2使用者都為hive使用者
hive.server2.enable.doAs=true,則為實際的使用者名稱
例項2
1.直接在Hive中建內部表:
create table lqioc_ioc_mid.ceshi6(
rowid string,
ordertypeno string,
ordertypename string,
ordertypecost string,
yearid string,
monthid string,
orderitemname string,
orderitemnum string)STORED BY'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES("hbase.columns.mapping"
=":key,ORDERTYPE:ORDERTYPENO,ORDERTYPE:ORDERTYPENAME,ORDERTYPE:ORDERTYPECOST,
TIME:YEARID,TIME:MONTHID,
ORDERITEM:ORDERITEMNAME,ORDERITEM:ORDERITEMNUM")
TBLPROPERTIES("hbase.table.name" = "ceshi6","hbase.mapred.output.outputtable" = "ceshi6");
2.hive中執行插入語句:
insert into table lqioc_ioc_mid.z_area_monitor_pasq
select
concat(FROM_UNIXTIME(UNIX_TIMESTAMP()),'_',AREA_NAME,'_',ORDER_TYPE_NO,
'_',ORDER_TYPE,'_',YEAR_ID,'_',MONTH_ID,'_',ORDER_ITEM),
ORDER_TYPE_NO,
ORDER_TYPE,
ORDER_TYPE_COST,
YEAR_ID,
MONTH_ID,
ORDER_ITEM,
ORDER_ITEM_NUM from lqioc_ioc_dw.pasq_modify;
3.檢測執行情況
select * from lqioc_ioc_dw.pasq_modify;
select * from lqioc_ioc_mid.ceshi6;
scan 'ceshi6'
get 'ceshi6','Z641201812龍洲新城'
4.先刪除Hbase表,再去查詢Hive對映表,會報錯(*^▽^*)
disable 'ceshi7'
drop 'ceshi7'
select * from lqioc_io_mid.ceshi7;
Error: java.io.IOException: org.apache.hadoop.hbase.TableNotFoundException: ceshi (state=,code=0)
5.之後在Hive檢視有這張表,但是刪除Hive中的表會報錯,show之後查不到表:
drop table lqioc_ioc_mid.ceshi6;
use lqioc_ioc_mid;
show tables;
Error: Error while processing statement: FAILED:
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:org.apache.hadoop.hbase.TableNotFoundException: ceshi7
6.根據rowkey查詢歷史資料
scan 'z_area_monitor_pasq', {STARTROW=>'2019-01-04 15:31:52', STOPROW=>'2019-01-04 15:36:52'}
scan 'z_area_monitor_pasq', {STARTROW=>'2019-01-04 15:31:52', ENDROW=>'2019-01-04 15:36:52'}
7.修改資料版本,獲取多版本資料
alter 'z_area_monitor_pasq',{NAME=>'ROWKEY1',VERSIONS=>3}
get 'z_area_monitor_pasq','rowkey1',{COLUMN=>'rowkey1:name',VERSIONS=>3}
引申:RowKey查詢(Scan和Get比較)
一、HBase查詢方式
hbase的查詢實現只提供兩種方式:
按指定rowkey獲取唯一一條記錄:get方法。
按指定條件獲取一批記錄:scan方法。
實現條件查詢功能使用的就是scan方式,scan在使用時有以下幾點值的注意:
scan可以通過setCaching與setBatch方法提高速度(以空間換時間)
scan可以通過setStartRow與setEndRow來限定範圍。範圍越小,效能越高。
scan可以通過setFilter方法新增過濾器,這也是分頁(效能差)、多條件查詢的基礎。
二、RowFilter介紹
operator | description |
less |
小於 |
less_or_equal |
小於等於 |
equal | 等於 |
not_equal | 不等於 |
greater_or_equal | 大於等於 |
greater | 大於 |
no_op | 排除所有 |
Comparator | description |
BinaryComparator | 使用bytes.comparaTo()比較 |
BinaryPrefixComparator | 和BinaryComparator差不多,從前面開始比較 |
NullComparator | |
BitComparator | |
RegexStringComparator | 正則表示式 |
subStringComparator | 把數字當成字串,用contains()來判斷 |
參考:
Get:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#setFilter-org.apache.hadoop.hbase.filter.Filter-
Scan:http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setFilter-org.apache.hadoop.hbase.filter.Filter-
RowKey使用數字比字串好:https://blog.csdn.net/alphags/article/details/53786777
掃描引數設定: http://grokbase.com/t/hbase/user/126vtkfr7h/scan-vs-put-vs-get
RowKey查詢(Scan和Get比較):https://blog.csdn.net/high2011/article/details/80205000
RowKey設計:http://blog.chedushi.com/archives/9720
對比:http://student-lp.iteye.com/blog/2309841
Hbase中多版本(version)資料獲取辦法:https://blog.csdn.net/shujuelin/article/details/83657485
Hbase常用操作整理:https://blog.csdn.net/sinat_36121406/article/details/82768846
Hadoop HBase概念學習系列之優秀行鍵設計(十六):https://www.cnblogs.com/zlslch/p/6140487.html
HBase的RowKey設計:https://www.cnblogs.com/zlzhoulei/p/5594773.html