1. 程式人生 > >hive → phoenix資料匯入匯出方案

hive → phoenix資料匯入匯出方案

主要是使用Phoenix Storage Handler for Apache Hive Feature Installation,優勢是phoenix自帶的feature,支援力度比較好
phoenix主頁上相關的介紹[https://phoenix.apache.org/hive_storage_handler.html ]

1. 調研的環境

phoenix 4.8
hive 1.2.1
測試環境共有兩個叢集。 hbase和hadoop

2. 操作過程

2.1 需要開啟的配置項

hbase-site.xml 叢集和客戶端都需要開啟這兩項配置

<property
>
<name>phoenix.schema.isNamespaceMappingEnabled</name> <value>true</value> </property> <property> <name>phoenix.schema.mapSystemTablesToNamespace</name> <value>false</value> </property>

hive-site.xml

<property>
    <name
>
hive.aux.jars.path</name> <value>file:///home/hadoop/apache-hive-1.2.1-bin/lib/phoenix-4.8.0-HBase-0.98-hive.jar</value> </property>

hive-env.sh

export HIVE_AUX_JARS_PATH=/home/hadoop/apache-hive-1.2.1-bin/lib/phoenix-4.8.0-HBase-0.98-hive.jar

2.2 建立phoenix schema

create schema if
not exists ABC;

2.3 建立hive內部表,關聯到phoenix

create table g_order_phoenix 
(
 order_id bigint,
 driver_id bigint,
 driver_phone string,
 passenger_id bigint
 )
STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'   TBLPROPERTIES
 (
"phoenix.table.name" = "ABC.G_ORDER_PHOENIX",
"phoenix.zookeeper.quorum" = "10.93.18.30,10.93.18.31,10.93.18.32",
"phoenix.zookeeper.znode.parent" = "/hbase",
"phoenix.zookeeper.client.port" = "2181",
"phoenix.rowkeys" = "order_id,driver_id",
"phoenix.column.mapping" = "order_id:order_id, driver_id:driver_id, driver_phone:driver_phone, passenger_id:passenger_id",
"phoenix.table.options" = "SALT_BUCKETS=10, DATA_BLOCK_ENCODING='DIFF'"
);

###2.4 建立待匯出資料的hive表,這部分資料在hadoop HDFS上

CREATE TABLE `g_order`
(
   `order_id` bigint COMMENT '訂單id', 
   `driver_id` bigint COMMENT '司機id,司機搶單前該值為0', 
   `driver_phone` string COMMENT '司機電話', 
   `passenger_id` bigint COMMENT '乘客id'
 );

2.5 插入資料

insert into table g_order 
select order_id, driver_id,driver_phone,passenger_id from g_order_phoenix where driver_id!=0 and order_id!=0;