hive按當天日期建立分割槽表 | 動態往日期分割槽插入資料
阿新 • • 發佈:2019-02-02
hive建立分割槽表,以當天日期(“2014-08-15”)作為分割槽依據,hql如下:
CREATE EXTERNAL TABLE IF NOT EXISTS product_sell( category_id BIGINT, province_id BIGINT, product_id BIGINT, price DOUBLE, sell_num BIGINT ) PARTITIONED BY (ds string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
然後以日期作為分割槽依據,插入資料,shell指令碼如下:
#!/bin/bash source /etc/profile; yesterday=$(date -d '-1 day' '+%Y-%m-%d') lastweek=$(date -d '-1 week' '+%Y-%m-%d') /usr/local/cloud/hive/bin/hive<<EOF INSERT OVERWRITE TABLE product_sell PARTITION (ds='$yesterday') select a.category_id, b.good_receiver_province_id as province_id, a.id as product_id, (b.sell_amount/b.sell_num) as price, b.sell_num from product a join (select si.product_id, s.good_receiver_province_id, sum(si.order_item_amount) sell_amount, sum(si.order_item_num) sell_num from so_item si join so s on (si.order_id=s.id) where si.is_gift=0 and si.is_hidden=0 and si.ds between '$lastweek' and '$yesterday' group by s.good_receiver_province_id, si.product_id) b on (a.id=b.product_id); EOF
這個部分難點在於hive中不知道如何呼叫形如shell日期變數的方式建立日期分割槽,mark一下!