專案實戰從0到1之hive（44）大資料專案之電商數倉（使用者行為資料）（十二）

阿新 • • 發佈：2021-01-21

第17章需求六：流失使用者數

流失使用者：最近7天未登入我們稱之為流失使用者

17.1 DWS層

使用日活明細表dws_uv_detail_day作為DWS層資料

17.2 ADS層

1）建表語句

drop table if exists ads_wastage_count;
create external table ads_wastage_count(
   `dt` string COMMENT '統計日期',
   `wastage_count` bigint COMMENT '流失裝置數'
)
row format delimited fields terminated by '\t'
location '/warehouse/gmall/ads/ads_wastage_count';

2）匯入2019-02-20資料

insert into table ads_wastage_count
select
   '2019-02-20',
   count(*)
from 
(
  select mid_id
from dws_uv_detail_day
  group by mid_id
  having max(dt)<=date_add('2019-02-20',-7)
)t1;

17.3 編寫指令碼

1）建立指令碼

[kgg@hadoop102 bin]$ vim ads_wastage_log.sh
   在指令碼中編寫如下內容
#!/bin/bash

if [ -n "$1" ];then
   do_date=$1
else
   do_date=`date -d "-1 day" +%F`
fi

hive=/opt/module/hive/bin/hive
APP=gmall

echo "-----------匯入日期$do_date-----------"

sql="
insert into table "$APP".ads_wastage_count
select
   '$do_date',
   count(*)
from 
(
  select mid_id
  from "$APP".dws_uv_detail_day
  group by mid_id
  having max(dt)<=date_add('$do_date',-7)
)t1;"

$hive -e "$sql"

2）增加指令碼執行許可權

chmod 777 ads_wastage_log.sh

3）指令碼使用

ads_wastage_log.sh 2019-02-20

4）查詢結果

 select * from ads_wastage_count;

5）指令碼執行時間

企業開發中一般在每日凌晨30分~1點

第18章需求七：最近連續三週活躍使用者數

最近3周連續活躍的使用者：通常是週一對前3周的資料做統計，該資料一週計算一次。

18.1 DWS層

使用周活明細表dws_uv_detail_wk作為DWS層資料

18.2 ADS層

1）建表語句

drop table if exists ads_continuity_wk_count;
create external table ads_continuity_wk_count( 
   `dt` string COMMENT '統計日期,一般用結束週週日日期,如果每天計算一次,可用當天日期',
   `wk_dt` string COMMENT '持續時間',
   `continuity_count` bigint
) 
row format delimited fields terminated by '\t'
location '/warehouse/gmall/ads/ads_continuity_wk_count';

2）匯入2019-02-20所在周的資料

insert into table ads_continuity_wk_count
select
   '2019-02-20',
   concat(date_add(next_day('2019-02-20','MO'),-7*3),'_',date_add(next_day('2019-02-20','MO'),-1)),
   count(*)
from 
(
  select mid_id
  from dws_uv_detail_wk
  where wk_dt>=concat(date_add(next_day('2019-02-20','MO'),-7*3),'_',date_add(next_day('2019-02-20','MO'),-7*2-1)) 
  and wk_dt<=concat(date_add(next_day('2019-02-20','MO'),-7),'_',date_add(next_day('2019-02-20','MO'),-1))
  group by mid_id
  having count(*)=3
)t1;

3）查詢

select * from ads_continuity_wk_count;

18.3 編寫指令碼

1）建立指令碼

[kgg@hadoop102 bin]$ vim ads_continuity_wk_log.sh
   在指令碼中編寫如下內容
#!/bin/bash

if [ -n "$1" ];then
   do_date=$1
else
   do_date=`date -d "-1 day" +%F`
fi

hive=/opt/module/hive/bin/hive
APP=gmall

echo "-----------匯入日期$do_date-----------"

sql="
insert into table "$APP".ads_continuity_wk_count
select 
   '$do_date',
   concat(date_add(next_day('$do_date','MO'),-7*3),'_',date_add(next_day('$do_date','MO'),-1)),
   count(*)
from 
(
  select mid_id
  from "$APP".dws_uv_detail_wk
  where wk_dt>=concat(date_add(next_day('$do_date','MO'),-7*3),'_',date_add(next_day('$do_date','MO'),-7*2-1)) 
  and wk_dt<=concat(date_add(next_day('$do_date','MO'),-7),'_',date_add(next_day('$do_date','MO'),-1))
  group by mid_id
  having count(*)=3
)t1;"

$hive -e "$sql"

2）增加指令碼執行許可權

chmod 777 ads_continuity_wk_log.sh

3）指令碼使用

ads_continuity_wk_log.sh 2019-02-20

4）查詢結果

select * from ads_continuity_wk_count;

5）指令碼執行時間

企業開發中一般在每週一凌晨30分~1點

第19章需求八：最近七天內連續三天活躍使用者數

說明：最近7天內連續3天活躍使用者數

19.1 DWS層

使用日活明細表dws_uv_detail_day作為DWS層資料

19.2 ADS層

1）建表語句

drop table if exists ads_continuity_uv_count;
create external table ads_continuity_uv_count( 
   `dt` string COMMENT '統計日期',
   `wk_dt` string COMMENT '最近7天日期',
   `continuity_count` bigint
) COMMENT '連續活躍裝置數'
row format delimited fields terminated by '\t'
location '/warehouse/gmall/ads/ads_continuity_uv_count';

2）寫出匯入資料的SQL語句

insert into table ads_continuity_uv_count
select
  '2019-02-12',
   concat(date_add('2019-02-12',-6),'_','2019-02-12'),
  count(*)
from
(
  select mid_id
  from
   (
    select mid_id   
    from
     (
      select 
         mid_id,
         date_sub(dt,rank) date_dif
      from
       (
        select 
           mid_id,
           dt,
           rank() over(partition by mid_id order by dt) rank
        from dws_uv_detail_day
        where dt>=date_add('2019-02-12',-6) and dt<='2019-02-12'
       )t1
     )t2 
    group by mid_id,date_dif
    having count(*)>=3
   )t3 
  group by mid_id
)t4;

3）查詢

select * from ads_continuity_uv_count;

19.3 編寫指令碼

1）建立指令碼

[kgg@hadoop102 bin]$ vim ads_continuity_log.sh
   在指令碼中編寫如下內容
#!/bin/bash

if [ -n "$1" ];then
  do_date=$1
else
  do_date=`date -d "-1 day" +%F`
fi

hive=/opt/module/hive/bin/hive
APP=gmall

echo "-----------匯入日期$do_date-----------"

sql="
insert into table "$APP".ads_continuity_uv_count
select 
   '$do_date',
   concat(date_add('$do_date',-6),'_','$do_date') dt,
   count(*) 
from 
(
   select mid_id
   from
   (
     select mid_id
     from 
     (
       select
         mid_id,
         date_sub(dt,rank) date_diff
       from 
       (
         select 
           mid_id,
           dt,
           rank() over(partition by mid_id order by dt) rank
         from "$APP".dws_uv_detail_day
         where dt>=date_add('$do_date',-6) and dt<='$do_date'
       )t1
     )t2
     group by mid_id,date_diff
     having count(*)>=3
   )t3 
   group by mid_id
)t4;
"

$hive -e "$sql"