Python進行access_log日誌解析2
阿新 • • 發佈：2018-11-17
本次的解析，是將accessLog日誌檔案解析出來，對於請求引數以map的形式存放在hive庫中。這樣可以保證日誌檔案的完整性和以後引數增加的可擴充套件性。
示例日誌檔案見上一篇： Python進行access_log日誌解析1-->https://blog.csdn.net/babyfish13/article/details/81027074
1、解析的python指令碼
/Users/nisj/PycharmProjects/BiDataProc/love/accessLogPhase2.py
/home/hadoop/nisj/xx/nisj/acclog_norm_data.txt
處理過程中 ，對日期時間在python中進行了相應的轉換；請求引數的格式也根據hive的map裝載要求進行了特殊處理； 另外在本指令碼的基礎上，是要進行資料檔案匯出的，待後續完善。
示例日誌檔案見上一篇： Python進行access_log日誌解析1-->https://blog.csdn.net/babyfish13/article/details/81027074
1、解析的python指令碼
/Users/nisj/PycharmProjects/BiDataProc/love/accessLogPhase2.py
2、解析出的日誌樣例# -*- coding=utf-8 -*- import re, time def readFileFromOs(filePath): with open('{filePath}'.format(filePath=filePath), 'r') as f: fileDatas = f.read().split('\n') PendingData_list = [] for pending_list in fileDatas: pending = re.split(' ',pending_list.replace('\n', '').replace('?', '?&').replace('&&', '&').replace('- - ','').replace(' - - ',' ')) PendingData_list.append(pending) for pdl in PendingData_list: if len(pdl) > 1: requestInfo = str(pdl[3]).replace('?&', ' ').split(' ') # requestPar='{\"' + requestInfo[2].replace('&', '\",\"').replace('=', '\":\"') + '\"}' requestPar=requestInfo[2].replace('&', ',').replace('=', ':') date_time = time.strptime(str(pdl[2]).replace(' +0800', ''), "[%d/%b/%Y:%H:%M:%S]") date_time_str = time.strftime("%Y-%m-%d %H:%M:%S", date_time) # print eval(requestPar) # print pdl[0], pdl[1], date_time_str, requestInfo[0], requestInfo[1], requestPar, requestInfo[3], pdl[4], pdl[5], pdl[6], pdl[7] print pdl[0] + '\t' + pdl[1] + '\t' + date_time_str + '\t' + requestInfo[0] + '\t' + requestInfo[1] + '\t' + requestPar + '\t' + requestInfo[3] + '\t' + pdl[4] + '\t' + pdl[5] + '\t' + pdl[6] + '\t' + pdl[7] readFileFromOs(filePath='/Users/nisj/Downloads/localhost_access_log.min.txt')
/home/hadoop/nisj/xx/nisj/acclog_norm_data.txt
3、hive建表及資料裝載112.123.103.141 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:212,_appVersion:5.0.3,_appkey:CSAndroid,_identifier:127793878,_imei:868981032402390,_loc:117.492657%2C30.653728,_locRadius:178.0,_locType:bd09ll,_sign:a7a3d52e7efe0256c9c67c9de78d8ae6,_t:1529690399659,_wsign:a360e9863060bcd77167ab096619115799316d00,codes:10006%2C10007%2C10009%2C10008,device_api_int:27,device_board:k71v1_64_bsp,device_brand:vivo,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:ff656d930ed6b0f859003ff011b13a20,device_devicename:PD1801,device_display:O11019+release-keys,device_ds:ff656d930ed6b0f859003ff011b13a20,device_es:38b8caf769c58d77a6c260ad52a892baa,device_finger:vivo%2FPD1801%2FPD1801%3A8.1.0%2FO11019%2F1527863420%3Auser%2Frelease-keys,device_fsa:bbc8151c9533667d4d61749470137a4b223,device_hardware:mt6771,device_host:compiler17,device_id:3d6b396903002f35,device_incremental:1527863420,device_mac:020000000000,device_manufacturer:vivo,device_model:vivo+X21i+A,device_product:PD1801,device_ram:6008250368,device_release:8.1.0,device_serial:55FQKFGAWS65SKNJ,device_tags:release-keys,device_type:user,device_user:compiler,device_versionid:O11019,os_version:27,screenHeight:2208,screenWidth:1080,token:1180bcd9a93cf9e8g2bd99ce8 HTTP/1.1 2048 0.002 Android ChushouTV/5.0.3 111.32.72.4 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:212,_appVersion:5.0.4,_appkey:CSAndroid,_identifier:8068017,_imei:868512031137750,_loc:117.27557%2C39.064607,_locRadius:40.0,_locType:bd09ll,_sign:1eb945b3563e3efb5efab476a879e8a5,_t:1529690399786,_wsign:88db3c161755e3abb7947661b127cb95d1d46813,_xappVersion:10601,codes:10006%2C10007%2C10009%2C10008,device_api_int:25,device_board:msm8953,device_brand:vivo,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:c8f51209714a3d702b07959b3f0515f1,device_devicename:PD1708,device_display:N2G47H+release-keys,device_ds:c8f51209714a3d702b07959b3f0515f1,device_es:34f41d329562eb6dafce10632af7871ea,device_finger:vivo%2FPD1708%2FPD1708%3A7.1.2%2FN2G47H%2Fcompil05152054%3Auser%2Frelease-keys,device_fsa:a696445f2b9a4129ad4d5fcd3f497a1b286,device_hardware:qcom,device_host:compiler028,device_id:276865e6a5b67878,device_incremental:eng.compil.20180515.205412,device_mac:020000000000,device_manufacturer:vivo,device_model:vivo+Y79A,device_product:PD1708,device_ram:3763171328,device_release:7.1.2,device_serial:4415da1e,device_tags:release-keys,device_type:user,device_user:compiler,device_versionid:N2G47H,os_version:25,screenHeight:1356,screenWidth:720,token:5c5ad6639c0aca6ag51e8336a HTTP/1.1 200 48 0.001 Android ChushouTV/5.0.4 182.138.212.243 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/feedback.htm advertTime:1529678125541,advertPrimaryKey:1529678125541326,opType:1,advertId:1028,positionId:22,scheduleId:118,marketId:1,_appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:0F564805-6266-4D15-B175-499706037464,_idfa:1F907805-77EE-4E2D-8669-A5A8C8A08A86,_loc:104.057351%2C30.710412,_locRadius:10.000000,_locType:bd09ll,_sign:5c0e290cf5c1f33fb6a053dbaa9376eb,_t:1529690399970,_wsign:09980838df4f5563fde1cf72f3732df2bf5e01db,advertId:1028,advertPrimaryKey:1529678125541326,advertTime:1529678125541,device_cs:6354a91b4a0d0579b5ab6b3a85e245cb,device_ds:0F564805-6266-4D15-B175-499706037464,device_es:hb3079bc8ba8e08328fd4f10bd36cbeb3,device_model:iPhone10%2C3,device_release:11.4,device_user:%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone,marketId:1,opType:1,positionId:22,scheduleId:118,token:71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.000 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526 182.138.212.243 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/feedback.htm advertTime:1529690399951,advertPrimaryKey:1529690399951487,opType:1,advertId:-1,positionId:4,scheduleId:79,marketId:2,_appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:0F564805-6266-4D15-B175-499706037464,_idfa:1F907805-77EE-4E2D-8669-A5A8C8A08A86,_loc:104.057351%2C30.710412,_locRadius:10.000000,_locType:bd09ll,_sign:4530f1951a3b71b663172e43dee6100e,_t:1529690400089,_wsign:5be3669bf8dba6b131f970dcef69f8910aee21bd,advertId:-1,advertPrimaryKey:1529690399951487,advertTime:1529690399951,device_cs:6354a91b4a0d0579b5ab6b3a85e245cb,device_ds:0F564805-6266-4D15-B175-499706037464,device_es:hb3079bc8ba8e08328fd4f10bd36cbeb3,device_model:iPhone10%2C3,device_release:11.4,device_user:%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone,marketId:2,opType:1,positionId:4,scheduleId:79,token:71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526 27.197.216.94 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:7D14547D-25BD-4FE4-8219-F9402CD96D7C,_idfa:968AF15C-9FBB-45CC-94FC-A7F6C7AE74D1,_locRadius:10.000000,_locType:bd09ll,_sign:e127a079c980b8877c4c2ac4806aabe3,_t:1529690399894,_wsign:8e200b73dacd2d340af4333bd9e6f4c3ba08b746,codes:20011,device_cs:e427a59ba4f043cc7b1f0ac8ab83d84e,device_ds:7D14547D-25BD-4FE4-8219-F9402CD96D7C,device_es:h531740cc6561647845c64f68921c1a42,device_model:iPhone9%2C1,device_release:11.2.6,device_user:iPhone%20%282%29,screenHeight:667,screenWidth:375,token:71a0b317df9d5861g46cd6161 HTTP/1.1 200 48 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D100 iOS ChushouTv/5.0.2.26526 121.207.60.108 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/feedback.htm _appSource:568,_appVersion:5.0.2.20594,_appkey:CSAndroid,_identifier:103726604,_imei:864034032232252,_loc:4.9E-324%2C4.9E-324,_locRadius:0.0,_locType:bd09ll,_sign:8a78a3f580ef638c5347ea597bc329f0,_t:1529690398293,_wsign:5b30ce86f84974e5c2be5d31fc3f927a8d51bbd3,advertId:-1,advertPrimaryKey:1529690395509908,advertTime:1529690395509,device_api_int:24,device_board:STF,device_brand:HONOR,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:23c75271ec687bebf5e3a622275c3d52,device_devicename:HWSTF,device_display:STF-TL10C01B155,device_ds:23c75271ec687bebf5e3a622275c3d52,device_es:381f3fc7874745e9b5ff990eee2984988,device_finger:HONOR%2FSTF-TL10%2FHWSTF%3A7.0%2FHUAWEISTF-TL10%2FC01B155%3Auser%2Frelease-keys,device_fsa:706bd931f1766b5c11f5a749dd08a3d42d4,device_hardware:hi3660,device_host:LFG1000749995,device_id:d97b1fcabc2ec44b,device_incremental:C01B155,device_mac:020000000000,device_manufacturer:HUAWEI,device_model:STF-TL10,device_product:STF-TL10,device_ram:6009204736,device_release:7.0,device_serial:ERMDU17719004449,device_tags:release-keys,device_type:user,device_user:test,device_versionid:HUAWEISTF-TL10,marketId:2,opType:6,os_version:24,positionId:3,scheduleId:78 HTTP/1.1 200 33 0.001 Android ChushouTV/5.0.2.20594 119.85.20.4 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:209,_appVersion:5.0.4,_appkey:CSAndroid,_identifier:2011225951,_imei:A1000049D9D9C8,_loc:108.308172%2C31.350436,_locRadius:47.356655,_locType:bd09ll,_sign:7f8c139fe129bdd22ecef2c62a0aea9a,_t:1529690399607,_wsign:4c6f92bce852cc29d359672cb8cf514d072a00a6,_xappVersion:10601,codes:10003,device_api_int:21,device_board:msm8916,device_brand:vivo,device_codename:REL,device_cpuabi:armeabi-v7a,device_cpuabi2:armeabi,device_cs:0e71fbf721118360ddcc9dd370fdea26,device_devicename:PD1421V,device_display:LRX22G+release-keys,device_ds:0e71fbf721118360ddcc9dd370fdea26,device_es:310f678fed9bfe961768042053abf89bd,device_finger:vivo%2FPD1421V%2FPD1421V%3A5.0.2%2FLRX22G%2Fcompiler08021524%3Auser%2Frelease-keys,device_fsa:c2710e37b759813a7e4608914c59d7b62e3,device_hardware:qcom,device_host:compiler033,device_id:91fc862c745108ae,device_incremental:eng.compiler.20160802,device_mac:3ca348953c68,device_manufacturer:vivo,device_model:vivo+X5Pro+V,device_product:PD1421V,device_ram:2003296256,device_release:5.0.2,device_serial:440bdedf,device_tags:release-keys,device_type:user,device_user:compiler,device_versionid:LRX22G,os_version:21,screenHeight:1920,screenWidth:1080,token:8d48df74439a6e63g522eeb9d HTTP/1.1 2048 0.001 Android ChushouTV/5.0.4 117.136.79.90 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:212,_appVersion:5.0.3,_appkey:CSAndroid,_identifier:897957993,_imei:867366030842857,_loc:113.319278%2C22.732977,_locRadius:124.0,_locType:bd09ll,_sign:2c4e728b5def29c7c9f8b6ac174dd7a1,_t:1529690399851,_wsign:acfe6651777e0cab2f56f518da0d4b0f78ade373,codes:10006%2C10007%2C10009%2C10008,device_api_int:25,device_board:sdm660,device_brand:vivo,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:cc34e938005b5e24815594d717049556,device_devicename:PD1709,device_display:NMF26X+release-keys,device_ds:cc34e938005b5e24815594d717049556,device_es:3823b17c729b058f9f0874c8b5e000bbc,device_finger:vivo%2FPD1709%2FPD1709%3A7.1.1%2FNMF26X%2Fcompil05151836%3Auser%2Frelease-keys,device_fsa:284399fe8a67b15f8fb48b4d4fb9b3ec288,device_hardware:qcom,device_host:compiler031,device_id:d985e31284298e19,device_incremental:eng.compil.20180515.183659,device_mac:020000000000,device_manufacturer:vivo,device_model:vivo+X20,device_product:PD1709,device_ram:3915128832,device_release:7.1.1,device_serial:9ba6c868,device_tags:release-keys,device_type:user,device_user:compiler,device_versionid:NMF26X,os_version:25,screenHeight:2034,screenWidth:1080,token:b5256f7df7f14d93g490cd76d HTTP/1.1 2048 0.002 Android ChushouTV/5.0.3 111.3.34.46 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:8A383EC6-BAF0-49A0-B71F-6DD9A478B462,_idfa:89BA8D0B-99A9-4B02-8DEF-685957C2CFAC,_loc:120.584361%2C30.645005,_locRadius:10.000000,_locType:bd09ll,_sign:317dc2e17fb338b55109547d0a617ffe,_t:1529690399965,_wsign:1cb470802b4abee1172c1a498ac7e658a3ac6f6a,codes:20006%2C20007,device_cs:6b9ed8e460a3a427ef3df1489bcd6b4f,device_ds:8A383EC6-BAF0-49A0-B71F-6DD9A478B462,device_es:h26261c09d067d22a3aac13df1bda725b,device_model:iPad6%2C11,device_release:11.4,device_user:iPad,screenHeight:480,screenWidth:320,token:641381dbfca5d674g49280a74 HTTP/1.1 200 48 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526 113.247.80.43 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:313,_appVersion:5.0.3,_appkey:CSAndroid,_identifier:291347367,_imei:867306030507209,_loc:112.409116%2C28.114568,_locRadius:1000.0,_locType:bd09ll,_sign:4ef9665b3ccb7265077052fa82c67591,_t:1529690399538,_wsign:069c0ee36a7c93425273a093b372c0c0c43d0066,codes:10003,device_api_int:25,device_board:msm8953,device_brand:xiaomi,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:f72a7db9bc3572a901e009f269b053f7,device_devicename:tiffany,device_display:QL1515-tiffany-build-20180525212811,device_ds:f72a7db9bc3572a901e009f269b053f7,device_es:3dd3e4e4b6c8fe90fa02f71481ad4a302,device_finger:xiaomi%2Ftiffany%2Ftiffany%3A7.1.2%2FN2G47H%2FV9.6.1.0.NDBCNFD%3Auser%2Frelease-keys,device_fsa:11ba46a58786fa35de03186112250a452b1,device_hardware:qcom,device_host:c3-miui-ota-bd67.bj,device_id:e1f871c0308c621e,device_incremental:V9.6.1.0.NDBCNFD,device_mac:020000000000,device_manufacturer:Xiaomi,device_model:MI+5X,device_product:tiffany,device_ram:3772723200,device_release:7.1.2,device_serial:5eaa9af90604,device_tags:release-keys,device_type:user,device_user:builder,device_versionid:N2G47H,os_version:25,screenHeight:1920,screenWidth:1080,token:2720d4f66ff05501g50461d01 HTTP/1.1 2048 0.001 Android ChushouTV/5.0.3 116.25.40.54 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:8F092F29-508A-47D8-A0A6-CACF49DC0020,_idfa:E4B17751-B2F5-438C-8666-68375D95A02F,_locRadius:10.000000,_locType:bd09ll,_sign:26bd34cc7ade33bd4b81d8d3cfaf8196,_t:1529690397989,_wsign:0fcecfc7576eb5b1cfd00839aef4f55d7e79d917,codes:20009%2C20008,device_cs:37aa9bbb6251aef5dc09896aacc3c575,device_ds:8F092F29-508A-47D8-A0A6-CACF49DC0020,device_es:h54a988e7a71fdfd6f16f508d46be4bd3,device_model:iPad2%2C2,device_release:9.3.5,device_user:iPad,screenHeight:480,screenWidth:320,token:e02a43b80262bb9fg5046b361 HTTP/1.1 200 48 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13G36 iOS ChushouTv/5.0.2.26526 182.138.212.243 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/feedback.htm advertTime:1529690399769,advertPrimaryKey:1529690399769488,opType:1,advertId:1061,positionId:22,scheduleId:114,marketId:1,_appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:0F564805-6266-4D15-B175-499706037464,_idfa:1F907805-77EE-4E2D-8669-A5A8C8A08A86,_loc:104.057351%2C30.710412,_locRadius:10.000000,_locType:bd09ll,_sign:e7cc8e2fb8bc6cc5ee79de1689cb1d01,_t:1529690400141,_wsign:ff9d5d7cba5678e593f81f43e18d76dc9f8c2afd,advertId:1061,advertPrimaryKey:1529690399769488,advertTime:1529690399769,device_cs:6354a91b4a0d0579b5ab6b3a85e245cb,device_ds:0F564805-6266-4D15-B175-499706037464,device_es:hb3079bc8ba8e08328fd4f10bd36cbeb3,device_model:iPhone10%2C3,device_release:11.4,device_user:%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone,marketId:1,opType:1,positionId:22,scheduleId:114,token:71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526 106.5.27.196 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:FAFBF3E0-3FAD-475C-A05D-FC2E5FCC3C3B,_idfa:950C0768-CF12-49D9-9288-8EBB3493AB3A,_loc:115.015795%2C26.844929,_locRadius:10.000000,_locType:bd09ll,_sign:4dc73501e3bfb8625ceeffa967fb552e,_t:1529690399393,_wsign:fb436d3166b65c9e7afee13c3c27a0373b52ec60,codes:20006%2C20007,device_cs:e9347fc7e65121d0dc4efd7d397e740a,device_ds:FAFBF3E0-3FAD-475C-A05D-FC2E5FCC3C3B,device_es:h57f5e8bbed3e3f57f974b123c5fbc0f6,device_model:iPhone9%2C2,device_release:11.3,device_user:iPhone%207%20Plus,screenHeight:736,screenWidth:414,token:e458c3ee44aa903fg9cec1 HTTP/1.1 200 48 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E216 iOS ChushouTv/5.0.2.26526 118.212.200.198 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appSource:311,_appVersion:5.0.3,_appkey:CSAndroid,_identifier:660838218,_imei:864320036032859,_loc:115.809154%2C28.696803,_locRadius:353.0,_locType:bd09ll,_sign:f0e9f2be33428d97ce8a7bea1d64a66e,_t:1529690399577,_wsign:1e1ee6b89eecb272e79870bc6c1567e4bfda299f,codes:10006%2C10007%2C10009%2C10008,device_api_int:23,device_board:msm8953,device_brand:OPPO,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:65f7826bc3e1635d0a6c7a9e19a84cf7,device_devicename:R9sk,device_display:R9sk_11_A.25_180418,device_ds:65f7826bc3e1635d0a6c7a9e19a84cf7,device_es:3b1ebe33f43ea5d253a7803dabad63b59,device_finger:OPPO%2FR9sk%2FR9sk%3A6.0.1%2FMMB29M%2F1522231161%3Auser%2Frelease-keys,device_fsa:21f28384a617e0945a8cdf4abf4794602b4,device_hardware:qcom,device_host:ubuntu-121-139,device_id:f6762dec1ae6601b,device_incremental:eng.root.20180418.162155,device_mac:020000000000,device_manufacturer:OPPO,device_model:OPPO+R9sk,device_product:R9sk,device_ram:3769978880,device_release:6.0.1,device_serial:bb30aa3d,device_tags:dev-keys,device_type:user,device_user:root,device_versionid:MMB29M,os_version:23,screenHeight:1920,screenWidth:1080,token:57c77ca2e66f0dbg51f49adb HTTP/1.1 200 48 0.001 Android ChushouTV/5.0.3 117.136.79.127 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/feedback.htm _appSource:306,_appVersion:5.0.4,_appkey:CSAndroid,_identifier:477494778,_imei:862266034478358,_loc:113.803454%2C22.746902,_locRadius:102.23267,_locType:bd09ll,_sign:e8692b993c7fe4f078961472ee4038a7,_t:1529690399349,_wsign:f9b56b7b5c7f31624bc6b075c35b01ece9ae88a6,_xappVersion:10601,advertId:-1,advertPrimaryKey:1529690398001848,advertTime:1529690398001,device_api_int:24,device_board:EVA-TL00,device_brand:HUAWEI,device_codename:REL,device_cpuabi:arm64-v8a,device_cpuabi2:armeabi-v7a,device_cs:839b394e24599fcc7cc8c0dcf4cfbf83,device_devicename:HWEVA,device_display:EVA-TL00C01B399SP17,device_ds:61ecb3763843971d54c8ab67b84eae48,device_es:3150dcf4fe3d8af1ac900b4acf3ebce6f,device_finger:HUAWEI%2FEVA-TL00%2FHWEVA%3A7.0%2FHUAWEIEVA-TL00%2FC01B399SP17%3Auser%2Frelease-keys,device_fsa:8621a630cad8a9646dd01e23bfb81446244,device_hardware:hi3650,device_host:huawei-105-38,device_id:bbba118f552c9f9c,device_incremental:C01B399SP17,device_mac:020000000000,device_manufacturer:HUAWEI,device_model:EVA-TL00,device_product:EVA-TL00,device_ram:2915397632,device_release:7.0,device_serial:B2T0216820010970,device_tags:release-keys,device_type:user,device_user:test,device_versionid:HUAWEIEVA-TL00,marketId:2,opType:6,os_version:24,positionId:3,scheduleId:78,token:49cf8aeee36b58bfg5044d3bf HTTP/1.1 200 33 0.001 Android ChushouTV/5.0.4 27.197.216.94 172.16.1.72 2018-06-23 02:00:00 GET /api/advert/getList.htm _appEnv:1,_appSource:811,_appVersion:5.0.2.26526,_appkey:CSIos,_identifier:7D14547D-25BD-4FE4-8219-F9402CD96D7C,_idfa:968AF15C-9FBB-45CC-94FC-A7F6C7AE74D1,_locRadius:10.000000,_locType:bd09ll,_sign:4d4eec8156a40ea1162111f6b72602c2,_t:1529690400323,_wsign:03e4805d9b0e516911121c23f387171544e22654,codes:20011,device_cs:e427a59ba4f043cc7b1f0ac8ab83d84e,device_ds:7D14547D-25BD-4FE4-8219-F9402CD96D7C,device_es:h531740cc6561647845c64f68921c1a42,device_model:iPhone9%2C1,device_release:11.2.6,device_user:iPhone%20%282%29,screenHeight:667,screenWidth:375,token:71a0b317df9d5861g46cd6161 HTTP/1.1 200 48 0.001 Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D100 iOS ChushouTv/5.0.2.26526
4、裝載後的資料檢視和使用drop table if exists xxxxx_accesslog_map_data; create table xxxxx_accesslog_map_data( ip1 string, ip2 string, date_time TIMESTAMP, request_mod string, request_uri string, request_par map<string, int>, http_version string, oth1 string, oth2 string, oth3 string, oth4 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':'; LOAD DATA LOCAL INPATH '/home/hadoop/nisj/xx/nisj/acclog_norm_data.txt' overwrite INTO TABLE xxxxx_accesslog_map_data;
select * from xxxxx_accesslog_map_data;
select date_time,request_par['_appSource'],request_par['_appVersion'],request_par['scheduleId'],request_par['_identifier'] from xxxxx_accesslog_map_data;處理過程中 ，對日期時間在python中進行了相應的轉換；請求引數的格式也根據hive的map裝載要求進行了特殊處理； 另外在本指令碼的基礎上，是要進行資料檔案匯出的，待後續完善。