Map Reduce資料清洗及Hive資料庫操作
阿新 • • 發佈:2020-12-28
1、資料清洗:按照進行資料清洗,並將清洗後的資料匯入hive資料庫中。
兩階段資料清洗:
(1)第一階段:把需要的資訊從原始日誌中提取出來
ip: 199.30.25.88
time: 10/Nov/2016:00:01:03 +0800
traffic: 62
文章:article/11325
視訊:video/3235
(2)第二階段:根據提取出來的資訊做精細化操作
ip--->城市city(IP)
date--> time:2016-11-10 00:01:03
day: 10
traffic:62
type:article/video
id:11325
(3)hive資料庫表結構:
createtabledata( ipstring, timestring, daystring, trafficbigint,
typestring, idstring )
由於很長時間沒有進行程式設計,今天只能完成 第一階段的檔案資料處理
package com.test.dao; import java.io.IOException; import java.util.ArrayList; import java.util.List; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; public class test1{ public static List<String> ips=new ArrayList<String>(); public static List<String> times=new ArrayList<String>(); public static List<String> traffic=new ArrayList<String>(); public static List<String> wen=new ArrayList<String>(); public static List<String> shi=new ArrayList<String>(); public static class Map extends Mapper<Object , Text , Text,Text>{ private static Text Name =new Text(); private static Text num=new Text(); public void map(Object key,Text value,Context context) throws IOException, InterruptedException{ String line=value.toString(); String arr[]=line.split(","); Name.set(arr[0]); num.set(arr[0]); context.write(Name,num); } } public static class Reduce extends Reducer< Text, Text,Text, Text>{ private static Text result= new Text(); int i=0; public void reduce(Text key,Iterable<Text> values,Context context) throws IOException, InterruptedException{ for(Text val:values){ context.write(key, val); ips.add(val.toString()); } } } public static int run()throws IOException, ClassNotFoundException, InterruptedException { Configuration conf=new Configuration(); conf.set("fs.defaultFS", "hdfs://localhost:9000"); FileSystem fs =FileSystem.get(conf); Job job =new Job(conf,"OneSort"); job.setJarByClass(test1.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); Path in=new Path("hdfs://localhost:9000/test2/in/result.txt"); Path out=new Path("hdfs://localhost:9000/test2/out/ip/1"); FileInputFormat.addInputPath(job,in); fs.delete(out,true); FileOutputFormat.setOutputPath(job,out); return(job.waitForCompletion(true) ? 0 : 1); } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException{ run(); } } }