python读取hive数据库mapreduce_Map Reduce数据清洗及Hive数据库操作

1、 数据清洗：按照进行数据清洗，并将清洗后的数据导入hive数据库中。

两阶段数据清洗：

(1)第一阶段：把需要的信息从原始日志中提取出来

ip:    199.30.25.88

time:  10/Nov/2016:00:01:03 +0800

traffic:  62

文章： article/11325

视频： video/3235

(2)第二阶段：根据提取出来的信息做精细化操作

ip--->城市 city(IP)

date--> time:2016-11-10 00:01:03

day: 10

traffic:62

type:article/video

id:11325

(3)hive数据库表结构:

create table data(  ip string,  time string , day string, traffic bigint,

type string, id   string )

packagecom.test.dao;importjava.io.IOException;importjava.util.ArrayList;importjava.util.List;importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.FileSystem;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.mapreduce.Job;importorg.apache.hadoop.mapreduce.Mapper;importorg.apache.hadoop.mapreduce.Reducer;importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;importorg.apache.hadoop.mapreduce.lib.input.TextInputFormat;importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;importorg.apache.hadoop.mapreduce.lib.output.TextOutputFormat;public classtest1{public static List ips=new ArrayList();public static List times=new ArrayList();public static List traffic=new ArrayList();public static List wen=new ArrayList();public static List shi=new ArrayList();public static class Map extends Mapper{private static Text Name =newText();private static Text num=newText();public void map(Object key,Text value,Context context) throwsIOException, InterruptedException{

String line=value.toString();

String arr[]=line.split(",");

Name.set(arr[0]);

num.set(arr[0]);

context.write(Name,num);

}

}public static class Reduce extends Reducer< Text, Text,Text, Text>{private static Text result= newText();int i=0;public void reduce(Text key,Iterable values,Context context) throwsIOException, InterruptedException{for(Text val:values){

context.write(key, val);

ips.add(val.toString());

}

}

}public static int run()throwsIOException, ClassNotFoundException, InterruptedException

{

Configuration conf=newConfiguration();

conf.set("fs.defaultFS", "hdfs://localhost:9000");

FileSystem fs=FileSystem.get(conf);

Job job=new Job(conf,"OneSort");

job.setJarByClass(test1.class);

job.setMapperClass(Map.class);

job.setReducerClass(Reduce.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(Text.class);

job.setInputFormatClass(TextInputFormat.class);

job.setOutputFormatClass(TextOutputFormat.class);

Path in=new Path("hdfs://localhost:9000/test2/in/result.txt");

Path out=new Path("hdfs://localhost:9000/test2/out/ip/1");

FileInputFormat.addInputPath(job,in);

fs.delete(out,true);

FileOutputFormat.setOutputPath(job,out);return(job.waitForCompletion(true) ? 0 : 1);

}public static void main(String[] args) throwsIOException, ClassNotFoundException, InterruptedException{

run();

}

}

}

2、数据处理：

·统计最受欢迎的视频/文章的Top10访问次数 (video/article)

·按照地市统计最受欢迎的Top10课程 (ip)

·按照流量统计最受欢迎的Top10课程 (traffic)

3、数据可视化：将统计结果倒入MySql数据库中，通过图形化展示的方式展现出来。