Day5----电商实战项目实现2
问题描述
根据电商日志文件,分析:
1 . 统计页面浏览量(每行记录就是一次浏览)
2 . 统计各个省份的浏览量 (需要解析IP)
3 . 日志的ETL操作(ETL:数据从来源端经过抽取(Extract)、转换(Transform)、加载(Load)至目的端的过程)
为什么要ETL:没有必要解析出所有数据,只需要解析出有价值的字段即可。本项目中需要解析出:ip、url、pageId(topicId对应的页面Id)、country、province、city
(2) 统计各个省份的浏览量 (需要解析IP)
- 将qqwry.dat导入到虚拟机,启动集群,利用idea编写代码,map和reduce具体关键代码用于处理日志文件 如下:
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
import java.util.Map;
/**
* 省份浏览量统计
*/
public class ProvinceStatApp {
public static void main(String[] args)throws Exception {
Configuration configuration =new Configuration();
Job job =Job.getInstance(configuration);
job.setJarByClass(ProvinceStatApp.class);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job,new Path(args[1]));
job.waitForCompletion(true);
}
static class MyMapper extends Mapper<LongWritable, Text,Text,LongWritable>{
private LongWritable ONE=new LongWritable(1);
private LogParser logParser;
@Override
protected void setup(Context context) throws IOException, InterruptedException {
logParser =new LogParser();
}
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String log =value.toString();
Map<String,String> info=logParser.parse(log);
String ip=info.get("ip");
if(StringUtils.isNotBlank(ip)){
IPParser.RegionInfo regionInfo= IPParser.getInstanc
- 将程序打成jar包,并上传结果至hdfs,查看网页最终结果如下: