实验简介:将java文件打包,在MapReduce环境中运行
实验环境:Hadoop1.2.1 JDK1.7
实验代码:
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
package com.qhy.wordcount; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.*; import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; /** * Created by Administrator on 2016/11/27. */ public class WordCount { public static class Map extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key , Text value , OutputCollector<Text,IntWritable> output , Reporter reporter) throws IOException{ String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while(tokenizer.hasMoreTokens()){ word.set(tokenizer.nextToken()); output.collect(word , one); } } } public static class Reduce extends MapReduceBase implements Reducer<Text,IntWritable,Text,IntWritable>{ public void reduce(Text key , Iterator<IntWritable> values , OutputCollector<Text,IntWritable> output , Reporter reporter) throws IOException{ int sum = 0; while(values.hasNext()){ sum += values.next().get(); } output.collect(key,new IntWritable(sum)); } } public static void main(String[] args) throws IOException{ JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setMapperClass(Map.class); conf.setReducerClass(Reduce.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf,new Path(args[0])); FileOutputFormat.setOutputPath(conf,new Path(args[1])); JobClient.runJob(conf); } }
这是一个简单的WordCount程序,想必大家都不陌生。
首先定义一些东西:hadoop的路径为:a/hadoop .java文件的地址为 b/WordCount.java
第一步:生成.class文件
javac -classpath a/hadoop/hadoop-core-1.2.1.jar -d 存放.class文件目录 b/WordCount
结束后在指定目录中可以看到三个.class文件(因为本人代码有所在包(com.qhy.wordcount),所以会存放在 目标目录/com/qhy/wordcout/文件夹下)
第二步:打包成jar文件
jar -cvf WordCount.jar -C 存放.class的文件夹路径/ .
WordCount.jar表示要打包成的jar名字;还有存放.class的路径,我这里这个路径指向com那个文件夹就行了;注意:最后有一个 “.”表示生成的jar文件存放在了当前目录。
第三步:上传输入文件到hdfs上
bin/hadoop dfs -mkdir wordcountInput bin/hadoop dfs -put 输入文件所在目录 wordcountInput
第四步:运行WordCount程序
bin/hadoop jar x/WordCount.jar com.qhy.wordcount.WordCount wordcountInput output
这这里,需要注意:第一个参数是jar文件的完整路径;第二个参数为运行的类,如果在某个包中,要把包的路径也写出来,如上所示;剩下两个参数一个是输入路径,一个输出路径
运行结果如下(通过网页查看的):