Hadoop示例程序之——WordCount

尝试一下Hadoop的Demo,熟悉下相关的MapReducer的计算框架

1、创建Mapper类

package hadoop.mr;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;
import java.util.StringTokenizer;

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

    private Text word = new Text();

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String wordline = value.toString();

        StringTokenizer st = new StringTokenizer(wordline);
        while (st.hasMoreTokens()) {
            word.set(st.nextToken());
            context.write(word, new IntWritable(1));
        }
    }
}

2、创建Reducer类

package hadoop.mr;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int count = 0;
        for (IntWritable value : values) {
            count += value.get();
        }
        context.write(key, new IntWritable(count));
    }
}

3、配置相关的Job作业

package hadoop.job;

import hadoop.mr.WordCountMapper;
import hadoop.mr.WordCountReducer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import java.io.IOException;

public class WordCount {
    public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
        Configuration config = new Configuration();
        Job job = new Job(config);
        job.setJarByClass(WordCount.class);
        job.setJobName("wordCount-Job");
        job.setMapperClass(WordCountMapper.class);
        job.setReducerClass(WordCountReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        FileInputFormat.addInputPath((JobConf) job.getConfiguration(), new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.waitForCompletion(true);
    }
}

4、引入相关依赖

由于我这边使用的是ide编辑器来进行编码的,并且是创建的相关的maven工程进行项目的管理,因此需要引入一下的依赖

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.5.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.5.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.5.2</version>
        </dependency>
    </dependencies>

5、打成jar包

<groupId>FirstTry</groupId>
    <artifactId>FirstTry</artifactId>
    <version>1.0-SNAPSHOT</version>

6、由于使用hdfs进行文件的上传

  相关命令如下:

        hadoop fs -mkdir hdfs://localhost:9000/user/input  --创建Hadoop存放文件的路径

        hadoop fs -put FileTest1.txt hdfs://localhost:9000/user/input --上传文件至Hdfs 具体的相关原理后期再讲

        hadoop fs -ls hdfs://localhost:9000/user/inout --查看hdfs中相关上传的文件

 7、运行jar包,执行相关的作业

   相关命令如下:

        1、进入到相关的jar包所在路径下 此处我的jar包应该是打成FirstTry.jar,因此进入到FirstTry.jar的路径下 执行以下相关命令

          hadoop jar FirstTry.jar hdfs://localhost:9000/user/input hdfs://localhost:9000/user/output

          如果出现提示:Exception in thread "main" java.io.IOException: Mkdirs failed to create ..... 需要将jar包中的META-INF/LICENSE文      件夹删除,找到相关的jar  -shift+delete 删除相关的目录

      2、由于我的jar包是FirstTry.jar 而我的main函数的class是WordCount,因此需要制定下相关的class,因此相关的Hadoop命令改成这样,

        hadoop jar FirstTry.jar hadoop.job.WordCount hdfs://locahost:9000/user/input hdfs://localhost:9000/user/output

     然后你会看见下面的流程:

          

  说明我们的第一个demo执行成功。

 8、查询相关的结果

      hadoop fs -ls hdfs://localhost:9000/user/output 查看输出文件part-r-00000

      查看文件内容  hadoop fs -cat hdfs://localhost:9000/user/output/part-r-00000

 9、计算单词数量成功 

     第一个小程序OK啦。

转载于:https://my.oschina.net/guanhe/blog/1865787

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Hadoop的第一个WordCount程序是一个非常简单的程序,它的主要目的是计算一个文本文件中每个单词出现的次数。 以下是一个基本的WordCount程序: 1. 创建一个Java项目并导入Hadoop库。 2. 创建一个Java类并实现以下Mapper和Reducer: Mapper类: ```java import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class WordCountMapper extends Mapper<LongWritable, Text, Text, LongWritable> { private final static LongWritable one = new LongWritable(1); private Text word = new Text(); @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] words = line.split(" "); for (String w : words) { word.set(w); context.write(word, one); } } } ``` Reducer类: ```java import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class WordCountReducer extends Reducer<Text, LongWritable, Text, LongWritable> { @Override public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException { long sum = 0; for (LongWritable val : values) { sum += val.get(); } context.write(key, new LongWritable(sum)); } } ``` 3. 在应用程序的main()方法中,创建一个Job并设置Mapper和Reducer类: ```java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(WordCountMapper.class); job.setCombinerClass(WordCountReducer.class); job.setReducerClass(WordCountReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } ``` 4. 在命令行中运行程序,并指定输入和输出文件路径: ```bash hadoop jar WordCount.jar WordCount /input /output ``` 其中,/input是输入文件路径,/output是输出文件路径。 这是最基本的WordCount程序,你可以根据需要进行修改和扩展。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值