Hadoop大数据技术-词频计数

呆呆小板栗

于 2024-03-13 21:30:17 发布

阅读量415

点赞数 7

分类专栏： Hadoop Hadoop大数据技术大数据文章标签：大数据 hadoop Hadoop大数据技术

本文链接：https://blog.csdn.net/m0_74394367/article/details/136692128

版权

大数据同时被 3 个专栏收录

44 篇文章 0 订阅

订阅专栏

Hadoop

13 篇文章 0 订阅

订阅专栏

Hadoop大数据技术

8 篇文章 0 订阅

订阅专栏

一、实验目的：

1、使用java对hdfs中的文件进行处理

二、实验要求：

1、使用java对hdfs中的文件进行词频计数，并得到结果。

三、实验设备：

Windows电脑一台、VMware虚拟机、WindTerm终端模拟器、IntelliJ IDEA

四、实验过程：

代码

package edu.sugon;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;

public class WordCount {
    static class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
        @Override
        protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {
            // super.map(key, value, context);
            String row = value.toString();
            String[] words = row.split(" ");
            for (String word : words) {
                context.write(new Text(word), new IntWritable(1));
            }
        }
    }

    static class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
            // super.reduce(key, values, context);
            int total = 0;
            for (IntWritable i : values) {
                total += i.get();
            }
            context.write(key, new IntWritable(total));
        }
    }

    public static void main(String[] args) throws Exception {
        System.setProperty("HADOOP_USER_NAME", "hadoop");
        //main方法在哪个类
        Job job = new Job(new Configuration());
        job.setJarByClass(WordCount.class);
        //设置mapper类
        job.setMapperClass(WordCount.WordCountMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);
        //设置reducer类
        job.setReducerClass(WordCount.WordCountReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        //设置输入文件，输出文件
        FileInputFormat.setInputPaths(job, new Path("hdfs://192.168.10.100/031701/c.txt"));
        FileOutputFormat.setOutputPath(job, new Path("hdfs://192.168.10.100/031701/wordcount"));
        job.waitForCompletion(true);
    }
}