使用intellij搭建运行MapReduce环境

使用intellij搭建运行MapReduce环境

说明：因为本人是在Windows中的idea中编写的MapReduce程序，每次编写玩程序后都需要打成jar包发布到集群中去检查程序的正确性，个人感觉比较麻烦。

一、实验环境

物理机：Windows10

idea：intellij 2017

二、具体需求

在idea中搭建一个用于运行MapReduce程序的环境，这样就可以在idea中运行或调试MapReduce程序的正确性，不再需要每次都打包到集群中去运行测试程序的正确性，方便了开发和调试。

三、环境搭建及测试

1、新建一个maven项目（这里不详述）

2、添加hadoop依赖包：一般需要导入common包下的hadoop-common.jar hadoop-nfs.jar、common小的lib包、mapreduce包、yarn包、hdfs包。（如果直接将所有包一起导入，运行MapReduce程序的时候可能会报错：Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper（我测试中是这样的））

3、编写wordcount程序进行测试

mapper：

package com.wp.demo.mapper;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;

public class AnasislyTypeMapper extends Mapper<LongWritable,Text,Text,IntWritable> {

    private Text k = new Text();
    private IntWritable v = new IntWritable();
    
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        String[] fields = value.toString().split(" ");
        for (String field : fields) {
            k.set(field);
            v.set(1);
            context.write(k,v);
        }
    }
}

reducer:

package reducer;
import org.apache.hadoop.hdfs.TestExtendedAcls;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;

public class AnasislyTypeReducer extends Reducer<Text,IntWritable,Text,IntWritable> {

    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {

        int count = 0;
        IntWritable v = new IntWritable();
        for (IntWritable value : values) {
            count = count + value.get();
        }
        v.set(count);
        context.write(key,v);
    }
}

job:(需要在指定的位置创建一个用于wordcount的文件)

package tool;
import com.wp.demo.mapper.AnasislyTypeMapper;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import reducer.AnasislyTypeReducer;

public class AnasislyTypeTool {

    public static void main(String[] args) throws Exception {
        args = new String[]{"E:\\input\\demoinput\\hello1.txt","E:\\input\\demooutput\\out1\\"};
        Configuration conf = new Configuration();
        //获取job对象
        Job job = Job.getInstance(conf);

        //设置jar包
        job.setJarByClass(AnasislyTypeTool.class);

        //关联mapper和reducer
        job.setMapperClass(AnasislyTypeMapper.class);
        job.setReducerClass(AnasislyTypeReducer.class);

        //设置map输出数据类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        //设置最终输出数据类型kv
        job.setOutputKeyClass(Text .class);
        job.setOutputValueClass(IntWritable.class);

        //设置输入输出文件路径
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        //提交到yarn集群
        job.waitForCompletion(true);
    }
}

**maven依赖如下：**虽然有些依赖不用导入，这里不进行去除

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>2.9.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>2.9.1</version>
        </dependency>
    </dependencies>

运行以上wordcount程序**

四、运行结果

控制台打印结果：

将程序打包以便在hadoop集群中运行
1.

2.配置打包信息

点击OK以后，选择build artifact进行打包！

打包完毕之后，在对应的目录中就可以看到打好的jar包（默认是在out目录中）