编写环境：windows ，IntelliJ IDEA 2018.1.4 x64 ，maven，jdk-1.8
运行环境：centos-7.3，hadoop-2.7.3，jdk-1.8
基本思路：在windows中的idea新建maven项目wordcount并编写，将项目打包成jar，上传至hadoop并执行作业
一、新建maven项目
1、菜单File——>New——>Project…——>Maven（编写环境的jdk和运行环境的jdk最好一致），结果如下:
2、点击Next，结果如下：
3、填好GroupId和ArtifactId，点击Next，结果如下：
4、Finish.
二、编写wordcount项目
1、建立项目结构目录
2、编写pom.xml（引入用到的jar包）
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cn.lg</groupId>
<artifactId>wordcount</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.3</version>
</dependency>
</dependencies>
</project>
3、编写项目代码
（1）WordcountMapper.java
package cn.lg.project;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import java.io.IOException;
public class WordcountMapper extends org.apache.hadoop.mapreduce.Mapper<LongWritable,Text,Text,IntWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line=value.toString();
String[] words=line.split(" ");
for (String word:words){
context.write(new Text(word),new IntWritable(1));
}
}
}
（2）WordcountReducer.java
package cn.lg.project;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
import java.util.Iterator;
public class WordcountReducer extends Reducer<Text,IntWritable,Text,IntWritable> {
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
Integer counts=0;
for (IntWritable value:values){
counts+=value.get();
}
context.write(key,new IntWritable(counts));
}
}
（3）WordcountMain.java
package cn.lg.project;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordcountMain {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "wordcount");
job.setJarByClass(WordcountMain.class);
job.setMapperClass(WordcountMapper.class);
job.setReducerClass(WordcountReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
boolean flag = job.waitForCompletion(true);
if (!flag) {
System.out.println("wordcount failed!");
}
}
}
三、将项目打包成jar
1、右键项目名称——>Open Module Settings，如下：
2、Artifacts——>+——>JAR——>From modules with dependencies…，如下：
3、填写Main Class（点击…选择WordcountMain），再然后下面有两个选项，第一个是
extract to the target JAR，指将项目及项目依赖的包都打包成一个JAR（结果运行比较慢，见附录），第二个是copy to the output directory and link via manifest，指其他依赖包分开放，结果为多个JAR，因为执行环境hadoop上已经有相关的依赖包，这里选第二个，点击ok，如下：
4、勾选include in project build ，其中Output directory为最后的输出目录，下面output layout是输出的各jar包，点击ok，如下：
5、点击菜单Build——>Build Aritifacts…，如下：
6、选择Build，结果可到前面4的output目录查看或者项目结构中的out目录，如下：
7、结果查看如下，找到wordcount.jar，并上传到hadoop：
四、执行验证
1、新建测试文本testdata，并添加内容“I believe that I will be successful”,上传至hdfs，如下可查看：
[hadoop@master ~]$ hdfs dfs -cat /user/hadoop/input/testdata
19/03/20 14:35:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I believe that I will be successful
2、执行命令，注意命令的形式，只有两个参数，主类已经在代码和打jar包的时候设置了，所以这里命令不用输入，和使用hadoop自带的wordcount不一样，况且代码中args[0]已经设置为输入路径了，args[1]为输出路径，查看结果：
[hadoop@master ~]$ hadoop jar wordcount.jar /user/hadoop/input/testdata /user/hadoop/output3
19/03/20 14:39:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/03/20 14:39:26 INFO client.RMProxy: Connecting to ResourceManager at master/172.16.0.17:8032
19/03/20 14:39:27 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/03/20 14:39:28 INFO input.FileInputFormat: Total input paths to process : 1
19/03/20 14:39:28 INFO mapreduce.JobSubmitter: number of splits:1
19/03/20 14:39:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553002961722_0009
19/03/20 14:39:29 INFO impl.YarnClientImpl: Submitted application application_1553002961722_0009
19/03/20 14:39:29 INFO mapreduce.Job: The url to track the job: http://master:8077/proxy/application_1553002961722_0009/
19/03/20 14:39:29 INFO mapreduce.Job: Running job: job_1553002961722_0009
19/03/20 14:39:39 INFO mapreduce.Job: Job job_1553002961722_0009 running in uber mode : false
19/03/20 14:39:39 INFO mapreduce.Job: map 0% reduce 0%
19/03/20 14:39:46 INFO mapreduce.Job: map 100% reduce 0%
19/03/20 14:39:53 INFO mapreduce.Job: map 100% reduce 100%
19/03/20 14:39:55 INFO mapreduce.Job: Job job_1553002961722_0009 completed successfully
19/03/20 14:39:56 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=84
FILE: Number of bytes written=236613
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=146
HDFS: Number of bytes written=46
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4552
Total time spent by all reduces in occupied slots (ms)=4754
Total time spent by all map tasks (ms)=4552
Total time spent by all reduce tasks (ms)=4754
Total vcore-milliseconds taken by all map tasks=4552
Total vcore-milliseconds taken by all reduce tasks=4754
Total megabyte-milliseconds taken by all map tasks=4661248
Total megabyte-milliseconds taken by all reduce tasks=4868096
Map-Reduce Framework
Map input records=1
Map output records=7
Map output bytes=64
Map output materialized bytes=84
Input split bytes=110
Combine input records=0
Combine output records=0
Reduce input groups=6
Reduce shuffle bytes=84
Reduce input records=7
Reduce output records=6
Spilled Records=14
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=157
CPU time spent (ms)=1320
Physical memory (bytes) snapshot=302055424
Virtual memory (bytes) snapshot=4166328320
Total committed heap usage (bytes)=165810176
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=36
File Output Format Counters
Bytes Written=46
第一个为hdfs中的输入目录，第二个为hdfs的输出目录，输出目录不能先存在，否则会报错。
3、再次验证：
[hadoop@master ~]$ hdfs dfs -cat /user/hadoop/output3/part-r-00000
19/03/20 14:44:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I 2
be 1
believe 1
successful 1
that 1
will 1
五、附录
若将整个项目都打成jar包，结果会很慢，运行如下：
[hadoop@master ~]$ hadoop jar wordcount.jar /user/hadoop/input/testdata /user/hadoop/output4
19/03/20 15:09:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/03/20 15:09:59 INFO client.RMProxy: Connecting to ResourceManager at master/172.16.0.17:8032
19/03/20 15:10:00 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/03/20 15:18:03 WARN hdfs.DFSClient: Slow waitForAckedSeqno took 41091ms (threshold=30000ms)
19/03/20 15:18:03 INFO input.FileInputFormat: Total input paths to process : 1
19/03/20 15:18:04 INFO mapreduce.JobSubmitter: number of splits:1
19/03/20 15:18:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553002961722_0011
19/03/20 15:18:05 INFO impl.YarnClientImpl: Submitted application application_1553002961722_0011
19/03/20 15:18:05 INFO mapreduce.Job: The url to track the job: http://master:8077/proxy/application_1553002961722_0011/
19/03/20 15:18:05 INFO mapreduce.Job: Running job: job_1553002961722_0011
19/03/20 15:26:37 INFO mapreduce.Job: Job job_1553002961722_0011 running in uber mode : false
19/03/20 15:26:37 INFO mapreduce.Job: map 0% reduce 0%
19/03/20 15:26:43 INFO mapreduce.Job: map 100% reduce 0%
19/03/20 15:26:50 INFO mapreduce.Job: map 100% reduce 100%
19/03/20 15:26:52 INFO mapreduce.Job: Job job_1553002961722_0011 completed successfully
19/03/20 15:26:52 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=84
FILE: Number of bytes written=236613
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=146
HDFS: Number of bytes written=46
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4610
Total time spent by all reduces in occupied slots (ms)=5078
Total time spent by all map tasks (ms)=4610
Total time spent by all reduce tasks (ms)=5078
Total vcore-milliseconds taken by all map tasks=4610
Total vcore-milliseconds taken by all reduce tasks=5078
Total megabyte-milliseconds taken by all map tasks=4720640
Total megabyte-milliseconds taken by all reduce tasks=5199872
Map-Reduce Framework
Map input records=1
Map output records=7
Map output bytes=64
Map output materialized bytes=84
Input split bytes=110
Combine input records=0
Combine output records=0
Reduce input groups=6
Reduce shuffle bytes=84
Reduce input records=7
Reduce output records=6
Spilled Records=14
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=180
CPU time spent (ms)=1360
Physical memory (bytes) snapshot=310743040
Virtual memory (bytes) snapshot=4172840960
Total committed heap usage (bytes)=165810176
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=36
File Output Format Counters
Bytes Written=46
六、补充
可能会有朋友在执行hadoop jar时出现找不到main主类的问题，这时候可以在命令中指定主类（前面的包名也需要），如：
hadoop jar xx.jar cn.lg.project.WordcountMain input_path output_path
857
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
