假设:Hadoop已安装好。Hadoop安装步骤:https://blog.csdn.net/LiuHuan_study/article/details/84347262
第一步:配置环境变量
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64/
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export CLASSPATH=.:${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.7.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-2.7.7.jar:${CLASSPATH}
可以以上配置放在cat ~/.bashrc中,每次启动服务器时,环境变量都有效。
或者放在一个启动文件vi /root/hadoop-env.sh中,每次服务器重启都需要,先执行这个脚本文件
或者,每次启动Hadoop时,手动执行一遍上诉命令
第二步:编写WordCount.java程序
WordCount.java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
第三步:借助Xshell工具上传到服务器,然后进行编译
上传命令:rz
我上传的路径为:/home/
编译:
#编译,打包jar文件。因为编译后可能会生成多个.class文件,所以需要加星号
javac WordCount.java
jar cvf WordCount.jar WordCount*class
第四步:使用WordCount.jar包,计算某一文档或者某一路径下(该路径下只有文档,没有文件夹)所有文档中单词的个数
/LiuHuan统计的路径;/output统计结果输出路径。
hadoop jar /home/WordCount.jar WordCount /LiuHuan /output
#或者
hadoop jar /home/WordCount.jar WordCount /LiuHuan/test.txt /output
注意:保证路径/output在HDFS中不存在,否则会报错。
第五步:查看统计结果
[root@localhost home]# hdfs dfs -cat /output/part-r-00000
Huan 2
Liu 2
hello 1
world! 1