安装好了hadoop集群环境,详细见(hadoop安装),当然得要跑一下Map/Reduce
package com.hadoop;
import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class MaxValue {
static class MaxValueMapper extends
Mapper<LongWritable, Text, Text, IntWritable> {
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
// 从每行的数据中分解要统计的key和value
String thekey = line.substring(0, 4);
int theValue = Integer.parseInt(line.substring(5, 8));
context.write(new Text(thekey), new IntWritable(theValue));
}
}
static class MaxValueReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException, InterruptedException {
int maxValue = Integer.MIN_VALUE;
// key求出最大的温度值
for (IntWritable value : values) {
maxValue = Math.max(maxValue, value.get());
}
context.write(key, new IntWritable(maxValue));
}
}
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Usage: MaxValue <input path> <output path>");
System.exit(-1);
}
Job job = new Job();
job.setJarByClass(MaxValue.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(MaxValueMapper.class);
job.setReducerClass(MaxValueReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
代码很简单,打包成jar包,随便命名成first.jar
然后写个程序,随机生成一批数据上传到hadoop中,为程序简单处理,生成一批格式如
2000 111
2012 333
2012 444
2000 222
类似一大批数据命名为temp.txt
上传
hadoop dfs -put temp1.txt /user/hadoop/input/
列出HDFS下的文件
hadoop dfs -ls
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2012-04-06 13:40 /user/hadoop/input
drwxr-xr-x - hadoop supergroup 0 2012-04-06 11:30 /user/hadoop/output
hadoop dfs -ls in 列出HDFS下某个文档中的文件
hadoop dfs -ls in /user/hadoop/input
Found 2 items
-rw-r--r-- 2 hadoop supergroup 2043 2012-02-29 18:18 /user/hadoop/input/slaves.sh
-rw-r--r-- 2 hadoop supergroup 10000 2012-04-06 13:39 /user/hadoop/input/temp1.txt
可以看到上传上去的temp1.txt文件
然后运行
hadoop jar first.jar com.hadoop.MaxValue /user/hadoop/input/temp1.txt output1
first.jar jar的名字
com.hadoop.MaxValue 类名
/user/hadoop/input/temp1.txt mian函数对应的输入
output1 mian函数对应的输出
然后可以看到
12/04/06 13:41:43 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/06 13:41:43 INFO input.FileInputFormat: Total input paths to process : 1
12/04/06 13:41:44 INFO mapred.JobClient: Running job: job_201203121856_0005
12/04/06 13:41:45 INFO mapred.JobClient: map 0% reduce 0%
12/04/06 13:41:58 INFO mapred.JobClient: map 100% reduce 0%
12/04/06 13:42:10 INFO mapred.JobClient: map 100% reduce 100%
12/04/06 13:42:15 INFO mapred.JobClient: Job complete: job_201203121856_0005
12/04/06 13:42:15 INFO mapred.JobClient: Counters: 25
12/04/06 13:42:15 INFO mapred.JobClient: Job Counters
12/04/06 13:42:15 INFO mapred.JobClient: Launched reduce tasks=1
12/04/06 13:42:15 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=12195
12/04/06 13:42:15 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/04/06 13:42:15 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/04/06 13:42:15 INFO mapred.JobClient: Launched map tasks=1
12/04/06 13:42:15 INFO mapred.JobClient: Data-local map tasks=1
12/04/06 13:42:15 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10085
12/04/06 13:42:15 INFO mapred.JobClient: File Output Format Counters
12/04/06 13:42:15 INFO mapred.JobClient: Bytes Written=27
12/04/06 13:42:15 INFO mapred.JobClient: FileSystemCounters
12/04/06 13:42:15 INFO mapred.JobClient: FILE_BYTES_READ=11006
12/04/06 13:42:15 INFO mapred.JobClient: HDFS_BYTES_READ=10111
12/04/06 13:42:15 INFO mapred.JobClient: FILE_BYTES_WRITTEN=63937
12/04/06 13:42:15 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=27
12/04/06 13:42:15 INFO mapred.JobClient: File Input Format Counters
12/04/06 13:42:15 INFO mapred.JobClient: Bytes Read=10000
12/04/06 13:42:15 INFO mapred.JobClient: Map-Reduce Framework
12/04/06 13:42:15 INFO mapred.JobClient: Reduce input groups=3
12/04/06 13:42:15 INFO mapred.JobClient: Map output materialized bytes=11006
12/04/06 13:42:15 INFO mapred.JobClient: Combine output records=0
12/04/06 13:42:15 INFO mapred.JobClient: Map input records=1000
12/04/06 13:42:15 INFO mapred.JobClient: Reduce shuffle bytes=11006
12/04/06 13:42:15 INFO mapred.JobClient: Reduce output records=3
12/04/06 13:42:15 INFO mapred.JobClient: Spilled Records=2000
12/04/06 13:42:15 INFO mapred.JobClient: Map output bytes=9000
12/04/06 13:42:15 INFO mapred.JobClient: Combine input records=0
12/04/06 13:42:15 INFO mapred.JobClient: Map output records=1000
12/04/06 13:42:15 INFO mapred.JobClient: SPLIT_RAW_BYTES=111
12/04/06 13:42:15 INFO mapred.JobClient: Reduce input records=1000
同时可以在控制台http://node1:50030/jobtracker.jsp
看到任务的运行情况,
运行完毕,可以执行
hadoop dfs -ls in /user/hadoop/output1
Found 3 items
-rw-r--r-- 2 hadoop supergroup 0 2012-04-06 13:42 /user/hadoop/output1/_SUCCESS
drwxr-xr-x - hadoop supergroup 0 2012-04-06 13:41 /user/hadoop/output1/_logs
-rw-r--r-- 2 hadoop supergroup 27 2012-04-06 13:42 /user/hadoop/output1/part-r-00000
生成了结果文件,
查看最终结果
hadoop dfs -cat /user/hadoop/output1/part-r-00000
2009 999
2010 129
2011 177
这是程序生成原始数据,然后手动加入2009 最大值999 可以检查结果是否正确
当然可以看看hadoop的示例,在http://hadoop.apache.org/common/docs/r0.20.2/cn/mapred_tutorial.html