Main-Class为HadoopStreaming,主函数:
public static void main(String[] args) throws Exception {
int returnStatus = 0;
StreamJob job = new StreamJob();
returnStatus = ToolRunner.run(job, args);
if (returnStatus != 0) {
System.err.println("Streaming Job Failed!");
System.exit(returnStatus);
}
}
核心类为StreamJob,StreamJob实现了org.apache.hadoop.util.Tool接口,其不带参数的构造函数做了两件事,调用setupOptions()初始化其支持的所有选项,以及初始化config_。
其支持的选项有:
name | desc | argName | max | required |
input | DFS input file(s) for the map step | path | Integer.MAX_VALUE | true |
output | DFS output directory for the reduce step | path | 1 | true |
mapper | the streaming command to run | cmd | 1 | false |
combiner | the streaming command to run | cmd | 1 | false |
reducer | the streaming command to run | cmd | 1 | false |
file | file to be shipped in the job jar file | file | Integer.MAX_VALUE | false |
dfs | Optional,override DFS configu |