可以利用override map函数的第三个参数。map函数如下:
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter)
String path = ((FileSplit) reporter.getInputSplit()).getPath().toString();
即可取得全路径。
下面这个方法可以做文件选取:
public class Filter implements PathFilter {
public boolean accept(Path path) {
return !(path.toString().indexOf("abc") > -1);
}
}
jobConf
.set(
"mapred.input.pathFilter.class"
,
"Filter"
);
org.apache.hadoop.fs
Interface PathFilter
-
All Known Implementing Classes:
- OutputLogFilter
public interface PathFilter
Method Summary | |
---|---|
boolean | accept(Path path) Tests whether or not the specified abstract pathname should be included in a pathname list. |
Method Detail |
---|
accept
boolean accept(Path path)
-
Tests whether or not the specified abstract pathname should be included in a pathname list.
-
Parameters:
-
path
- The abstract pathname to be tested
Returns:
-
true
if and only ifpathname
should be included
-