从该职位:How does the number of partitions affect `wholeTextFiles` and `textFiles`?
wholeTextFiles uses WholeTextFileInputFormat … Because it extends CombineFileInputFormat, it will try to combine
groups of smaller files into one partition … Each record in the RDD
… has the entire contents of the file
RDD> wholeTextFiles(String path, int
minPartitions)
Read a directory of text files from HDFS, a local
file system (available on all nodes), or any Hadoop-supported file
system URI.
A org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat for
reading whole text files. Each file is read as key-value pair, where the key is the file path and
the value is the entire content of file.
作为记录,Hadoop CombineInputFormat是在单个Mapper中填充多个小文件的标准方法.它可以在Hive中使用,其属性为hive.hadoop.supports.splittable.combineinputformat和hive.input.format.
Spark WholeTextFiles()重用该Hadoop功能,但有两个缺点:
(a)您必须使用整个目录,不能在加载文件之前按名称过滤掉文件(只能在加载后过滤)
(b)如果需要,您必须通过将每个文件分成多个记录来对RDD进行后处理