java获取hdfs总大小_Java-HDFS上的Apache Spark：一次读取10k-100k的小...

从该职位：How does the number of partitions affect `wholeTextFiles` and `textFiles`?

wholeTextFiles uses WholeTextFileInputFormat … Because it extends CombineFileInputFormat, it will try to combine

groups of smaller files into one partition … Each record in the RDD

… has the entire contents of the file

RDD> wholeTextFiles(String path, int

minPartitions)

Read a directory of text files from HDFS, a local

file system (available on all nodes), or any Hadoop-supported file

system URI.

A org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat for

reading whole text files. Each file is read as key-value pair, where the key is the file path and

the value is the entire content of file.

作为记录,Hadoop CombineInputFormat是在单个Mapper中填充多个小文件的标准方法.它可以在Hive中使用,其属性为hive.hadoop.supports.splittable.combineinputformat和hive.input.format.

Spark WholeTextFiles()重用该Hadoop功能,但有两个缺点：

(a)您必须使用整个目录,不能在加载文件之前按名称过滤掉文件(只能在加载后过滤)

(b)如果需要,您必须通过将每个文件分成多个记录来对RDD进行后处理