hdfs 多个文件合并,如何将HDFS小文件合并为一个大文件？

I have number of small files generated from Kafka stream so I like merge small files to one single file but this merge is based on the date i.e. the original folder may have number of previous files but I only like to merge for given date files to one single file.

Any suggestions?

解决方案

Use something like the code below to iterate over the smaller files and aggregate them into a big one (assuming that source contains the HDFS path to your smaller files, and target is the path where you want your big result file):

val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)

fs.listStatus(new Path(source)).map(_.getPath.toUri.getPath).

foreach(name => spark.read.text(name).coalesce(1).write.mode(Append).text(target))

This example assumes text file format, but you can just as well read any Spark-supported format, and you can use different formats for source and target, as well