HDFS小文件解决方案:
https://blog.csdn.net/Androidlushangderen/article/details/54176511
Flume传递文件到HDFS解决小文件问题:
设置回滚参数:三个参数当满足其中一项时进行回滚操作,通常情况下三者搭配使用。
hdfs.rollSize | 1024 | File size to trigger roll, in bytes (0: never roll based on file size) |
hdfs.rollCount | 10 | Number of events written to file before it rolled (0 = never roll based on number of events) |
hdfs.idleTimeout | 0 | Timeout after which inactive files get closed (0 = disable automatic closing of idle files) |