parquet文件上传到hdfs,从HDFS收集Parquet数据到本地文件系统

Given a Parquet dataset distributed on HDFS (metadata file + may .parquet parts), how to correctly merge parts and collect the data onto local file system? dfs -getmerge ... doesn't work - it merges metadata with actual parquet files..

解决方案

There is a way involving Apache Spark APIs - which provides a solution, but more efficient method without third-party tools may exist.

spark> val parquetData = sqlContext.parquetFile("pathToMultipartParquetHDFS")

spark> parquet.repartition(1).saveAsParquetFile("pathToSinglePartParquetHDFS")

bash> ../bin/hadoop dfs -get pathToSinglePartParquetHDFS localPath

Since Spark 1.4 it's better to use DataFrame::coalesce(1) instead of DataFrame::repartition(1)