Parquet 文件格式分析

Glossary （术语）

• Block (hdfs block): This means a block in hdfs and the meaning is unchanged for describing this file format. The file format is designed to work well on top of hdfs.

• File: A hdfs file that must include the metadata for the file. It does not need to actually contain the data.

• Row group: A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset.

• Column chunk: A chunk of the data for a particular column. These live in a particular row group and is guaranteed to be contiguous in the file.

• Page: Column chunks are divided up into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). There can be multiple page types which is interleaved in a column chunk.

Hierarchically(层次地讲), a file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages.

Unit of parallelization

• MapReduce - File/Row Group
• IO - Column chunk
• Encoding/Compression - Page

看下真实数据

# cat /home/spark-bin/spark-2.4.1-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet
PAR1"FReynoldReynolcb```fdA©y󨔽ё,Hroot 
                                     %name%&
                                           namep<ReynoldReynoldp)org.apache.spark.sql.parquet.row.metadataZ{"type":"struct","fields":[{"name":"name","type":"string","nullable":true,"metadata":{}}]}arquet-mr version 1.6.0rc3PAR1

File format

4-byte magic number "PAR1"
<Column 1 Chunk 1 + Column Metadata>
<Column 2 Chunk 1 + Column Metadata>
...
<Column N Chunk 1 + Column Metadata>
<Column 1 Chunk 2 + Column Metadata>
<Column 2 Chunk 2 + Column Metadata>
...
<Column N Chunk 2 + Column Metadata>
...
<Column 1 Chunk M + Column Metadata>
<Column 2 Chunk M + Column Metadata>
...
<Column N Chunk M + Column Metadata>
File Metadata
4-byte length in bytes of file metadata
4-byte magic number "PAR1"

Metadata

The file metadata contains the locations of all the column metadata start locations.

There are three types of metadata: file metadata, column (chunk) metadata and page header metadata.