Parquet 笔记

Glossary （术语）

Block (hdfs block): This means a block in hdfs and the meaning is unchanged for describing this file format. The file format is designed to work well on top of hdfs.

File: A hdfs file that must include the metadata for the file. It does not need to actually contain the data.

Row group: A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset.

Column chunk: A chunk of the data for a particular column. These live in a particular row group and is guaranteed to be contiguous in the file.

Page: Column chunks are divided up into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). There can be multiple page types which is interleaved in a column chunk.

Hierarchically, a file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages.

parquet-tools

可参考 http://blog.51yip.com/hadoop/2322.html

parquet 1.10.0 版本里有 tools

wget https://github.com/apache/parquet-mr/archive/apache-parquet-1.10.0.tar.gz
tar zxvf  apache-parquet-1.10.0.tar.gz
cd parquet-mr-apache-parquet-1.10.0/parquet-tools && mvn clean package -Plocal

mvn编译时加上了-Plocal，这样本地用java就可以查看。如果不加，就要用hadoop命令去查看

$ java -jar ./target/parquet-tools-1.10.0.jar schema /home/hadoop/tools/000000_0
message hive_schema {
  optional int64 cc_call_center_sk;
  optional binary cc_call_center_id (UTF8);
  optional int32 cc_rec_start_date (DATE);
  optional int32 cc_rec_end_date (DATE);
  optional int64 cc_closed_date_sk;
  optional int64 cc_open_date_sk;
  optional binary cc_name (UTF8);
  optional binary cc_class (UTF8);
  optional int32 cc_employees;
  optional int32 cc_sq_ft;
  optional binary cc_hours (UTF8);
  optional binary cc_manager (UTF8);

java -jar ./target/parquet-tools-1.10.0.jar meta  /home/hadoop/tools/000000_0

file:              file:/home/hadoop/tools/000000_0 
creator:           parquet-mr version 1.10.0 (build 031a6654009e3b82020012a18434c582bd74c73a) 
extra:             writer.time.zone = Asia/Shanghai 

file schema:       hive_schema 
--------------------------------------------------------------------------------
cc_call_center_sk: OPTIONAL INT64 R:0 D:1
cc_call_center_id: OPTIONAL BINARY O:UTF8 R:0 D:1
cc_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
cc_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
cc_closed_date_sk: OPTIONAL INT64 R:0 D:1
cc_open_date_sk:   OPTIONAL INT64 R:0 D:1

$ java -jar parquet-tools-1.6.0rc3-SNAPSHOT.jar meta  part-00354-4b3c5b7c-fc6c-4e9b-a947-0efc927a3d16-c000-03e97306-305a-4270-8ec9-f33038d0d547.snappy.parquet
file:                  file:/home/hadoop/tmp/part-00354-4b3c5b7c-fc6c-4e9b-a947-0efc927a3d16-c000-03e97306-305a-4270-8ec9-f33038d0d547.snappy.parquet 
creator:               parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94) 
extra:                 org.apache.spark.version = 3.3.2 
extra:                 org.apache.spark.sql.parquet.row.metadata = {"type":"struct","fields":[{"name":"ss_sold_date_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_sold_time_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_item_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_customer_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_cdemo_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_hdemo_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_addr_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_store_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_promo_sk","type":"integer","nullable":true,"metadata":{}},{"name":"ss_ticket_number","type":"long","nullable":true,"metadata":{}},{"name":"ss_quantity","type":"integer","nullable":true,"metadata":{}},{"name":"ss_wholesale_cost","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_list_price","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_sales_price","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_ext_discount_amt","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_ext_sales_price","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_ext_wholesale_cost","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_ext_list_price","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_ext_tax","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_coupon_amt","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_net_paid","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_net_paid_inc_tax","type":"decimal(7,2)","nullable":true,"metadata":{}},{"name":"ss_net_profit","type":"decimal(7,2)","nullable":true,"metadata":{}}]} 

file schema:           spark_schema 
--------------------------------------------------------------------------------
ss_sold_date_sk:       OPTIONAL INT32 R:0 D:1
ss_sold_time_sk:       OPTIONAL INT32 R:0 D:1
ss_item_sk:            OPTIONAL INT32 R:0 D:1
ss_customer_sk:        OPTIONAL INT32 R:0 D:1
ss_cdemo_sk:           OPTIONAL INT32 R:0 D:1
ss_hdemo_sk:           OPTIONAL INT32 R:0 D:1
ss_addr_sk:            OPTIONAL INT32 R:0 D:1
ss_store_sk:           OPTIONAL INT32 R:0 D:1
ss_promo_sk:           OPTIONAL INT32 R:0 D:1
ss_ticket_number:      OPTIONAL INT64 R:0 D:1
ss_quantity:           OPTIONAL INT32 R:0 D:1
ss_wholesale_cost:     OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_list_price:         OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_sales_price:        OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_ext_discount_amt:   OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_ext_sales_price:    OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_ext_wholesale_cost: OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_ext_list_price:     OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_ext_tax:            OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_coupon_amt:         OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_net_paid:           OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_net_paid_inc_tax:   OPTIONAL INT32 O:DECIMAL R:0 D:1
ss_net_profit:         OPTIONAL INT32 O:DECIMAL R:0 D:1

row group 1:           RC:2976358 TS:156129236 OFFSET:4 
--------------------------------------------------------------------------------
ss_sold_date_sk:        INT32 SNAPPY DO:4 FPO:7323 SZ:1121973/1195173/1.07 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_sold_time_sk:        INT32 SNAPPY DO:1121977 FPO:1305054 SZ:1347635/1430390/1.06 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_item_sk:             INT32 SNAPPY DO:0 FPO:2469612 SZ:11912216/11910938/1.00 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_customer_sk:         INT32 SNAPPY DO:14381828 FPO:15364121 SZ:2343714/2419615/1.03 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_cdemo_sk:            INT32 SNAPPY DO:16725542 FPO:17656783 SZ:2291466/2367217/1.03 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_hdemo_sk:            INT32 SNAPPY DO:19017008 FPO:19045838 SZ:1169352/1241986/1.06 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_addr_sk:             INT32 SNAPPY DO:20186360 FPO:21158337 SZ:2332103/2409319/1.03 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_store_sk:            INT32 SNAPPY DO:22518463 FPO:22520494 SZ:1090598/1164267/1.07 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_promo_sk:            INT32 SNAPPY DO:23609061 FPO:23615087 SZ:4182307/4256114/1.02 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_ticket_number:       INT64 SNAPPY DO:27791368 FPO:28311576 SZ:2074071/12832779/6.19 VC:2976358 ENC:PLAIN,PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_quantity:            INT32 SNAPPY DO:29865439 FPO:29865866 SZ:2756151/2829008/1.03 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_wholesale_cost:      INT32 SNAPPY DO:32621590 FPO:32661225 SZ:5282909/5355362/1.01 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_list_price:          INT32 SNAPPY DO:0 FPO:37904499 SZ:11610806/11703283/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_sales_price:         INT32 SNAPPY DO:49515305 FPO:49590223 SZ:5629708/5702719/1.01 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_ext_discount_amt:    INT32 SNAPPY DO:55145013 FPO:55998427 SZ:3920409/6285563/1.60 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_ext_sales_price:     INT32 SNAPPY DO:0 FPO:59065422 SZ:11631176/11703759/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_ext_wholesale_cost:  INT32 SNAPPY DO:0 FPO:70696598 SZ:11631392/11704018/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_ext_list_price:      INT32 SNAPPY DO:0 FPO:82327990 SZ:11630825/11703709/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_ext_tax:             INT32 SNAPPY DO:93958815 FPO:94275318 SZ:6443847/6516681/1.01 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_coupon_amt:          INT32 SNAPPY DO:100402662 FPO:101256076 SZ:3920409/6285563/1.60 VC:2976358 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE
ss_net_paid:            INT32 SNAPPY DO:0 FPO:104323071 SZ:11632742/11705341/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_net_paid_inc_tax:    INT32 SNAPPY DO:0 FPO:115955813 SZ:11630771/11703077/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE
ss_net_profit:          INT32 SNAPPY DO:0 FPO:127586584 SZ:11630241/11703355/1.01 VC:2976358 ENC:PLAIN,BIT_PACKED,RLE

row group 2:           RC:2976358 TS:156138550 OFFSET:139216825

被缩减的字段说明:

RC = Record Count, TS = Total Byte Size
DO = DictionaryPageOffset
FPO = FirstDataPageOffset
SZ : 字段第一个值 = TotalSize, 第二个值 = TotalUncompressedSize， 第三个值 ratio = TotalUncompressedSize / TotalSize
VC = ValueCount
ENC = Encodings

• TS 指的压缩前的大小
• row group大小为 parquet.block.size(默认128M，压缩后的)，row group是一个切片的最小单位；
• page大小为parquet.page.size(默认1M，压缩后的)，page是压缩和一次读取的最小单位；

parquet.block.size
parquet.page.size

 * # The block size is the size of a row group being buffered in memory
 * # this limits the memory usage when writing
 * # Larger values will improve the IO when reading but consume more memory when writing
 * parquet.block.size=134217728 # in bytes, default = 128 * 1024 * 1024
 *
 * # The page size is for compression. When reading, each page can be decompressed independently.
 * # A block is composed of pages. The page is the smallest unit that must be read fully to access a single record.
 * # If this value is too small, the compression will deteriorate
 * parquet.page.size=1048576 # in bytes, default = 1 * 1024 * 1024

参考链接：

https://github.com/apache/parquet-java/blob/c928c4bfc15717d39057ba14663ecda7ff3f8bf2/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java#L68