Parquet 笔记

Glossary （术语）

Block (hdfs block): This means a block in hdfs and the meaning is unchanged for describing this file format. The file format is designed to work well on top of hdfs.

File: A hdfs file that must include the metadata for the file. It does not need to actually contain the data.

Row group: A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset.

Column chunk: A chunk of the data for a particular column. These live in a particular row group and is guaranteed to be contiguous in the file.

Page: Column chunks are divided up into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). There can be multiple page types which is interleaved in a column chunk.

Hierarchically, a file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages.

parquet-tools

可参考 http://blog.51yip.com/hadoop/2322.html

parquet 1.10.0 版本里有 tools

wget https://github.com/apache/parquet-mr/archive/apache-parquet-1.10.0.tar.gz
tar zxvf  apache-parquet-1.10.0.tar.gz
cd parquet-mr-apache-parquet-1.10.0/parquet-tools && mvn clean package -Plocal

mvn编译时加上了-Plocal，这样本地用java就可以查看。如果不加，就要用hadoop命令去查看

$ java -jar ./target/parquet-tools-1.10.0.jar schema /home/hadoop/tools/000000_0
message hive_schema {
  optional int64 cc_call_center_sk;
  optional binary cc_call_center_id (UTF8);
  optional int32 cc_rec_start_date (DATE);
  optional int32 cc_rec_end_date (DATE);
  optional int64 cc_closed_date_sk;
  optional int64 cc_open_date_sk;
  optional binary cc_name (UTF8);
  optional binary cc_class (UTF8);
  optional int32 cc_employees;
  optional int32 cc_sq_ft;
  optional binary cc_hours (UTF8);
  optional binary cc_manager (UTF8);

java -jar ./target/parquet-tools-1.10.0.jar meta  /home/hadoop/tools/000000_0

file:              file:/home/hadoop/tools/000000_0 
creator:           parquet-mr version 1.10.0 (build 031a6654009e3b82020012a18434c582bd74c73a) 
extra:             writer.time.zone = Asia/Shanghai 

file schema:       hive_schema 
--------------------------------------------------------------------------------
cc_call_center_sk: OPTIONAL INT64 R:0 D:1
cc_call_center_id: OPTIONAL BINARY O:UTF8 R:0 D:1
cc_rec_start_date: OPTIONAL INT32 O:DATE R:0 D:1
cc_rec_end_date:   OPTIONAL INT32 O:DATE R:0 D:1
cc_closed_date_sk: OPTIONAL INT64 R:0 D:1
cc_open_date_sk:   OPTIONAL INT64 R:0 D:1