Hudi学习笔记2 - Hudi配置

https://hudi.apache.org/docs/configurations

Hudi配置分类

• Spark Datasource Configs

Spark Datasource 的配置。

• Flink Sql Configs

Flink SQL source/sink connectors 的配置，如：index.type、write.tasks、write.operation、clean.policy、clean.retain_commits、clean.retain_hours、compaction.max_memory、hive_sync.db、hive_sync.table、hive_sync.metastore.uris、write.retry.times、write.task.max.size 等。

• Write Client Configs

控制 Hudi 使用 RDD 的 HoodieWriteClient API 的配置。

• Metastore and Catalog Sync Configs

同步外部元数据的配置。

• Metrics Configs

度量配置。

• Record Payload Config

低级别定制配置，比如设置 Payload 的配置 hoodie.compaction.payload.class 等。

• Kafka Connect Configs

使用 Kafka 作为 Sink Connector 的写 Hudi 表的配置。

• Amazon Web Services Configs

亚马逊 Web Service 配置。

Write Client Configs

• Layout Configs

• Clean Configs

• Memory Configurations

• Archival Configs

• Metadata Configs

• Consistency Guard Configurations

• FileSystem Guard Configurations

• Write Configurations

• Metastore Configs

• Key Generator Options

• Storage Configs

• Compaction Configs

• File System View Storage Configurations

• Clustering Configs

• Common Configurations

• Bootstrap Configs

• Commit Callback Configs

• Lock Configs

• Index Configs

Metastore and Catalog Sync Configs

• Common Metadata Sync Configs

• Global Hive Sync Configs

• DataHub Sync Configs

• BigQuery Sync Configs

• Hive Sync Configs

Metrics Configs

• Metrics Configurations for Datadog reporter

• Metrics Configurations for Amazon CloudWatch

• Metrics Configurations

• Metrics Configurations for Jmx

• Metrics Configurations for Prometheus

• Metrics Configurations for Graphite

Record Payload Config

• Payload Configurations

配置项	是否必须	默认值	配置说明
hoodie.compaction.payload.class	N	org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
hoodie.payload.event.time.field	N	ts
hoodie.payload.ordering.field	N	ts	在合并和写入存储之前，对相同主键进行排序的字段名，默认为 ts。

Kafka Connect Configs

• Kafka Sink Connect Configurations

配置项	是否必须	默认值	配置说明
hadoop.conf.dir	Y	N/A
hadoop.home	Y	N/A
bootstrap.servers	N	bootstrap.servers	Kafka 集群的 bootstrap.servers
hoodie.kafka.control.topic	N	hudi-control-topic
hoodie.meta.sync.classes	N	org.apache.hudi.hive.HiveSyncTool
hoodie.meta.sync.enable	N	false
hoodie.meta.sync.enable	N	org.apache.hudi.schema.FilebasedSchemaProvider
hoodie.kafka.coordinator.write.timeout.secs	N	300
hoodie.kafka.compaction.async.enable	N	true

Amazon Web Services Configs

配置项	是否必须	默认值	配置说明
hoodie.aws.access.key	Y	N/A	AWS access key id
hoodie.aws.secret.key	Y	N/A	AWS secret key
hoodie.aws.session.token	N	N/A	AWS session token