【大数据入门核心技术-Iceberg】（四）Iceberg与Spark整合

一、下载部署

wget https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.2_2.12/1.1.0/iceberg-spark-runtime-3.2_2.12-1.1.0.jar

cp iceberg-spark-runtime-3.2_2.12-1.1.0.jar /data/bigdata/spark-3.2.1/jars/

二、Spark 配置 Catalog

Spark中支持两种Catalog的设置：hive和hadoop，Hive Catalog就是Iceberg表存储使用Hive默认的数据路径，Hadoop Catalog需要指定Iceberg格式表存储路径。

vim spark-defaults.conf
1）Hive Catalog
spark.sql.catalog.hive_prod = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.hive_prod.type = hive
spark.sql.catalog.hive_prod.uri = thrift://hadoop101:9083

2）Hadoop Catalog
spark.sql.catalog.hadoop_prod = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.hadoop_prod.type = hadoop
spark.sql.catalog.hadoop_prod.warehouse = hdfs://hadoop101:9000/warehouse/spark-iceberg

hadoop fs -mkdir -p hdfs://h