spark3.3.2集成hudi同步hive表

环境概述

1. hadoop3.x
2. hive3.x
3. spark3.3.2
4. scala2.12
5. hudi0.13

spark集成hive

假设我的spark安装目录为
/data/spark

将
hive-site.xml
传入
/data/spark/conf目录下

spark集成hudi

根据你实际的spark版本和scala版本去找对应版本的jar包
下载jar包
链接: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.3-bundle_2.12/0.13.0/hudi-spark3.3-bundle_2.12-0.13.0.jar

把
hudi-spark3.3-bundle_2.12-0.13.0.jar
传入
/data/spark/jars
目录下

hudi集成hive

下载jar包
链接: https://repo1.maven.org/maven2/org/apache/hudi/hudi-hadoop-mr-bundle/0.13.0/hudi-hadoop-mr-bundle-0.13.0.jar

把
hudi-hadoop-mr-bundle-0.13.0.jar
传入
$HIVE_HOME/auxlib目录下

我这里是用ambari安装的，目录为
/usr/hdp/3.1.5.0-152/hive/auxlib/

然后重启HIVE!!!
然后重启HIVE!!!
然后重启HIVE!!!

然后重启HIVE!!!重要的事说三遍

通过写入hudi同步到hive表代码实例

1.进入spark-shell

/data/spark/bin/spark-shell \
  --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
  --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
  --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'

2.复制下面的代码
我这里是写入了ods.hudi_cow
路径是/warehouse/hudi/ods/hudi_cow
可以根据你们的实际情况改变
注意：
/warehouse/hudi/ods/hudi_cow
这个路径必须有权限!!!
这个路径必须有权限!!!
这个路径必须有权限!!!

这个路径必须有权限!!!重要的事说三遍

import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row


val tableName = "hudi_cow"
val databaseName= "ods"
val basePath = "/warehouse/hudi/ods/hudi_cow"

val schema = StructType(Array(
StructField("rowId", StringType,true),
StructField("partitionId", StringType,true),
StructField("preComb", LongType,true),
StructField("name", StringType,true),
StructField("versionId", StringType,true),
StructField("toBeDeletedStr", StringType,true),
StructField("intToLong", IntegerType,true),
StructField("longToInt", LongType,true)
))

val data0 = Seq(Row("row_1", "2021/01/01",0L,"bob","v_0","toBeDel0",0,1000000L), 
               Row("row_2", "2021/01/01",0L,"john","v_0","toBeDel0",0,1000000L), 
               Row("row_3", "2021/01/02",0L,"tom","v_0","toBeDel0",0,1000000L))

var dfFromData0 = spark.createDataFrame(data0,schema)

dfFromData0.write.format("hudi").
  options(getQuickstartWriteConfigs).
  option(PRECOMBINE_FIELD_OPT_KEY, "preComb").
  option(RECORDKEY_FIELD_OPT_KEY, "rowId").
  option(PARTITIONPATH_FIELD_OPT_KEY, "partitionId").
  option(TABLE_NAME, tableName).
  option(TABLE_TYPE.key, COW_TABLE_TYPE_OPT_VAL).
  option(OPERATION_OPT_KEY, "upsert").
  option("hoodie.index.type","SIMPLE").
  option("hoodie.datasource.write.hive_style_partitioning","true").
  option("hoodie.datasource.hive_sync.jdbcurl","jdbc:hive2://bigdata01:10000/").
  option("hoodie.datasource.hive_sync.database",databaseName).
  option("hoodie.datasource.hive_sync.table",tableName ).
  option("hoodie.datasource.hive_sync.partition_fields","partitionId").
  option("hoodie.datasource.hive_sync.enable","true").
  mode(Overwrite).
  save(basePath)

验证

查询hive

select * from ods.hudi_cow

成功