spark3.3.2集成hudi同步hive表
环境概述
- hadoop3.x
- hive3.x
- spark3.3.2
- scala2.12
- hudi0.13
spark集成hive
假设我的spark安装目录为
/data/spark
将
hive-site.xml
传入
/data/spark/conf
目录下
spark集成hudi
根据你实际的spark版本和scala版本去找对应版本的jar包
下载jar包
链接: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.3-bundle_2.12/0.13.0/hudi-spark3.3-bundle_2.12-0.13.0.jar
把
hudi-spark3.3-bundle_2.12-0.13.0.jar
传入
/data/spark/jars
目录下
hudi集成hive
把
hudi-hadoop-mr-bundle-0.13.0.jar
传入
$HIVE_HOME/auxlib目录下
我这里是用ambari安装的,目录为
/usr/hdp/3.1.5.0-152/hive/auxlib/
然后重启HIVE!!!
然后重启HIVE!!!
然后重启HIVE!!!
然后重启HIVE!!!重要的事说三遍
通过写入hudi同步到hive表代码实例
1.进入spark-shell
/data/spark/bin/spark-shell \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
2.复制下面的代码
我这里是写入了ods.hudi_cow
路径是/warehouse/hudi/ods/hudi_cow
可以根据你们的实际情况改变
注意:
/warehouse/hudi/ods/hudi_cow
这个路径必须有权限!!!
这个路径必须有权限!!!
这个路径必须有权限!!!
这个路径必须有权限!!!重要的事说三遍
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val tableName = "hudi_cow"
val databaseName= "ods"
val basePath = "/warehouse/hudi/ods/hudi_cow"
val schema = StructType(Array(
StructField("rowId", StringType,true),
StructField("partitionId", StringType,true),
StructField("preComb", LongType,true),
StructField("name", StringType,true),
StructField("versionId", StringType,true),
StructField("toBeDeletedStr", StringType,true),
StructField("intToLong", IntegerType,true),
StructField("longToInt", LongType,true)
))
val data0 = Seq(Row("row_1", "2021/01/01",0L,"bob","v_0","toBeDel0",0,1000000L),
Row("row_2", "2021/01/01",0L,"john","v_0","toBeDel0",0,1000000L),
Row("row_3", "2021/01/02",0L,"tom","v_0","toBeDel0",0,1000000L))
var dfFromData0 = spark.createDataFrame(data0,schema)
dfFromData0.write.format("hudi").
options(getQuickstartWriteConfigs).
option(PRECOMBINE_FIELD_OPT_KEY, "preComb").
option(RECORDKEY_FIELD_OPT_KEY, "rowId").
option(PARTITIONPATH_FIELD_OPT_KEY, "partitionId").
option(TABLE_NAME, tableName).
option(TABLE_TYPE.key, COW_TABLE_TYPE_OPT_VAL).
option(OPERATION_OPT_KEY, "upsert").
option("hoodie.index.type","SIMPLE").
option("hoodie.datasource.write.hive_style_partitioning","true").
option("hoodie.datasource.hive_sync.jdbcurl","jdbc:hive2://bigdata01:10000/").
option("hoodie.datasource.hive_sync.database",databaseName).
option("hoodie.datasource.hive_sync.table",tableName ).
option("hoodie.datasource.hive_sync.partition_fields","partitionId").
option("hoodie.datasource.hive_sync.enable","true").
mode(Overwrite).
save(basePath)
验证
查询hive
select * from ods.hudi_cow
成功