hudi spark avro兼容性问题

1. 在学习hudi过程中，参考了：https://hudi.apache.org/docs/quick-start-guide.html，在使用spark shell执行用例：

进入spark shell:

spark-shell \
--packages org.apache.spark:spark-avro_2.11:2.4.6,org.apache.avro:avro:1.8.2 \
--repositories http://maven.aliyun.com/nexus/content/groups/public \
--jars $HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar,/data/opt/hudi/hudi-spark-bundle_2.11-0.6.0.jar \
--conf spark.driver.extraClassPath=$HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \
--conf spark.executor.extraClassPath=$HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'

spark 程序：

import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._

val tableName = "hudi_trips_cow"
val basePath = "file:///tmp/hudi_trips_cow"
val dataGen = new DataGenerator

val inserts = convertToStringList(dataGen.generateInserts(10))
val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
df.write.format("hudi").
  options(getQuickstartWriteConfigs).
  option(PRECOMBINE_FIELD_OPT_KEY, "ts").
  option(RECORDKEY_FIELD_OPT_KEY, "uuid").
  option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
  option(TABLE_NAME, tableName).
  mode(Overwrite).
  save(basePath)

因为我使用的是spark 2.4.6，hudi使用的是0.6.0,这里会有两个兼容性问题：

1）java.lang.NoClassDefFoundError: org/apache/avro/LogicalType

这个是由于当前spark版本使用的avro序列化版本是1.7.7，没有LogicalType这个类，该类在org.apache.avro:avro:1.8.0之后的版本才出现。所以需要升级spark avro的序列化版本。

2）NoSuchMethodError: org.apache.avro.Schema.createUnion

该问题极可能是因为，任务下发到excutor之后，excutor无法找到对应的依赖引起的，使用local执行程序即可。

2. 解决思路升级avro版本，spark shell改成local模式运行

spark-shell \
--master local[2] \
--packages org.apache.spark:spark-avro_2.11:2.4.6,org.apache.avro:avro:1.8.2 \
--repositories http://maven.aliyun.com/nexus/content/groups/public \
--jars $HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar,/data/opt/hudi/hudi-spark-bundle_2.11-0.6.0.jar \
--conf spark.driver.extraClassPath=$HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \
--conf spark.executor.extraClassPath=$HOME/.ivy2/jars/org.apache.avro_avro-1.8.2.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'

也可以将org.apache.avro_avro-1.8.2.jar放到spark集群各个节点lib【或hdfs】下

spark-shell \
--jars /usr/hdp/3.0.1.0-187/spark2/jars/hudi-spark-bundle_2.11-0.6.0.jar \
--conf spark.driver.extraClassPath=/usr/hdp/3.0.1.0-187/spark2/jars/org.apache.avro_avro-1.8.2.jar \
--conf spark.executor.extraClassPath=/usr/hdp/3.0.1.0-187/spark2/jars/org.apache.avro_avro-1.8.2.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'

注意，要写入到hdfs的应用程序：

import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._

val tableName = "hudi_trips_cow"
val basePath = "hdfs:///tmp/hudi_trips_cow" //写入到hdfs
val dataGen = new DataGenerator

val inserts = convertToStringList(dataGen.generateInserts(10))
val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
df.write.format("hudi").
  options(getQuickstartWriteConfigs).
  option(PRECOMBINE_FIELD_OPT_KEY, "ts").
  option(RECORDKEY_FIELD_OPT_KEY, "uuid").
  option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
  option(TABLE_NAME, tableName).
  mode(Overwrite).
  save(basePath)

各位大神，文章写的不好和缺漏的地方，请各位多多指导。谢谢观ka a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a