spark2.3 Json 错误 Incompatible Jackson version: 2.9.2

最新推荐文章于 2021-03-30 19:01:54 发布

光远006

最新推荐文章于 2021-03-30 19:01:54 发布

阅读量3.8k

点赞数 2

分类专栏： spark jackson sql

本文链接：https://blog.csdn.net/Yuanr_/article/details/86524774

版权

spark 同时被 3 个专栏收录

1 篇文章 0 订阅

订阅专栏

jackson

1 篇文章 0 订阅

订阅专栏

sql

1 篇文章 0 订阅

订阅专栏

spark 2.3.1 读取Json文件报错

- 描述
- 解决方案

描述

当我使用Spark官方示例代码代码链接地址
提示如下错误信息

Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.inferFromDataset(JsonDataSource.scala:101)
	at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.infer(JsonDataSource.scala:96)
	at org.apache.spark.sql.execution.datasources.json.JsonDataSource.inferSchema(JsonDataSource.scala:63)
	at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:57)
	at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:202)
	at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:202)
	at scala.Option.orElse(Option.scala:289)
	at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:201)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:392)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
	at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:397)
	at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:340)
	at cn.qphone.spark.areaRoadFlow.TmpGroupConcatDistinctUDAF$.main(TmpGroupConcatDistinctUDAF.scala:65)
	at cn.qphone.spark.areaRoadFlow.TmpGroupConcatDistinctUDAF.main(TmpGroupConcatDistinctUDAF.scala)


	
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.2
	

	at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
	at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
	at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
	at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
	at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

错误信息中被多行空行分割的地方提示jackson.databind版本与 jackson.module版本不匹配，于是我去查询了maven的Jar依赖关系，发现spark 2.3.1中引用了jackson core 2.9.2 在Hbase与sparksql中引用了较低版本的jackson.module
导致jar包冲突

解决方案

手动在Pom里添加对应版本的jackson-module
依赖如下

      <dependency>
          <groupId>com.fasterxml.jackson.module</groupId>
          <artifactId>jackson-module-scala_2.11</artifactId>
          <version>2.9.2</version>
      </dependency>

解决问题！