Spark2升级为Spark3 UDF org.apache.spark.sql.AnalysisException: You‘re using untyped报错解决方案

已于 2023-02-03 16:28:18 修改
阅读量1k 收藏 2
点赞数 1
分类专栏： Spark 文章标签： scala spark 大数据
于 2023-01-31 15:46:14 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_39285950/article/details/128818502
版权

最开始spark版本为2.4.3,因业务需要升级为3.1.0,版本升级时踩了几个坑,将这些错误记录下来,供大家参考

1.udf返回类型问题

spark2.4.3版本中需要我们指定该方法的返回类型,但正因为这个, 导致在Spark3中报错

def udfGenPartitionWindow(features: Array[FeatureInfo]): UserDefinedFunction = {
    udf((featureId: String, timestamp: Long) => genPartitionWindow(features, featureId, timestamp), ArrayType(StringType))
  }

报错信息:

Exception in thread "main" org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could:
1. use typed Scala UDF APIs(without return type parameter), e.g. `udf((x: Int) => x)`
2. use Java UDF APIs, e.g. `udf(new UDF1[String, Integer] { override def call(s: String): Integer = s.length() }, IntegerType)`, if input types are all non primitive
3. set spark.sql.legacy.allowUntypedScalaUDF to true and use this API with caution;

从报错信息中,我们可以看出,官方不再建议我们自己添加返回类型,如果添加了则会报错,并且提供了三种解决方法

1.scala UDF方式

将返回类型直接删除即可

def udfGenPartitionWindow(features: Array[FeatureInfo]): UserDefinedFunction = {
    udf((featureId: String, timestamp: Long) => genPartitionWindow(features, featureId, timestamp))
  }

然后会报错:

Caused by: java.io.NotSerializableException: ...
Serialization stack:
        - object not serializable (class: ..., value: ...$ConversionUtils$@63ed11dd)
        - field (class: ...$ConversionUtils$$anonfun$3, name: $outer, type: class ...$ConversionUtils$)
        - object (class ...$ConversionUtils$$anonfun$3, <function1>)
        - field (class: org.apache.spark.sql.catalyst.expressions.ScalaUdf$$anonfun$2, name: func$2, type: interface scala.Function1)
        - object (class org.apache.spark.sql.catalyst.expressions.ScalaUdf$$anonfun$2, <function1>)
        - field (class: org.apache.spark.sql.catalyst.expressions.ScalaUdf, name: f, type: interface scala.Function1)
        - object (class org.apache.spark.sql.catalyst.expressions.ScalaUdf, scalaUDF(ts_str#2683))
        - field (class: org.apache.spark.sql.catalyst.expressions.Alias, name: child, type: class org.apache.spark.sql.catalyst.expressions.Expression)
        - object (class org.apache.spark.sql.catalyst.expressions.Alias, scalaUDF(ts_str#2683) AS ts#7146)
        - element of array (index: 35)
        - array (class [Ljava.lang.Object;, size 36)
        - field (class: scala.collection.mutable.ArrayBuffer, name: array, type: class [Ljava.lang.Object;)
        - object (class scala.collection.mutable.ArrayBuffer,.....

这里我们直接将此类继承 Serializable即可

object UdfUtils extends Serializable {...}

到此,问题解决

2. java UDF方式(略)

3. 设置参数方式

sparkSession.sql("set spark.sql.legacy.allowUntypedScalaUDF=true")

2.读取多namespace表

因为业务需求,表名前有两个namespace空间命名, 但是spark3不支持这种方式

select *
from udw_ns.default.test_table

报错信息:

User class threw exception: org.apache.spark.sql.AnalysisException: spark_catalog requires a single-part namespace, but got [udw_ns, default]
详情:
[23-01-31 10:42:44.542][INFO][main][Client][Logging.scala:57] 
     client token: N/A
     diagnostics: User class threw exception: org.apache.spark.sql.AnalysisException: spark_catalog requires a single-part namespace, but got [udw_ns, default]
    at org.apache.spark.sql.errors.QueryCompilationErrors$.requiresSinglePartNamespaceError(QueryCompilationErrors.scala:850)
    at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog$TableIdentifierHelper.asTableIdentifier(V2SessionCatalog.scala:192)
    at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadTable(V2SessionCatalog.scala:65)
    at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:281)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.loaded$lzycompute$1(Analyzer.scala:1250)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.loaded$1(Analyzer.scala:1250)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupRelation$3(Analyzer.scala:1287)
    at scala.Option.orElse(Option.scala:447)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupRelation(Analyzer.scala:1286)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$15.applyOrElse(Analyzer.scala:1204)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$15.applyOrElse(Analyzer.scala:1167)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$2(AnalysisHelper.scala:135)
    at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1122)
    at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1121)
    at org.apache.spark.sql.catalyst.plans.logical.OrderPreservingUnaryNode.mapChildren(LogicalPlan.scala:206)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:135)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$2(AnalysisHelper.scala:135)
    at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1122)
    at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1121)
    at org.apache.spark.sql.catalyst.plans.logical.OrderPreservingUnaryNode.mapChildren(LogicalPlan.scala:206)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:135)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130)
    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:30)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1167)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1133)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
    at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
    at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
    at scala.collection.immutable.List.foldLeft(List.scala:91)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:222)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:218)
    at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:167)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:218)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:182)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:203)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:202)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:75)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:183)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:183)
    at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:75)
    at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:73)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:65)
    at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
    at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
    at com.baidu.anti.module.process.impl.Extractor.udwMartLogExtract(Extractor.scala:145)
    at com.baidu.anti.module.process.impl.Extractor.getLogData(Extractor.scala:125)
    at com.baidu.anti.module.process.impl.Extractor.run(Extractor.scala:68)
    at com.baidu.anti.runner.OfflineThemis$.runJob(OfflineThemis.scala:64)
    at com.baidu.anti.runner.OfflineThemis$.main(OfflineThemis.scala:32)
    at com.baidu.anti.runner.OfflineThemis.main(OfflineThemis.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)

对于此种办法, 可以尝试在表后添加 .history

select *
from udw_ns.default.test_table.history

也可去尝试 iceberg 方式.

Souvenirser
关注
专栏目录
Spark 2.2.1 SQL UDAF用户自定义函数案例
段智华的博客
02-21 1449
Spark 2.2.1 SQL UDAF用户自定义函数案例UDAF：User Defined Aggregation Function，用户自定义的聚合函数，函数本身作用于数据集合，能够在聚合操作的基础上进行自定义操作。例如：UDF会被SparkSQL中的Catalyst封装成为Expression，最终会通过eval方法来计算输入的数据Row。UDAF有大量的Aggregation之类的操作，对...
理解Spark中SparkSQL模块基础使用
杨鑫newlife的专栏
04-30 273
1.Starting Point: SparkSession Spark中所有功能的入口点是SparkSession类。 要创建一个基本的SparkSession，只需使用SparkSession.builder(): 例子： import org.apache.spark.sql.SparkSession var spark = SparkSession.builder().ap...
spark2升级spark3问题记录20230506
weixin_47120324的博客
05-06 449
spark2.3.2使用scala2.11.8版本，spark3.1.1使用scala2.12.14版本，sacla低版本打包导致，推荐切换scala2.12.10版本进行打包。①将commons-configuration-1.6.jar放入spark3/jars目录，可使代码依赖的低版本类找到。②一劳永逸办法：针对新版本commons-configuration2-2.1.1.jar进行适配改造。spark3集群运行依赖jars包含commons-configuration2-2.1.1.jar。
Spark 2 迁移 Spark 3 参考手册
最新发布
weixin_47499396的博客
08-07 1027
Apache Spark是一个广泛应用于大规模数据处理的开源统一分析引擎。自发布以来，它已经成为大数据处理的事实标准。2020年发布的Spark 3.0带来了许多新特性和改进，极大地提升了性能和易用性。如果你习惯使用Spark 2，那么了解新版本的变化将非常有帮助。 本文将重点介绍Spark 2和Spark 3在语法上的主要差异，帮助开发者顺利过渡到新版本。希望这篇文章能为你在Spark 3上的开发提供一些有用的参考。
Spark2.X 升级spark3.X 遇到的问题
weixin_43181992的博客
11-26 4327
java.lang.NoSuchFieldError: JAVA_9 问题：这个错误主要是因为Spark-3.x会依赖commons-lang3这个包 解决方法：普通的解决方法就不说了，解决冲突，缺少commons-lang3的加上去等等 如果还不行，就检查下是否有hive-exec这个依赖的存在，打开它你会发现它里面竟然也有commons-lang3，但是它的JavaVersion这个类竟然不一样！！！ 如果是上面的问题，排除掉hive-exec就好了 <!-- https://mvnrepos.
SparkSQL学习 （二）SparkSQL的自定义函数UDF
potter
05-04 1137
在Spark中，也支持Hive中的自定义函数。自定义函数大致可以分为三种：UDF(User-Defined-Function)，即最基本的自定义函数，类似to_char,to_date等UDAF（User- Defined Aggregation Funcation），用户自定义聚合函数，类似在group by之后使用的sum,avg等UDTF(User-Defined Table-Generat...
Spark.3.0最新版UDF实现几何平均数
weixin_48109576的博客
08-21 340
package com._51doit.spark.SQLUDF import java.lang import org.apache.log4j.{Level, Logger} import org.apache.spark.sql.expressions.Aggregator import org.apache.spark.sql.{Dataset, Encoder, Encoders, SparkSession} object UDAFGeo_Mean { /** * Author:.
spark sql 异常
liuxiaoer1的专栏
08-25 1293
1.spark SQL 测试过程中报如下错误org.apache.spark.sql.AnalysisException -- 20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg start 20/08/25 11:42:08 INFO Client: resolveAppExceptionMsg, msg:User class threw exception: org.apache.spark.sql.AnalysisExce...
Spark SQL 中org.apache.spark.sql.functions归纳
xiaohu21的博客
10-06 2429
Spark SQL 中org.apache.spark.sql.functions归纳 注意，这里使用的是scala 2.12.12，spark版本是最新的3.0.1版本 1. Sort functions /** * Returns a sort expression based on ascending order of the column. * {{{ * df.sort(asc("dept"), desc("age")) * }}} * * @group s
大数据重点面试知识点总结
weixin_44961626的博客
02-15 1343
Linux 1、常见命令 2、linux的启动顺序 通电后读取ROM的BIOS程序进行硬件自检，自检成功后把计算机控制权交给BIOS中BOOTsequence中的下一个有效设备，读取该设备MBR找到操作系统，载入linux的bootloader，一般是grub。之后载入kernel，执行 /etc/rc.d/sysinit ，开启其他组件（/etc/modules.conf），执行运行级别，执行/...
SparkSQL初级(聚合，Parquet,JSON,JDBC,Hive表)
南国的技术栈
12-15 851
紧接着系列博客上一篇Spark05-SparkSQL入门 的学习，这篇博客 主要讲解一些SparkSQL初级使用。 聚合（Aggregations） 内置的DataFrames函数提供常见的聚合，如count()、countDistinct()、avg()、max()、min()等。此外，用户并不局限于预定义的聚合函数，还可以创建自己的聚合函数。 无用户定义的聚合函数(Untyped User-D...
Exception in thread “main“ java.lang.ClassNotFoundException: org.apache.spark.sql.adapter.Spark3_1Ad
m0_74302467的博客
06-24 493
将对应版本的jar包放入spark/jars下即可。使用spark写hudi时报错。
org.apache.spark.sql.adapter.Spark3Adapter 问题以及spark-hudi提交yarn脚本
LZX的博客
02-07 3386
问题描述： sparkstructedstreaming 将kafka数据写入hudi 并与hive同步 执行报错如下： Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.adapter.Spark3Adapter 解决办法： 问题显示没有发现适配器这个包？ spark也没这个包啊？ 查看启动脚本，忘记添加hudi --jars了… --jars /opt/soft/hudi/hudi-0.9.0/packaging/hu
CDH | Spark升级
justlpf的专栏
02-07 1293
参考文章：CDH| Spark升级 目录 升级之Spark升级 升级过程 1 离线包下载 2 离线包上传 页面操作 命令行查看命令 升级之Spark升级 在CDH5.12.1集群中，默认安装的Spark是1.6版本，这里需要将其升级为Spark2.1版本。经查阅官方文档，发现Spark1.6和2.x是可以并行安装的，也就是说可以不用删除默认的1.6版本，可以直接安装2.x版本，它们各自用的端口也是不一样的。 Cloudera发布Apache Spark 2概述(可以在这里面找到安装方..
Spark UDF实例详解
hiscoming的博客
05-27 354
Spark UDF实例详解 需求List: (1) 统计orders中produce的数量统计 – 商品被购买的数据量 hive : group by count spark: scala> val orders=sql(“select * from badou.orders”) scala> val products=sql(“select * from badou.products”) scala> val priors=sql(“select * from badou.priors”
Spark sql 自定义函数(UDF、UDTF、UDAF）Spark版本 3.0之前 AND 3.0之后
weixin_44824330的博客
04-02 633
Spark自定义UDF、UDAF、UDTF 函数
org.apache.spark.sql.AnalysisException: Table or view not found: testtable
LAFEU的专栏
04-16 1848
问题 User class threw exception: org.apache.spark.sql.AnalysisException: Table or view not found: testtable; line 1 ； 解决办法，引入--files /data0/spark/spark-2.2.1-bin/conf/hive-site.xml ...
User class threw exception:org.apache.spark.sql.streaming
半丶晨曦的博客
07-14 905
hudi整合hive提交到yarn报错 User class threw exception: org.apache.spark.sql.streaming.StreamingQueryException: Following instants have timestamps >= compactionInstant (20210710211157) Instants :[[20210713205004__deltacommit__COMPLETED], [20210713205200__deltac
spark2.x学习笔记：spark sql的sql
03-16
Spark SQL是Spark的一个模块，它提供了一种基于结构化数据的编程接口。Spark SQL支持使用SQL语句来查询和操作数据，这使得它非常适合处理结构化数据。Spark SQL支持多种数据源，包括Hive、JSON、Parquet、Avro等。Spark SQL还支持使用DataFrame API来进行编程，这使得它可以与其他Spark组件无缝集成。Spark SQL的SQL语法与标准SQL语法非常相似，因此对于熟悉SQL的开发人员来说，学习Spark SQL非常容易。Spark SQL还支持使用用户自定义函数（UDF）来扩展其功能。总之，Spark SQL是一个非常强大的工具，可以帮助我们轻松地处理和分析结构化数据。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值