在使用Spark SQL执行查询时遇到了`java.io.NotSerializableException: org.apache.spark.unsafe.types.UTF8String$IntWrapper`的问题。问题出现在`select usernum from xx.test where day=20210715 group by usernum`的SQL语句中，由于分区字段`day`未加引号，导致聚合操作时出现问题。解决方案是为分区字段值添加引号，如`select usernum from test where day='20210715' group by usernum`。原因可能是SQL中的聚合操作在集群间传输时，由于缺少schema，Spark尝试将`day`字段强制转换，而`UTF8String$IntWrapper`未实现序列化。
菜鸟一只，谨代表自己观点，欢迎大家指教
写了一个spark.sql("select usernum from xx.test where day=20210715 group by usernum ").show 居然报java.io.NotSerializableException。非常奇怪的问题。
-
环境：spark2.2.0+jdk1.8
-
前期准备工作：
create table test (usernum string,type string) partitioned by(day string) row format delimited fields terminated by ‘\t’;
alter table test add is not exists partitioned (day=‘20210715’);
load data local inpath ‘/home/1.txt’ into table test; -
代码
spark.sql(“select usernum from test where day=20210715 group by usernum”).show -
报错
Exception in thread "main" org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288) at org.apache.spark.util.ClosureCleaner$.
最低0.47元/天 解锁文章
1737
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
