菜鸟一只,谨代表自己观点,欢迎大家指教
写了一个spark.sql("select usernum from xx.test where day=20210715 group by usernum ").show 居然报java.io.NotSerializableException。非常奇怪的问题。
-
环境:spark2.2.0+jdk1.8
-
前期准备工作:
create table test (usernum string,type string) partitioned by(day string) row format delimited fields terminated by ‘\t’;
alter table test add is not exists partitioned (day=‘20210715’);
load data local inpath ‘/home/1.txt’ into table test; -
代码
spark.sql(“select usernum from test where day=20210715 group by usernum”).show -
报错
Exception in thread "main" org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288) at org.apache.spark.util.ClosureCleaner$.