pivotMaxValues报错

最新推荐文章于 2021-07-31 19:06:45 发布

孤数不证

最新推荐文章于 2021-07-31 19:06:45 发布

阅读量413

点赞数

本文链接：https://blog.csdn.net/weixin_43668299/article/details/106998759

版权

博客内容介绍了在遇到pivotMaxValues错误时，从错误操作、具体错误到解决方法的全过程，包括在配置文件中添加相关设置的建议。

摘要由CSDN通过智能技术生成

pivotMaxValues报错

1、出现错误的操作

在列转行且用指定的列的值填充时报错，且列转行的字段个数超过10000个；

2、具体错误

Exception in thread "main" org.apache.spark.sql.AnalysisException: The     pivot column field_name has more than 10000 distinct values, this cou
ld indicate an error. If this was intended, set spark.sql.pivotMaxValues to at least the number of distinct values of the pivot column.;
    at org.apache.spark.sql.RelationalGroupedDataset.pivot(RelationalGroupedDataset.scala:327)
    at com.rong360.featureAnalyse.FeatherAnalyseOnlineStep.getPivotData(FeatherAnalyseOnlineStep.java:564)
    at com.rong360.featureAnalyse.FeatherAnalyseOnlineStep.main(FeatherAnalyseOnlineStep.java:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

3、解决方法

 SparkSession spark = SparkSession.builder()
			.config( "spark.driver.maxResultSize","40g")
			.config("spark.sql.pivotMaxValues", 20000)
			.appName("Features analyse step")
			.enableHiveSupport()
			.getOrCreate();

当然也可以在配置文件中加；

原文链接：https://blog.csdn.net/jeffiny/article/details/80191800