spark读取文件异常 AnalysisException: Since Spark 2.3, the queries from raw JSON/CSV files are disallowed

最新推荐文章于 2024-03-21 20:45:00 发布

芮呆萌061511332

最新推荐文章于 2024-03-21 20:45:00 发布

阅读量810

点赞数

分类专栏： bug异常大数据

本文链接：https://blog.csdn.net/RUIMENG061511332/article/details/119639105

版权

bug异常同时被 2 个专栏收录

21 篇文章 1 订阅

订阅专栏

大数据

20 篇文章 2 订阅

订阅专栏

博客内容讲述了在使用Spark读取JSON文件时遇到的AnalysisException异常，问题根源在于文件中的中文字符不是UTF-8编码导致。解决方案是将文件转换为UTF-8格式。提醒读者注意Windows新建文本文件默认编码可能是ANSI，而Notepad++新建文件默认为UTF-8。

摘要由CSDN通过智能技术生成

spark读取json文件异常：

Exception in thread "main" org.apache.spark.sql.AnalysisException: Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column
(named _corrupt_record by default). For example:
spark.read.schema(schema).json(file).filter($"_corrupt_record".isNotNull).count()
and spark.read.schema(schema).json(file).select("_corrupt_record").show().
Instead, you can cache or save the parsed results and then send the same query.
For example, val df = spark.read.schema(schema).json(file).cache() and then
df.filter($"_corrupt_record".isNotNull).count().;
	at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.buildReader(JsonFileFormat.scala:120)
	at org.apache.spark.sql.execution.datasources.FileFormat$class.buildReaderWithPartitionValues(FileFormat.scala:129)
	at org.apache.spark.sql.execution.datasources.TextBasedFileFormat.buildReaderWithPartitionValues(FileFormat.scala:165)
	at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD$lzycompute(DataSourceScanExec.scala:309)
	at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD(DataSourceScanExec.scala:305)