SparkSql读取json文件指定与不指定schema的区别

最新推荐文章于 2024-07-23 12:57:59 发布

Eunicema

最新推荐文章于 2024-07-23 12:57:59 发布

阅读量1.4k

点赞数 1

文章标签： spark

本文链接：https://blog.csdn.net/Eunicema/article/details/115861290

版权

sparksql 可以读取json文件，转换成dataframe,并且可以自行推断schema；如果显式的提供schema可以避免额外的扫描判断。

数据准备

{"name":"jack","age":12,"address":"昌平区东小","location":[116.4021289,40.05688698]}
{"name":"lili","age":22,"address":"昌平区回龙","location":[116.3412241, 40.07942604]}
{"name":"cc","age":11,"address":"昌平区北七","location":[116.4179459,40.11644403]}
{"name":"vv","age":13,"address":"昌平区阳坊","location":[116.1332611,40.13701398]}

1、直接读取json文件

    val spark = SparkSession.builder.appName("spark read")
      .master("local[2]").getOrCreate()

    val df = spark.read.json("D:\\IdeaProjects\\test\\src\\main\\resources\\sparkjson.txt")
    df.show()
    df.printSchema()

运行结果如下：