spark处理jsonFile

最新推荐文章于 2024-07-17 15:13:58 发布

白乔

最新推荐文章于 2024-07-17 15:13:58 发布

阅读量9k

点赞数 1

分类专栏：大数据技术与系统

本文链接：https://blog.csdn.net/bluejoe2000/article/details/44174091

版权

大数据技术与系统专栏收录该内容

101 篇文章 2 订阅

订阅专栏

按照spark的说法，这里的jsonFile是特殊的文件：

Note that the file that is offered as jsonFile is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

它是按行分隔多个JSON对象，否则的话就会出错。

以下是一个jsonFile的内容：

scala> val path = "examples/src/main/resources/people.json"
path: String = examples/src/main/resources/people.json

scala> Source.fromFile(path).foreach(print)
{"name":"Michael"}
{"name":"Andy", "age":30}
{"name":"Justin", "age":19}

可以获取到一个SchemaRDD：

scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
scala> val jsonFile = sqlContext.jsonFile(path)
scala> jsonFile.printSchema()
root
 |-- age: integer (nullable = true)
 |-- name: string (nullable = true)

针对该SchemaRDD可以做遍历操作：

jsonFile.filter(row=>{val age=row(0).asInstanceOf[Int];age>=13&&age<=19}).collect

既然是SchemaRDD，就可以采用SQL：

scala> jsonFile.registerTempTable("people")
scala> val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
scala> teenagers.foreach(println)