rapidjson读取json文件_SPARK入门 - json文件读取

环境说明

spark.version=2.4.4

scala.version=2.11.12

文件示例

{"name": "Michael",  "age": 12}{"name": "Andy",  "age": 13}{"name": "Justin",  "age": 8, "friend_cnt": 2,"friends": [{"name": "Jobs"}, {"name": "Jacky"}]}

应用代码

object App {  def main(args: Array[String]): Unit = {    import org.apache.spark.sql.SparkSession    import org.apache.spark.storage.StorageLevel    import org.apache.spark.HashPartitioner    import org.apache.spark.Partitioner    val spark = SparkSession.builder().appName("App").master("local[4]").getOrCreate()    spark.sparkContext.setLogLevel("ERROR")    val jsonData = spark.read.json(      "C:bdsparkest3datainputjson"    )        jsonData.createOrReplaceTempView("tmp_view")    jsonData.printSchema()    spark.sql(      "select name,friends[0].name from tmp_view where friend_cnt>0"    ).collect().foreach(println)  }}

运行结果

root

|-- age: long (nullable = true)

|-- friend_cnt: long (nullable = true)

|-- friends: array (nullable = true)

| |-- element: struct (containsNull = true)

| | |-- name: string (nullable = true)

|-- name: string (nullable = true)

[Justin,Jobs]

补充说明

• jsonData是一个DataFrame(org.apache.spark.sql.DataFrame)，简称df

• printSchema()可以输出该df的结构信息

• 可以通过createOrReplaceTempView将df注册为一个临时表，通过用 [element] 指定下标来访问array中的元素，通过.访问struct中的嵌套元素，如friends[0].name