spark读取json文件

最新推荐文章于 2023-10-11 10:47:37 发布
最新推荐文章于 2023-10-11 10:47:37 发布
阅读量481 收藏 3
点赞数
文章标签： filter mysql sql 数据库 spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zhens1mple/article/details/109967830
版权

 JSON文件

1593136280858|{"cm":{"ln":"-55.0","sv":"V2.9.6","os":"8.0.4","g":"C6816QZ0@gmail.com","mid":"489","nw":"3G","l":"es","vc":"4","hw":"640*960","ar":"MX","uid":"489","t":"1593123253541","la":"5.2","md":"sumsung-18","vn":"1.3.4","ba":"Sumsung","sr":"I"},"ap":"app","et":[{"ett":"1593050051366","en":"loading","kv":{"extend2":"","loading_time":"14","action":"3","extend1":"","type":"2","type1":"201","loading_way":"1"}},{"ett":"1593108791764","en":"ad","kv":{"activityId":"1","displayMills":"78522","entry":"1","action":"1","contentType":"0"}},{"ett":"1593111271266","en":"notification","kv":{"ap_time":"1593097087883","action":"1","type":"1","content":""}},{"ett":"1593066033562","en":"active_background","kv":{"active_source":"3"}},{"ett":"1593135644347","en":"comment","kv":{"p_comment_id":1,"addtime":"1593097573725","praise_count":973,"other_id":5,"comment_id":9,"reply_count":40,"userid":7,"content":"辑赤蹲慰鸽抿肘捎"}}]}
1593136280858|{"cm":{"ln":"-114.9","sv":"V2.7.8","os":"8.0.4","g":"NW0S962J@gmail.com","mid":"490","nw":"3G","l":"pt","vc":"8","hw":"640*1136","ar":"MX","uid":"490","t":"1593121224789","la":"-44.4","md":"Huawei-8","vn":"1.0.1","ba":"Huawei","sr":"O"},"ap":"app","et":[{"ett":"1593063223807","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"1","type1":"102","loading_way":"1"}},{"ett":"1593095105466","en":"ad","kv":{"activityId":"1","displayMills":"1966","entry":"3","action":"2","contentType":"0"}},{"ett":"1593051718208","en":"notification","kv":{"ap_time":"1593095336265","action":"2","type":"3","content":""}},{"ett":"1593100021275","en":"comment","kv":{"p_comment_id":4,"addtime":"1593098946009","praise_count":220,"other_id":4,"comment_id":9,"reply_count":151,"userid":4,"content":"抄应螟皮釉倔掉汉蛋蕾街羡晶"}},{"ett":"1593105344120","en":"praise","kv":{"target_id":9,"id":7,"type":1,"add_time":"1593098545976","userid":8}}]}

操作步骤

1.val fileRDD=sc.textFile("hdfs://192.168.28.70:9000/kb09file/op.log")

fileRDD.collect.foreach(println)
1593136280858|{"cm":{"ln":"-55.0","sv":"V2.9.6","os":"8.0.4","g":"C6816QZ0@gmail.com","mid":"489","nw":"3G","l":"es","vc":"4","hw":"640*960","ar":"MX","uid":"489","t":"1593123253541","la":"5.2","md":"sumsung-18","vn":"1.3.4","ba":"Sumsung","sr":"I"},"ap":"app","et":[{"ett":"1593050051366","en":"loading","kv":{"extend2":"","loading_time":"14","action":"3","extend1":"","type":"2","type1":"201","loading_way":"1"}},{"ett":"1593108791764","en":"ad","kv":{"activityId":"1","displayMills":"78522","entry":"1","action":"1","contentType":"0"}},{"ett":"1593111271266","en":"notification","kv":{"ap_time":"1593097087883","action":"1","type":"1","content":""}},{"ett":"1593066033562","en":"active_background","kv":{"active_source":"3"}},{"ett":"1593135644347","en":"comment","kv":{"p_comment_id":1,"addtime":"1593097573725","praise_count":973,"other_id":5,"comment_id":9,"reply_count":40,"userid":7,"content":"辑赤蹲慰鸽抿肘捎"}}]}
1593136280858|{"cm":{"ln":"-114.9","sv":"V2.7.8","os":"8.0.4","g":"NW0S962J@gmail.com","mid":"490","nw":"3G","l":"pt","vc":"8","hw":"640*1136","ar":"MX","uid":"490","t":"1593121224789","la":"-44.4","md":"Huawei-8","vn":"1.0.1","ba":"Huawei","sr":"O"},"ap":"app","et":[{"ett":"1593063223807","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"1","type1":"102","loading_way":"1"}},{"ett":"1593095105466","en":"ad","kv":{"activityId":"1","displayMills":"1966","entry":"3","action":"2","contentType":"0"}},{"ett":"1593051718208","en":"notification","kv":{"ap_time":"1593095336265","action":"2","type":"3","content":""}},{"ett":"1593100021275","en":"comment","kv":{"p_comment_id":4,"addtime":"1593098946009","praise_count":220,"other_id":4,"comment_id":9,"reply_count":151,"userid":4,"content":"抄应螟皮釉倔掉汉蛋蕾街羡晶"}},{"ett":"1593105344120","en":"praise","kv":{"target_id":9,"id":7,"type":1,"add_time":"1593098545976","userid":8}}]}

2. val jsonStrRDD=fileRDD.map(x=>x.split('|')).map(x=>(x(0),x(1)))
val jsonRDD=jsonStrRDD.map(x=>{val jsonStr=x._2;val jsonStr1=jsonStr.substring(0,jsonStr.length-1); jsonStr1+",\"id\":\""+x._1+"\"}"})

 jsonRDD.collect.foreach(println)
{"cm":{"ln":"-55.0","sv":"V2.9.6","os":"8.0.4","g":"C6816QZ0@gmail.com","mid":"489","nw":"3G","l":"es","vc":"4","hw":"640*960","ar":"MX","uid":"489","t":"1593123253541","la":"5.2","md":"sumsung-18","vn":"1.3.4","ba":"Sumsung","sr":"I"},"ap":"app","et":[{"ett":"1593050051366","en":"loading","kv":{"extend2":"","loading_time":"14","action":"3","extend1":"","type":"2","type1":"201","loading_way":"1"}},{"ett":"1593108791764","en":"ad","kv":{"activityId":"1","displayMills":"78522","entry":"1","action":"1","contentType":"0"}},{"ett":"1593111271266","en":"notification","kv":{"ap_time":"1593097087883","action":"1","type":"1","content":""}},{"ett":"1593066033562","en":"active_background","kv":{"active_source":"3"}},{"ett":"1593135644347","en":"comment","kv":{"p_comment_id":1,"addtime":"1593097573725","praise_count":973,"other_id":5,"comment_id":9,"reply_count":40,"userid":7,"content":"辑赤蹲慰鸽抿肘捎"}}],"id":"1593136280858"}
{"cm":{"ln":"-114.9","sv":"V2.7.8","os":"8.0.4","g":"NW0S962J@gmail.com","mid":"490","nw":"3G","l":"pt","vc":"8","hw":"640*1136","ar":"MX","uid":"490","t":"1593121224789","la":"-44.4","md":"Huawei-8","vn":"1.0.1","ba":"Huawei","sr":"O"},"ap":"app","et":[{"ett":"1593063223807","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"1","type1":"102","loading_way":"1"}},{"ett":"1593095105466","en":"ad","kv":{"activityId":"1","displayMills":"1966","entry":"3","action":"2","contentType":"0"}},{"ett":"1593051718208","en":"notification","kv":{"ap_time":"1593095336265","action":"2","type":"3","content":""}},{"ett":"1593100021275","en":"comment","kv":{"p_comment_id":4,"addtime":"1593098946009","praise_count":220,"other_id":4,"comment_id":9,"reply_count":151,"userid":4,"content":"抄应螟皮釉倔掉汉蛋蕾街羡晶"}},{"ett":"1593105344120","en":"praise","kv":{"target_id":9,"id":7,"type":1,"add_time":"1593098545976","userid":8}}],"id":"1593136280858"}

3.val jsonDF=jsonRDD.toDF
val jsonDF2=jsonDF.select(get_json_object($"value","$.cm").alias("cm")
,get_json_object($"value","$.ap").alias("ap")
,get_json_object($"value","$.et").alias("et")
,get_json_object($"value","$.id").alias("id"))

jsonDF2.show
+--------------------+---+--------------------+-------------+
|                  cm| ap|                  et|           id|
+--------------------+---+--------------------+-------------+
|{"ln":"-55.0","sv...|app|[{"ett":"15930500...|1593136280858|
|{"ln":"-114.9","s...|app|[{"ett":"15930632...|1593136280858|
+--------------------+---+--------------------+-------------+

4.val jsonDF3=jsonDF2.select(get_json_object($"cm","$.ln").alias("ln"),get_json_object($"cm","$.sv").alias("sv")
,get_json_object($"cm","$.os").alias("os"),get_json_object($"cm","$.g").alias("g")
,get_json_object($"cm","$.mid").alias("mid"),get_json_object($"cm","$.nw").alias("nw")
,get_json_object($"cm","$.l").alias("l"),get_json_object($"cm","$.vc").alias("vc")
,get_json_object($"cm","$.hw").alias("hw"),get_json_object($"cm","$.ar").alias("ar")
,get_json_object($"cm","$.uid").alias("uid"),get_json_object($"cm","$.t").alias("t")
,get_json_object($"cm","$.la").alias("la"),get_json_object($"cm","$.md").alias("md")
,get_json_object($"cm","$.vn").alias("vn"),get_json_object($"cm","$.ba").alias("ba")
,get_json_object($"cm","$.sr").alias("sr"),$"et",$"id",$"ap"
)

 jsonDF3.show
+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+-------------+---+
|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|                  et|           id| ap|
+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+-------------+---+
| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[{"ett":"15930500...|1593136280858|app|
|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[{"ett":"15930632...|1593136280858|app|
+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+-------------+---+

5.val jsonDF4 = jsonDF3.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",from_json($"et",ArrayType(StructType(StructField("ett",StringType)::StructField("en",StringType)::StructField("kv",StringType)::Nil))).alias("event"))
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|               event|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[[1593050051366, ...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[[1593063223807, ...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+

6.val josnDF5 = jsonDF4.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",explode($"event").alias("event"))
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|               event|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[1593050051366, l...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[1593108791764, a...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[1593111271266, n...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[1593066033562, a...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[1593135644347, c...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[1593063223807, l...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[1593095105466, a...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[1593051718208, n...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[1593100021275, c...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[1593105344120, p...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+


7.val jsonDF6 = josnDF5.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"event.ett",$"event.en",$"event.kv")
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-----------------+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|          ett|               en|                  kv|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-----------------+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593050051366|          loading|{"extend2":"","lo...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593108791764|               ad|{"activityId":"1"...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593111271266|     notification|{"ap_time":"15930...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593066033562|active_background|{"active_source":...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593135644347|          comment|{"p_comment_id":1...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593063223807|          loading|{"extend2":"","lo...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593095105466|               ad|{"activityId":"1"...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593051718208|     notification|{"ap_time":"15930...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593100021275|          comment|{"p_comment_id":4...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593105344120|           praise|{"target_id":9,"i...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-----------------+--------------------+

8.val loadingDF=jsonDF6.filter(jsonDF6("en")==="loading").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.extend2").alias("extend2"),get_json_object($"kv","$.loading_time").alias("loading_time"),get_json_object($"kv","$.action").alias("action"),get_json_object($"kv","$.extend1").alias("extend1"),get_json_object($"kv","$.type").alias("type"),get_json_object($"kv","$.type1").alias("type1"),get_json_object($"kv","$.loading_way").alias("loading_way"))

9.val adDF=jsonDF6.filter(jsonDF6("en")==="ad").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.activityId").alias("activityId"),get_json_object($"kv","$.displayMills").alias("displayMills"),get_json_object($"kv","$.entry").alias("entry"),get_json_object($"kv","$.action").alias("action"),get_json_object($"kv","$.contentType").alias("contentType"))

10.val notificationDF=jsonDF6.filter(jsonDF6("en")==="notification").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.ap_time").alias("ap_time"),get_json_object($"kv","$.action").alias("action"),get_json_object($"kv","$.type").alias("type"),get_json_object($"kv","$.content").alias("content"))

11.val active_backgroundDF=jsonDF6.filter(jsonDF6("en")==="active_background").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.active_source").alias("active_source"))

12.val commentDF=jsonDF6.filter(jsonDF6("en")==="comment").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.p_comment_id").alias("p_comment_id"),get_json_object($"kv","$.addtime").alias("addtime"),get_json_object($"kv","$.praise_count").alias("praise_count"),get_json_object($"kv","$.other_id").alias("other_id"),get_json_object($"kv","$.comment_id").alias("comment_id"),get_json_object($"kv","$.reply_count").alias("reply_count"),get_json_object($"kv","$.userid").alias("userid"),get_json_object($"kv","$.content").alias("content"))

13.val praiseDF=jsonDF6.filter(jsonDF6("en")==="praise").select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.target_id").alias("target_id"),get_json_object($"kv","$.id").alias("id"),get_json_object($"kv","$.type").alias("type"),get_json_object($"kv","$.add_time").alias("add_time"),get_json_object($"kv","$.userid").alias("userid"))

 

zhenS1mple
关注
spark 读取json解析Schema问题汇总
yy的博客
05-04 445
问题1: json中 kv的v为null,则上游可能不会在json里写这个kv问题2: 如果某个v都是null 那么spark.read.json解析出来就认为是StringType 如果你下游是Array之类的,就会导致后续报错问题3: spark2普通模式 spark.read.schema(...).json 读取json并手动指定schema 如果某行json中某列数据类型和schema不对应,无法解析,那么该行所有数据列都会被解析为null。
spark中读取json_【spark】文件读写和JSON数据解析
weixin_33773670的博客
02-11 3293
1.读文件通过sc.textFile(“file://")方法来读取文件到rdd中。val lines = sc.textFile("file://")//文件地址或者HDFS文件路径本地地址"file:///home/hadoop/spark-1.6.0-bin-hadoop2.6/examples/src/main/resources/people.json"HDFS文件地址"hdfs://...
Spark--读取JSON文件
最新发布
Natsu_natsu的博客
10-11 535
spark读取json
使用pyspark解析json文件，并将统计结果写入InfluxDB中
06-20
使用pyspark解析json文件，并将统计结果写入InfluxDB中
spark操作json数据
盛源的博客
08-09 1万+
一、转换到JSON格式2. spark-sql转json串1. 把某些列封装到json串2. 把所有列封装到json串二、 特殊字符1. key中含有特殊字符三、数组操作四、 spark读取json文件1. dataframe指定schema读取。
pyspark读取json文件中的内容
微电子学与固体电子学-俞驰
07-20 6138
  代码如下： #-*- coding:utf- -*- import sys reload(sys) sys.setdefaultencoding('utf-8') from pyspark import SparkConf,SparkContext from pyspark.sql import SQLContext conf=SparkConf().setAppName("spark_...
spark高级操作之json文件的分析处理
weixin_49165958的博客
11-23 903
步骤一，上传一篇json文件到hdfs，文件内容可以用百度搜素：json文件在线解析 如下： hdfs dfs -put /opt/kb09file/op.log /kb09file 步骤二：，分析json文件，按照“|”切割，分成两部分 步骤三：对jsonStrRDD的第二部分进行截取（substring）操作 val jsonRDD=jsonStrRDD.map(x=>{var jsonStr=x._2;jsonStr= jsonStr.substring(0,jsonStr.length-
Spark SQL操作JSON字段的小技巧
09-09
Spark SQL是一款强大的大数据处理工具，它提供了对JSON数据的内置支持，使得在处理JSON格式的数据时更加便捷。本文将详细介绍Spark SQL操作JSON字段的几个关键函数：get_json_object、from_json 和 to_json，以及...
spark中读取json_【Spark】使用DataFrame读取复杂JSON中的嵌套数组
weixin_32821579的博客
12-29 1104
众所周知，在早期Spark版本中就已经支持读取Json格式的数据文件，并能够直接转换为数据库表，以方便我们进行处理数据，在本篇文章中我们将介绍如何通过Spark API很简单地读取Json数据，并进一步讲解，读取复杂Json中的嵌套数组。本次使用的各个组件版本设定如下：Spark： 2.1.0Scala 2.11.8Hadoop 2.6.2加载Json数据我们可以简单地通过SQLContext读取...
Spark读取json格式文件
热门推荐
墨玉浮白的博客
07-30 1万+
一、普通json格式 val session = SparkSession.builder().appName("sql").master("local").getOrCreate() val df = session.read.format("json").load("./data/json") 二、读取嵌套的json格式文件 /** * 格式： * {"name":"zhangs...
Spark SQL数据源：JSON数据集
梁辰兴的博客
06-14 2566
Spark SQL可以自动推断JSON文件的Schema，并将其加载为DataFrame。在加载和写入JSON文件时，除了可以使用load()方法和save()方法外，还可以直接使用Spark SQL内置的json()方法。该方法不仅可以读写JSON文件，还可以将Dataset[String]类型的数据集转为DataFrame。需要注意的是，要想成功地将一个JSON文件加载为DataFrame，JSON文件的每一行必须包含一个独立有效的JSON对象，而不能将一个JSON对象分散在多行。
Spark-SQL问题-读取json文件
chbxw
04-01 381
由于源数据只有一种结构，spark-sql字段推断结构，没有bc。取值不同，data中字段结构不同。有三种json文件, 根据字段。将三种数据同步到ods层。
spark读取json并输出
芒果summer小仙女
05-11 602
package com.baidu.buildtable import org.apache.spark.sql.SparkSession //import org.apache.hadoop.hive.ql.exec.spark.session.SparkSession object SparHiv { def main(args: Array[String]): Unit = { /* val conf = new SparkConf().setAppName("Hel...
spark中读取json_Spark解析Json数据(非Sql方式)
weixin_39735012的博客
12-19 886
import play.api.libs.json._val input = sc.parallelize(List( """{"name":"过往记忆","website":"www.iteblog.com"}""","""{"other":"过往记忆"}"""))val parsed = input.map(Json.parse)parsed.collectoutput:{"name":"过往...
spark中读取json_在缺少json数据源的spark中读取json
weixin_35698035的博客
12-29 174
我试图使用下面的代码将示例json文件读入SqlContext，但它失败并出现数据源错误 .val sqlContext = new org.apache.spark.sql.SQLContext(sc)val path = "C:\\samplepath\\sample.json"val jsondata = sqlContext.read.json(path)java.lang.ClassNo...
大数据实时计算Spark学习笔记（9）—— Spar SQL(1) 读取 json 文件
一角残叶的博客
12-29 1166
1 Spark SQL 编程方式：（1）SQL;(2) DataFrame API scala> case class Customer(id:Int,name:String,age:Int) defined class Customer scala> val arr = Array("1,Mike,20","2,Mary,19","3,Jerry,23&quo
【Spark】使用DataFrame读取复杂JSON中的嵌套数组
Big data enthusiast
06-01 3670
所周知，在早期Spark版本中就已经支持读取Json格式的数据文件，并能够直接转换为数据库表，以方便我们进行处理数据，在本篇文章中我们将介绍如何通过Spark API很简...
SparkSQL读取json文件简单实现（一）
南望山下砖瓦工
06-07 9122
运行前提： 搭建Hadoop + Spark集群，安装jdk1.8及以上版本，Scala2.11，maven3 1、新建maven项目 这里需要注意的是我们为了方便，直接勾选Create from archetype，然后从下面列表中选择scala archetype simple 2、填写GAV 这里按照个人设置，说白了GroupID就是公司的域名倒过来写 artifactId...
c++读取json文件_Spark本地文件读写
05-26
这两个问题看起来不是很相关，我先回答第一个问题：如何在C++中读取JSON文件。 在C++中，可以使用第三方库来读取JSON文件，例如nlohmann/json或者RapidJSON。以下是使用nlohmann/json库读取JSON文件的示例代码： `...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值