Spark解析json格式文件(spark-shell和Idea scala版本)

最新推荐文章于 2022-06-17 09:14:05 发布
置顶 最新推荐文章于 2022-06-17 09:14:05 发布
阅读量528 收藏 4
点赞数
分类专栏： Spark 文章标签： json spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43434273/article/details/109957929
版权

Spark解析json文件


现在有一个文件名为 op.log 的json文件内容是： 

1593136280858|{"cm":{"ln":"-55.0","sv":"V2.9.6","os":"8.0.4","g":"C6816QZ0@gmail.com","mid":"489","nw":"3G","l":"es","vc":"4","hw":"640*960","ar":"MX","uid":"489","t":"1593123253541","la":"5.2","md":"sumsung-18","vn":"1.3.4","ba":"Sumsung","sr":"I"},"ap":"app","et":[{"ett":"1593050051366","en":"loading","kv":{"extend2":"","loading_time":"14","action":"3","extend1":"","type":"2","type1":"201","loading_way":"1"}},{"ett":"1593108791764","en":"ad","kv":{"activityId":"1","displayMills":"78522","entry":"1","action":"1","contentType":"0"}},{"ett":"1593111271266","en":"notification","kv":{"ap_time":"1593097087883","action":"1","type":"1","content":""}},{"ett":"1593066033562","en":"active_background","kv":{"active_source":"3"}},{"ett":"1593135644347","en":"comment","kv":{"p_comment_id":1,"addtime":"1593097573725","praise_count":973,"other_id":5,"comment_id":9,"reply_count":40,"userid":7,"content":"辑赤蹲慰鸽抿肘捎"}}]}
1593136280858|{"cm":{"ln":"-114.9","sv":"V2.7.8","os":"8.0.4","g":"NW0S962J@gmail.com","mid":"490","nw":"3G","l":"pt","vc":"8","hw":"640*1136","ar":"MX","uid":"490","t":"1593121224789","la":"-44.4","md":"Huawei-8","vn":"1.0.1","ba":"Huawei","sr":"O"},"ap":"app","et":[{"ett":"1593063223807","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"1","type1":"102","loading_way":"1"}},{"ett":"1593095105466","en":"ad","kv":{"activityId":"1","displayMills":"1966","entry":"3","action":"2","contentType":"0"}},{"ett":"1593051718208","en":"notification","kv":{"ap_time":"1593095336265","action":"2","type":"3","content":""}},{"ett":"1593100021275","en":"comment","kv":{"p_comment_id":4,"addtime":"1593098946009","praise_count":220,"other_id":4,"comment_id":9,"reply_count":151,"userid":4,"content":"抄应螟皮釉倔掉汉蛋蕾街羡晶"}},{"ett":"1593105344120","en":"praise","kv":{"target_id":9,"id":7,"type":1,"add_time":"1593098545976","userid":8}}]}

1.找json在线解析看一下结构，是这样的


	 1593136280858|{
	"cm": {
		"ln": "-55.0",
		"sv": "V2.9.6",
		"os": "8.0.4",
		"g": "C6816QZ0@gmail.com",
		"mid": "489",
		"nw": "3G",
		"l": "es",
		"vc": "4",
		"hw": "640*960",
		"ar": "MX",
		"uid": "489",
		"t": "1593123253541",
		"la": "5.2",
		"md": "sumsung-18",
		"vn": "1.3.4",
		"ba": "Sumsung",
		"sr": "I"
	},
	"ap": "app",
	"et": [{
		"ett": "1593050051366",
		"en": "loading",
		"kv": {
			"extend2": "",
			"loading_time": "14",
			"action": "3",
			"extend1": "",
			"type": "2",
			"type1": "201",
			"loading_way": "1"
		}
	}, {
		"ett": "1593108791764",
		"en": "ad",
		"kv": {
			"activityId": "1",
			"displayMills": "78522",
			"entry": "1",
			"action": "1",
			"contentType": "0"
		}
	}, {
		"ett": "1593111271266",
		"en": "notification",
		"kv": {
			"ap_time": "1593097087883",
			"action": "1",
			"type": "1",
			"content": ""
		}
	}, {
		"ett": "1593066033562",
		"en": "active_background",
		"kv": {
			"active_source": "3"
		}
	}, {
		"ett": "1593135644347",
		"en": "comment",
		"kv": {
			"p_comment_id": 1,
			"addtime": "1593097573725",
			"praise_count": 973,
			"other_id": 5,
			"comment_id": 9,
			"reply_count": 40,
			"userid": 7,
			"content": "辑赤蹲慰鸽抿肘捎"
		}
	}]
}

2.spark 读取json并转成DataFrame进行数据表格化 进而分析数据

spark-shell版本

//加载存储在hdfs上的json文件
scala> val fileRDD=sc.textFile("hdfs://192.168.116.60:9000/kb09file/op.log")

//按照"|" 切割 
scala> val jsonStrRDD=fileRDD.map(x=>x.split('|')).map(x=>(x(0),x(1)))

//将手机号作为id保留放在最后
scala> val jsonRDD=jsonStrRDD.map(x=>{var jsonStr=x._2;jsonStr 
= jsonStr.substring(0,jsonStr.length-1);jsonStr+",\"id\":\""+x._1+"\"}"})   //加双引号才是json字符串 很重要

//变成DataFrame 进行后续操作
scala> val jsonDF=jsonRDD.toDF

scala> jsonDF.printSchema
root
 |-- value: string (nullable = true)

//导入包 便于使用get_json_object函数
scala> import spark.implicits._
scala> import org.apache.spark.sql.functions._
scala> import org.apache.spark.sql._
scala> import org.apache.spark.sql.types._

scala> jsonDF.select(get_json_object($"value","$.cm").alias("cm")).printSchema
root
 |-- cm: string (nullable = true)

scala> jsonDF.select(get_json_object($"value","$.cm").alias("cm")).show
+--------------------+
|                  cm|
+--------------------+
|{"ln":"-55.0","sv...|
|{"ln":"-114.9","s...|
+--------------------+

scala> jsonDF.select(get_json_object($"value","$.cm").alias("cm"),get_json_object($"value","$.ap").alias("ap"),
get_json_object($"value","$.et").alias("et"),get_json_object($"value","$.id").alias("id")).show()
+--------------------+---+--------------------+-------------+
|                  cm| ap|                  et|           id|
+--------------------+---+--------------------+-------------+
|{"ln":"-55.0","sv...|app|[{"ett":"15930500...|1593136280858|
|{"ln":"-114.9","s...|app|[{"ett":"15930632...|1593136280858|
+--------------------+---+--------------------+-------------+

//将json字符串结构化
//表头   cm   ap   et  id  
//列     a1   b1   c1  d1
scala> val jsonDF2 = jsonDF.select(get_json_object($"value","$.cm").alias("cm"),get_json_object($"value","$.ap").alias("ap")
,get_json_object($"value","$.et").alias("et"),get_json_object($"value","$.id").alias("id"))

scala> jsonDF2.printSchema
root
 |-- cm: string (nullable = true)
 |-- ap: string (nullable = true)
 |-- et: string (nullable = true)
 |-- id: string (nullable = true)

scala> jsonDF2.select($"id",$"ap",$"cm",$"et").show
+-------------+---+--------------------+--------------------+
|           id| ap|                  cm|                  et|
+-------------+---+--------------------+--------------------+
|1593136280858|app|{"ln":"-55.0","sv...|[{"ett":"15930500...|
|1593136280858|app|{"ln":"-114.9","s...|[{"ett":"15930632...|
+-------------+---+--------------------+--------------------+


scala> val jsonDF3 = jsonDF2.select($"id",$"ap",get_json_object($"cm","$.ln") alias("ln"),
get_json_object($"cm","$.sv") alias("sv"),get_json_object($"cm","$.os") alias("os"),
get_json_object($"cm","$.g") alias("g"),get_json_object($"cm","$.mid") alias("mid"),
get_json_object($"cm","$.nw") alias("nw"),get_json_object($"cm","$.l") alias("l"),
get_json_object($"cm","$.vc") alias("vc"),get_json_object($"cm","$.hw") alias("hw"),
get_json_object($"cm","$.ar") alias("ar"),get_json_object($"cm","$.uid") alias("uid"),
get_json_object($"cm","$.t") alias("t"),get_json_object($"cm","$.la") alias("la"),
get_json_object($"cm","$.md") alias("md"),get_json_object($"cm","$.vn") alias("vn"),
get_json_object($"cm","$.ba") alias("ba"),get_json_object($"cm","$.sr") alias("sr"),$"et")

scala> jsonDF3.printSchema
root
 |-- id: string (nullable = true)
 |-- ap: string (nullable = true)
 |-- ln: string (nullable = true)
 |-- sv: string (nullable = true)
 |-- os: string (nullable = true)
 |-- g: string (nullable = true)
 |-- mid: string (nullable = true)
 |-- nw: string (nullable = true)
 |-- l: string (nullable = true)
 |-- vc: string (nullable = true)
 |-- hw: string (nullable = true)
 |-- ar: string (nullable = true)
 |-- uid: string (nullable = true)
 |-- t: string (nullable = true)
 |-- la: string (nullable = true)
 |-- md: string (nullable = true)
 |-- vn: string (nullable = true)
 |-- ba: string (nullable = true)
 |-- sr: string (nullable = true)
 |-- et: string (nullable = true)

scala> jsonDF3.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|                  et|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|[{"ett":"15930500...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|[{"ett":"15930632...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+--------------------+

//from_json把字符串 
[
{"ett":"a1","en":"a2","kv":"a3"},
{"ett":"b1","en":"b2","kv":"b3"},
{"ett":"c1","en":"c2","kv":"c3"}
]   结构化
    ett    en    kv 
    a1     a2    a3
	b1	   b2    b3
	c1     c2    c3
	
val jsonDF4 =  jsonDF3.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",
from_json($"et",ArrayType(StructType(StructField("ett",StringType)::StructField("en",StringType)::StructField("kv",StringType)::Nil))).alias("event"))

jsonDF4大脑想到的结构是这样：
       id           ap      mid     nw     ***     event
1593136280858      app      12345   dsdwe  ****   [[ett en kv],	[b1 b2 b3],[c1 c2 c3] ] 


       id           ap      mid     nw     ***     ett  en  kv
1593136280858      app      12345   dsdwe  ****    a1   a2  a3	
1593136280858      app      12345   dsdwe  ****    b1   b2   b3  
1593136280858      app      12345   dsdwe  ****    c1   c2   c3  
------------思考结束-----------------

开干。。。。。
第一步
       id           ap      mid     nw     ***     event
1593136280858      app      12345   dsdwe  ****   [[ett en kv],	[b1 b2 b3],[c1 c2 c3] ]
搞成
		id           ap      mid     nw     ***     event
 1593136280858      app      12345   dsdwe  ****   [ett en kv]
 1593136280858      app      12345   dsdwe  ****   [b1 b2 b3]
 1593136280858      app      12345   dsdwe  ****   [c1 c2 c3]
scala> val jsonDF5 = jsonDF4.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",explode($"event").alias("event"))
第二步
		id           ap      mid     nw     ***     ett    en   kv
 1593136280858      app      12345   dsdwe  ****   a1    a2     a3
 1593136280858      app      12345   dsdwe  ****   b1    b2      b3
 1593136280858      app      12345   dsdwe  ****   c1    c2     c3

		
scala> val jsonDF5 = jsonDF4.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",
explode($"event").alias("event"))

scala> val jsonDF6 = jsonDF5.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",
($"event.kv").alias("kv"),($"event.ett").alias("ett"),($"event.en").alias("en"))

scala> jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en"
,$"kv").filter($"en"==="loading").show(false)

scala> val loadingDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr"
,$"ett",$"en",$"kv").filter($"en"==="loading")

scala> loadingDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.extend2") alias("extend2"),get_json_object($"kv","$.loading_time") alias("loading_time"),get_json_object($"kv","$.action") alias("action"),get_json_object($"kv","$.extend1") alias("extend1"),get_json_object($"kv","$.type") alias("type"),get_json_object($"kv","$.type1") alias("type1"),get_json_object($"kv","$.loading_way") alias("loading_way")).printSchema
root
 |-- id: string (nullable = true)
 |-- ap: string (nullable = true)
 |-- ln: string (nullable = true)
 |-- sv: string (nullable = true)
 |-- os: string (nullable = true)
 |-- g: string (nullable = true)
 |-- mid: string (nullable = true)
 |-- nw: string (nullable = true)
 |-- l: string (nullable = true)
 |-- vc: string (nullable = true)
 |-- hw: string (nullable = true)
 |-- ar: string (nullable = true)
 |-- uid: string (nullable = true)
 |-- t: string (nullable = true)
 |-- la: string (nullable = true)
 |-- md: string (nullable = true)
 |-- vn: string (nullable = true)
 |-- ba: string (nullable = true)
 |-- sr: string (nullable = true)
 |-- ett: string (nullable = true)
 |-- en: string (nullable = true)
 |-- extend2: string (nullable = true)
 |-- loading_time: string (nullable = true)
 |-- action: string (nullable = true)
 |-- extend1: string (nullable = true)
 |-- type: string (nullable = true)
 |-- type1: string (nullable = true)
 |-- loading_way: string (nullable = true)


scala> val loadingDF1 = loadingDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
$"sr",$"ett",$"en",get_json_object($"kv","$.extend2") alias("extend2"),get_json_object($"kv","$.loading_time") alias("loading_time"),get_json_object($"kv","$.action") alias("action"),get_json_object($"kv","$.extend1") alias("extend1"),get_json_object($"kv","$.type") alias("type"),get_json_object($"kv","$.type1") alias("type1"),get_json_object($"kv","$.loading_way") alias("loading_way"))
scala> loadingDF1.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----+-----------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|          ett|     en|extend2|loading_time|action|extend1|type|type1|loading_way|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----+-----------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593050051366|loading|       |          14|     3|       |   2|  201|          1|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593063223807|loading|       |           0|     3|       |   1|  102|          1|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----+-----------+



scala> val adDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",$"kv").filter($"en"==="ad")
scala> val adDF1 = adDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.activityId") alias("activityId"),get_json_object($"kv","$.displayMills") alias("displayMills"),get_json_object($"kv","$.entry") alias("entry"),get_json_object($"kv","$.action") alias("action"),get_json_object($"kv","$.contentType") alias("contentType"))
scala> adDF1.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|          ett| en|activityId|displayMills|entry|action|contentType|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593108791764| ad|         1|       78522|    1|     1|          0|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593095105466| ad|         1|        1966|    3|     2|          0|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+


scala> val notificationDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",$"kv").filter($"en"==="notification")
scala> val notificationDF1 = notificationDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.ap_time") alias("ap_time"),get_json_object($"kv","$.action") alias("action"),get_json_object($"kv","$.type") alias("type"),get_json_object($"kv","$.content") alias("content"))
scala> notificationDF1.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|          ett|          en|      ap_time|action|type|content|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593111271266|notification|1593097087883|     1|   1|       |
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593051718208|notification|1593095336265|     2|   3|       |
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+


scala> val active_backgroundDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",$"kv").filter($"en"==="active_background")
scala> val active_backgroundDF1 = active_backgroundDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.active_source") alias("active_source"))
scala> active_backgroundDF1.show
+-------------+---+-----+------+-----+------------------+---+---+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+
|           id| ap|   ln|    sv|   os|                 g|mid| nw|  l| vc|     hw| ar|uid|            t| la|        md|   vn|     ba| sr|          ett|               en|active_source|
+-------------+---+-----+------+-----+------------------+---+---+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+
|1593136280858|app|-55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4|640*960| MX|489|1593123253541|5.2|sumsung-18|1.3.4|Sumsung|  I|1593066033562|active_background|            3|
+-------------+---+-----+------+-----+------------------+---+---+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+


scala> val commentDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",$"kv").filter($"en"==="comment")
scala> val commentDF1 = commentDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",get_json_object($"kv","$.p_comment_id") alias("p_comment_id"),get_json_object($"kv","$.addtime") alias("addtime"),get_json_object($"kv","$.praise_count") alias("praise_count"),get_json_object($"kv","$.other_id") alias("other_id"),get_json_object($"kv","$.comment_id") alias("comment_id"),get_json_object($"kv","$.reply_count") alias("reply_count"),get_json_object($"kv","$.userid") alias("userid"),get_json_object($"kv","$.content") alias("content"))
scala> commentDF1.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+--------------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|        md|   vn|     ba| sr|          ett|     en|p_comment_id|      addtime|praise_count|other_id|comment_id|reply_count|userid|                   content|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+--------------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|sumsung-18|1.3.4|Sumsung|  I|1593135644347|comment|           1|1593097573725|         973|       5|         9|         40|     7|          辑赤蹲慰鸽抿肘捎|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|  Huawei-8|1.0.1| Huawei|  O|1593100021275|comment|           4|1593098946009|         220|       4|         9|        151|     4|抄应螟皮釉倔掉汉蛋蕾街羡晶|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+--------------------------+

scala> val praiseDF = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",$"sr",$"ett",$"en",$"kv").filter($"en"==="praise")
scala> val praiseDF1 = praiseDF.select($"id",$"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba"
,$"sr",$"ett",$"en",get_json_object($"kv","$.target_id") alias("target_id"),get_json_object($"kv","$.id") alias("id1"),
get_json_object($"kv","$.type") alias("type"),get_json_object($"kv","$.add_time") alias("add_time"),get_json_object($"kv","$.userid") alias("userid"))

scala>  praiseDF1.show
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|      md|   vn|    ba| sr|          ett|    en|target_id|id1|type|     add_time|userid|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|Huawei-8|1.0.1|Huawei|  O|1593105344120|praise|        9|  7|   1|1593098545976|     8|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+






scala> loadingDF1.join(adDF1,Seq("id","id"),"inner").join(notificationDF1,Seq("id","id"),"inner").join(active_backgroundDF1,Seq("id","id"),"inner").join(commentDF1,Seq("id","id"),"inner").show

Idea scala版本

package Json

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types.StructType

/**
  * @Author Bright
  * @Date 2020/11/25
  * @Description
  */
object jsonToDF {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").appName("jsonToDF").getOrCreate()
    val sc = spark.sparkContext

    import spark.implicits._

    //加载文件RDD
    val fileRDD = sc.textFile("in/op.log")
    val opjsonStr = fileRDD.map(x=>x.split('|')).map(x=>(x(0),x(1)))
      .map(x=>{
        var jsonStr = x._2;
        jsonStr = jsonStr.substring(0, jsonStr.length - 1);
        jsonStr+",\"id\":\""+x._1+"\"}"})

    //RDD转成DF
    val opjsonDF = opjsonStr.toDF()
    //opjsonDF.printSchema()
    //opjsonDF.show(false)

    //json最内层结构
    val et = new StructType()
      .add($"ett".string)
      .add($"en".string)
      .add($"kv".string)

    //json中层结构
    val cm = new StructType()
      .add($"ln".string).add($"sv".string).add($"os".string).add($"g".string).add($"mid".string).add($"nw".string)
      .add($"hw".string).add($"ar".string).add($"uid".string).add($"t".string).add($"la".string).add($"md".string)
      .add($"vn".string).add($"ba".string).add($"sr".string)

    //json 外层套中层 完成 最外层结构
    val schema = new StructType()
      .add($"cm".struct(cm))
      .add($"ap".string)
      .add($"et".array(et))
      .add($"id".string)

    //使用from_json方法 由json数据 与结构匹配
    val frame = opjsonDF.select(from_json($"value",schema).alias("values"))
//    frame.printSchema()
//    frame.show(false)

    //将列名数据查询出来 et层使用explode拆开并取别名为event
    val frame2 = frame.select(
      $"values.ap", $"values.cm.ln", $"values.cm.sv", $"values.cm.os", $"values.cm.g", $"values.cm.mid", $"values.cm.nw",
      $"values.cm.hw", $"values.cm.ar", $"values.cm.uid", $"values.cm.t", $"values.cm.la", $"values.cm.md", $"values.cm.vn",
      $"values.cm.ba", $"values.cm.sr",
      explode($"values.et").alias("event")
    )

//    frame2.printSchema()
//    frame2.show(false)

    //将ett en kv 单独查出来
    val frame3 = frame2.select(
      $"ap", $"ln", $"sv", $"os", $"g", $"mid", $"nw", $"hw", $"ar", $"uid", $"t", $"la", $"md", $"vn", $"ba", $"sr",
      $"event.ett", $"event.en", $"event.kv"
    )

//    frame3.printSchema()
//    frame3.show(false)

    //过滤出来en层数据的6种情况  分别查询出来
    val loadingDF = frame3.where($"en" === "loading")
      .select(
        $"ap", $"ln", $"sv", $"os", $"g", $"mid", $"nw", $"hw", $"ar", $"uid", $"t", $"la", $"md", $"vn", $"ba",
        $"sr", $"ett", $"en",
        get_json_object($"kv", "$.extend2").alias("extend2"),
        get_json_object($"kv", "$.loading_time").alias("loading_time"),
        get_json_object($"kv", "$.action").alias("action"),
        get_json_object($"kv", "$.extend1").alias("extend1"),
        get_json_object($"kv", "$.type").alias("type"),
        get_json_object($"kv", "$.loading_way").alias("loading_way")
      )
    loadingDF.show(false)

    //写入保存到hdfs上面  文件端口：9000  需要赋权限且文件夹不存在才行  会自动生成新文件夹
    loadingDF.write.save("hdfs://192.168.116.60:9000/kb09file2/output/loadingDF")

    val adDF: DataFrame = frame3.where($"en" === "ad")
      .select($"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
        $"sr",$"ett",$"en",
        get_json_object($"kv", "$.activityId").alias("activityId"),
        get_json_object($"kv", "$.displayMills").alias("displayMills"),
        get_json_object($"kv", "$.entry").alias("entry"),
        get_json_object($"kv", "$.action").alias("action"),
        get_json_object($"kv", "$.contentType").alias("contentType")
      )
    adDF.show(false)

    val notificationDF: DataFrame = frame3.where($"en" === "notification")
      .select($"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
        $"sr",$"ett",$"en",
        get_json_object($"kv", "$.ap_time").alias("ap_time"),
        get_json_object($"kv", "$.action").alias("action"),
        get_json_object($"kv", "$.type").alias("type"),
        get_json_object($"kv", "$.content").alias("content")
      )
    notificationDF.show(false)

    val activeBackgroundDF: DataFrame = frame3.where($"en" === "active_background")
      .select($"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
        $"sr",$"ett",$"en",
        get_json_object($"kv", "$.active_source").alias("active_source")
      )
    activeBackgroundDF.show(false)

    val commentDF: DataFrame = frame3.where($"en" === "comment")
      .select($"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
        $"sr",$"ett",$"en",
        get_json_object($"kv", "$.p_comment_id").alias("p_comment_id"),
        get_json_object($"kv", "$.addtime").alias("addtime"),
        get_json_object($"kv", "$.praise_count").alias("praise_count"),
        get_json_object($"kv", "$.other_id").alias("other_id"),
        get_json_object($"kv", "$.comment_id").alias("comment_id"),
        get_json_object($"kv", "$.reply_count").alias("reply_count"),
        get_json_object($"kv", "$.userid").alias("userid"),
        get_json_object($"kv", "$.content").alias("content")
      )
    commentDF.show(false)

    val praiseDF: DataFrame = frame3.where($"en" === "praise")
      .select($"ap",$"ln",$"sv",$"os",$"g",$"mid",$"nw",$"hw",$"ar",$"uid",$"t",$"la",$"md",$"vn",$"ba",
        $"sr",$"ett",$"en",
        get_json_object($"kv", "$.target_id").alias("target_id"),
        get_json_object($"kv", "$.id").alias("id1"),
        get_json_object($"kv", "$.type").alias("type"),
        get_json_object($"kv", "$.add_time").alias("add_time"),
        get_json_object($"kv", "$.userid").alias("userid")
      )
    praiseDF.show(false)
  }
}


//输出结果：
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----------+
|ap |ln    |sv    |os   |g                 |mid|nw |hw      |ar |uid|t            |la   |md        |vn   |ba     |sr |ett          |en     |extend2|loading_time|action|extend1|type|loading_way|
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----------+
|app|-55.0 |V2.9.6|8.0.4|C6816QZ0@gmail.com|489|3G |640*960 |MX |489|1593123253541|5.2  |sumsung-18|1.3.4|Sumsung|I  |1593050051366|loading|       |14          |3     |       |2   |1          |
|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490|3G |640*1136|MX |490|1593121224789|-44.4|Huawei-8  |1.0.1|Huawei |O  |1593063223807|loading|       |0           |3     |       |1   |1          |
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+-------+------------+------+-------+----+-----------+

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+
|ap |ln    |sv    |os   |g                 |mid|nw |hw      |ar |uid|t            |la   |md        |vn   |ba     |sr |ett          |en |activityId|displayMills|entry|action|contentType|
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+
|app|-55.0 |V2.9.6|8.0.4|C6816QZ0@gmail.com|489|3G |640*960 |MX |489|1593123253541|5.2  |sumsung-18|1.3.4|Sumsung|I  |1593108791764|ad |1         |78522       |1    |1     |0          |
|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490|3G |640*1136|MX |490|1593121224789|-44.4|Huawei-8  |1.0.1|Huawei |O  |1593095105466|ad |1         |1966        |3    |2     |0          |
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+---+----------+------------+-----+------+-----------+

+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+
|ap |ln    |sv    |os   |g                 |mid|nw |hw      |ar |uid|t            |la   |md        |vn   |ba     |sr |ett          |en          |ap_time      |action|type|content|
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+
|app|-55.0 |V2.9.6|8.0.4|C6816QZ0@gmail.com|489|3G |640*960 |MX |489|1593123253541|5.2  |sumsung-18|1.3.4|Sumsung|I  |1593111271266|notification|1593097087883|1     |1   |       |
|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490|3G |640*1136|MX |490|1593121224789|-44.4|Huawei-8  |1.0.1|Huawei |O  |1593051718208|notification|1593095336265|2     |3   |       |
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+------------+-------------+------+----+-------+

+---+-----+------+-----+------------------+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+
|ap |ln   |sv    |os   |g                 |mid|nw |hw     |ar |uid|t            |la |md        |vn   |ba     |sr |ett          |en               |active_source|
+---+-----+------+-----+------------------+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+
|app|-55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489|3G |640*960|MX |489|1593123253541|5.2|sumsung-18|1.3.4|Sumsung|I  |1593066033562|active_background|3            |
+---+-----+------+-----+------------------+---+---+-------+---+---+-------------+---+----------+-----+-------+---+-------------+-----------------+-------------+

+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+-------------+
|ap |ln    |sv    |os   |g                 |mid|nw |hw      |ar |uid|t            |la   |md        |vn   |ba     |sr |ett          |en     |p_comment_id|addtime      |praise_count|other_id|comment_id|reply_count|userid|content      |
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+-------------+
|app|-55.0 |V2.9.6|8.0.4|C6816QZ0@gmail.com|489|3G |640*960 |MX |489|1593123253541|5.2  |sumsung-18|1.3.4|Sumsung|I  |1593135644347|comment|1           |1593097573725|973         |5       |9         |40         |7     |辑赤蹲慰鸽抿肘捎     |
|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490|3G |640*1136|MX |490|1593121224789|-44.4|Huawei-8  |1.0.1|Huawei |O  |1593100021275|comment|4           |1593098946009|220         |4       |9         |151        |4     |抄应螟皮釉倔掉汉蛋蕾街羡晶|
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+----------+-----+-------+---+-------------+-------+------------+-------------+------------+--------+----------+-----------+------+-------------+

+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+
|ap |ln    |sv    |os   |g                 |mid|nw |hw      |ar |uid|t            |la   |md      |vn   |ba    |sr |ett          |en    |target_id|id1|type|add_time     |userid|
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+
|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490|3G |640*1136|MX |490|1593121224789|-44.4|Huawei-8|1.0.1|Huawei|O  |1593105344120|praise|9        |7  |1   |1593098545976|8     |
+---+------+------+-----+------------------+---+---+--------+---+---+-------------+-----+--------+-----+------+---+-------------+------+---------+---+----+-------------+------+
Bright Huang
关注
专栏目录
流式数据采集和计算（六）：IDEA+MAVEN+Scala配置进行spark开发
叶子叶来
12-12 8478
一、环境配置 IDEA作为常用的开发工具使用maven进行依赖包的统一管理，配置Scala的开发环境，进行Spark Streaming的API开发； 1、下载并破解IDEA，并加入汉化的包到lib，重启生效； 2、在IDEA中导入离线的Scala插件：首先下载IDEA的Scala插件，无须解压，然后将其添加到IDEA中，具体为new---setting--plug...
[Spark版本更新]--2.3.0发行说明
欢迎来到我的博客，一起探索代码里的世界！
03-03 1万+
自从2017年12月1日发布spark-2.2.1以来，已有3个月时间。2018年2月28日，spark官方发布了一个大版本Spark-2.3.0，解决了1399个大大小小的问题。一、DataBricks做了相关说明今天，我们很高兴地宣布Databricks上的Apache Spark 2.3.0作为其Databricks Runtime 4.0的一部分。我们要感谢Apache Spark社区为S...
Scala解析Json字符串的实例详解
10-19
主要介绍了 Scala解析Json字符串的实例详解的相关资料,希望通过本文能帮助到大家，让大家学习理解这部分内容，需要的朋友可以参考下
json解析 spark_Spark SQL解析Json
weixin_39802020的博客
12-23 149
学习链接：https://www.shiyanlou.com/courses/809首先切换用户：su hadoophadoop进入opt目录cd /opt第一次使用时，最好先把core-site.xml中的tmp文件位置改一下，然后格式化hdfs，hadoop-2.6.1/bin/hdfs namenode -format启动hadoop集群(可通过jps判断是否启动成功)，创建person.j...
SparkShell读取json文件
张俊杰 的博客
11-20 1305
SparkShell读取Linux本地json文件 root/soft/person.json就是本机路径 var df = spark.read.format(“json”).load(“file:///root/soft/person.json”) scala> var df = spark.read.format("json").load("file:///root/soft/person.json") df: org.apache.spark.sql.DataFrame = [_cor
小白学习Spark系列五：scala解析多级json格式字符串
weixin_30301449的博客
11-19 449
一、背景 　　处理json格式的字符串，key值一定为String类型，但value不确定是什么类型，也可能嵌套json字符串，以下是使用JSON.parseFull 来解析多层json。 二、实例代码 import collection.mutable.Map import scala.util.parsing.json._ def regJson(json:Opti...
scala 读取大的文本文件
nfzhlk的专栏
11-11 745
在做测试开发时，发现有时候接口返回的数据太长，直接一个变量赋值，超出了字符串的最大范围。于是，需要先将接口的返回值存储成一个文本文件，如txt文件或者.json文件，然后再通过代码取读取。只需要短短的两行，就可以实现了。 如：数据量写的时候，我可以： val response = """{ | "ResponseStatus": { | "Timestamp"
spark解析json格式文件(spark-shell和idea scala版本)
最新发布
03-16
以下是使用Spark Shell和IDEA Scala版本解析JSON格式文件的步骤： 1. 在Spark Shell中，首先需要创建一个SparkSession对象： ``` val spark = SparkSession.builder().appName("JSONParser").master("local[*]")....
spark中读取json_【spark】文件读写和JSON数据解析
weixin_33773670的博客
02-11 3280
1.读文件通过sc.textFile(“file://")方法来读取文件到rdd中。val lines = sc.textFile("file://")//文件地址或者HDFS文件路径本地地址"file:///home/hadoop/spark-1.6.0-bin-hadoop2.6/examples/src/main/resources/people.json"HDFS文件地址"hdfs://...
十一、Spark SQL数据源 - JSON数据集
zlwm000的博客
06-17 581
Spark SQL可以自动推断JSON文件的Schema，并将其加载为DataFrame。在加载和写入JSON文件时，除了可以使用load()方法和save()方法外，还可以直接使用Spark SQL内置的json()方法。该方法不仅可以读写JSON文件，还可以将Dataset[String]类型的数据集转为DataFrame。 需要注意的是，要想成功地将一个JSON文件加载为DataFrame，JSON文件的每一行必须包含一个独立有效的JSON对象，而不能将一个JSON对象分散在多行。创建user.jso
最新版Spark2.2读取多种文件格式数据
NON blog
12-19 4439
Spark2.0+的文件读取Spark可以读取多种格式文件，csv，json，parque。因此对应就有很多函数与之对应。在Spark2.0以后一般使用SparkSession来操作DataFrame、Dataset来完成数据分析。这些读取不同格式文件的函数就是SparkSession的成员DataFrameReader的方法。该类就是将文件系统（HDFS，LocalFileSystem（一定要在每
spark2 sql读取json文件的格式要求
about云
12-07 3893
问题导读1.spark2 sql如何读取json文件？2.spark2读取json格式文件有什么要求？3.spark2是如何处理对于带有表名信息的json文件的?spark有多个数据源，json是其中一种。那么对于json格式的数据，spark在操作的过程中，可能会遇到哪些问题？这里首先我们需要对json格式的数据有一定的了解。json数据有两种格式：1.对象表示2.数组表示二者也有嵌套形式。比如
spark操作json数据
盛源的博客
08-09 9980
一、转换到JSON格式2. spark-sql转json串1. 把某些列封装到json串2. 把所有列封装到json串二、 特殊字符1. key中含有特殊字符三、数组操作四、 spark读取json文件1. dataframe指定schema读取。
spark scala 实时解析Json 数组
yyz0126的博客
01-29 1766
spark scala 实时解析Json 数组 实时数据源Kafka 大数据接入实时数据早已大势已去，很多大厂更是实时数据常态化，但是最近发现网上并没有类似的数据处理过程推荐，这里是我结合工作场景写的一篇文章，有很多不足，烦请指正,谢谢 JSON数组是各个业务系统最喜欢生成的文件，平日里有许多log文件95%以上都是json格式的数据，最近SDK系统产生的数据放入kafka中，提供大数据中心来消费 先看数据 这是一个业务线每天产生的数据，数据在kafka中有同样的数据，这里使用flume备份到h
spark 读取json文件并分析
热门推荐
archer的技术故事
01-07 2万+
spark 读取json文件并分析 本文主要介绍如何通过读取json文件到spark中然后进行分析。 本文阅读前提： 1、能够正常的跑起来spark 2、有一定的scala语言知识 3、了解json4s的基本功能和用法（http://blog.csdn.net/wild46cat/article/details/54171973和http://blog.csdn.net/wild46ca
spark处理json数据Demo
我的博客
02-26 3060
/** * @Auther: sss * @Date: 2019/2/26 14:24 * @Description: * ps:数据是json 如下（数据是a#b#格式） 需求：lat,lng;timestamp#lat,lng;timestamp#lat,lng;timestamp （最后要是a#b格式） */ object xiaoM { ...
spark-shell 基础操作（持续更新）
jiede1的博客
09-11 1642
概述 Spark SQL 是 Spark 处理结构化数据的一个模块。与基础的 Spark RDD API 不同，Spark SQL 提供了查询结构化数据及计算结果等信息的接口。在内部，Spark SQL 使用这个额外的信息去执行额外的优化。有几种方式可以跟 Spark SQL 进行交互，包括 SQL 和 Dataset API。当使用相同执行引擎进行计算时，无论使用哪种 API / 语言都可以快速
[Spark] spark-shell 命令使用
Cindy的博客
07-12 2万+
环境：操作系统：CentOS7.3Java: jdk1.8.0_45Hadoop：hadoop-2.6.0-cdh5.14.0.tar.gz 1. spark-shell 使用帮助[hadoop@hadoop01 ~]$ cd app/spark-2.2.0-bin-2.6.0-cdh5.7.0/bin [hadoop@hadoop01 bin]$ ./spark-shell --help Usa...
Spark对HDFS上json数据的简单操作
strongyoung的专栏
07-25 1万+
Spark对HDFS上json数据的操作非常方便，本文以两种方式进行简单介绍，分别为Spark Shell 和 编写Scala应用程序。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值