JSON数据清理(详解)

二、JSON数据清洗

1、JSON数据

仅以两条数据为例

1593136280858|{"cm":{"ln":"-55.0","sv":"V2.9.6","os":"8.0.4","g":"C6816QZ0@gmail.com","mid":"489","nw":"3G","l":"es","vc":"4","hw":"640*960","ar":"MX","uid":"489","t":"1593123253541","la":"5.2","md":"sumsung-18","vn":"1.3.4","ba":"Sumsung","sr":"I"},"ap":"app","et":[{"ett":"1593050051366","en":"loading","kv":{"extend2":"","loading_time":"14","action":"3","extend1":"","type":"2","type1":"201","loading_way":"1"}},{"ett":"1593108791764","en":"ad","kv":{"activityId":"1","displayMills":"78522","entry":"1","action":"1","contentType":"0"}},{"ett":"1593111271266","en":"notification","kv":{"ap_time":"1593097087883","action":"1","type":"1","content":""}},{"ett":"1593066033562","en":"active_background","kv":{"active_source":"3"}},{"ett":"1593135644347","en":"comment","kv":{"p_comment_id":1,"addtime":"1593097573725","praise_count":973,"other_id":5,"comment_id":9,"reply_count":40,"userid":7,"content":"辑赤蹲慰鸽抿肘捎"}}]}
1593136280859|{"cm":{"ln":"-114.9","sv":"V2.7.8","os":"8.0.4","g":"NW0S962J@gmail.com","mid":"490","nw":"3G","l":"pt","vc":"8","hw":"640*1136","ar":"MX","uid":"490","t":"1593121224789","la":"-44.4","md":"Huawei-8","vn":"1.0.1","ba":"Huawei","sr":"O"},"ap":"app","et":[{"ett":"1593063223807","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"1","type1":"102","loading_way":"1"}},{"ett":"1593095105466","en":"ad","kv":{"activityId":"1","displayMills":"1966","entry":"3","action":"2","contentType":"0"}},{"ett":"1593051718208","en":"notification","kv":{"ap_time":"1593095336265","action":"2","type":"3","content":""}},{"ett":"1593100021275","en":"comment","kv":{"p_comment_id":4,"addtime":"1593098946009","praise_count":220,"other_id":4,"comment_id":9,"reply_count":151,"userid":4,"content":"抄应螟皮釉倔掉汉蛋蕾街羡晶"}},{"ett":"1593105344120","en":"praise","kv":{"target_id":9,"id":7,"type":1,"add_time":"1593098545976","userid":8}}]}

上面的字符串|前面的不属于JSON字符串,|后面部分才是JSON字符串,可把前面那串数字当做标号或者id

利用JSON解析工具解析JSON字符串后的内容如下图所示:
在这里插入图片描述

2、读取JSON文件数据

2.1、导包

		import org.apache.spark.sql.types._
		import org.apache.spark.sql.functions._
		import spark.implicits._
		import org.apache.spark.sql.functions._
		import org.apache.spark.sql.types._
		import org.apache.spark.sql._

第一步先导入所有需要的包

2.2、上传JSON文件到HDFS文件系统

hdfs dfs -put /opt/op.log /file

2.3、通过文件读取JSON数据存入RDD中

    val fileRDD=sc.textFile("hdfs://192.168.8.99:9000/file/op.log")

3、JSON数据清洗

3.1、转换为JSON格式

  • 读取的JSON文件不完全是一个JSON格式,只是一个字符串
  • 所以需要先将读取到的字符串转换成JSON格式,即将|前面的编号加到|后面的JSON字符串中
  • 将转换后的 jsonRDD 转换为 jsonDataFrame
  • RDD 转 DataFrame 需要导入spark.implicits._
    val jsonStrRDD=fileRDD.map(x=>x.split('|')).map(x=>(x(0),x(1)))
	val jsonRDD = jsonStrRDD.map(x=>{var jsonStr=x._2;jsonStr=jsonStr.substring(0,jsonStr.length-1);jsonStr+",\"id\":\""+x._1+"\"}" })
	val jsonDF = jsonRDD.toDF

jsonDF.show

+--------------------+
|               value|
+--------------------+
|{"cm":{"ln":"-55....|
|{"cm":{"ln":"-114...|
+--------------------+

2.3、get_json_object 函数清洗JSON数据

  • 使用 get_json_object 函数解析获取JSON数据,清洗 value
  • 使用 get_json_object 函数前需要导入 org.apache.spark.sql.functions._
		val jsonDF2=jsonDF.select(get_json_object($"value","$.cm").alias("cm"),
		get_json_object($"value","$.ap").alias("ap"),
		get_json_object($"value","$.et").alias("et"),
		get_json_object($"value","$.id").alias("id"))

jsonDF2.show

+--------------------+---+--------------------+-------------+
|                  cm| ap|                  et|           id|
+--------------------+---+--------------------+-------------+
|{"ln":"-55.0","sv...|app|[{"ett":"15930500...|1593136280858|
|{"ln":"-114.9","s...|app|[{"ett":"15930632...|1593136280858|
+--------------------+---+--------------------+-------------+
  • 再使用 get_json_object 清洗 jsonDF2.show 中的 cm
		val jsonDF3=jsonDF2.select($"id",$"ap",
		get_json_object($"cm","$.ln").alias("ln"),
		get_json_object($"cm","$.sv").alias("sv"),
		get_json_object($"cm","$.os").alias("os"),
		get_json_object($"cm","$.g").alias("g"),
		get_json_object($"cm","$.mid").alias("mid"),
		get_json_object($"cm","$.nw").alias("nw"),
		get_json_object($"cm","$.l").alias("l"),
		get_json_object($"cm","$.vc").alias("vc"),
		get_json_object($"cm","$.hw").alias("hw"),
		get_json_object($"cm","$.ar").alias("ar"),
		get_json_object($"cm","$.uid").alias("uid"),
		get_json_object($"cm","$.t").alias("t"),
		get_json_object($"cm","$.la").alias("la"),
		$"et")

jsonDF3.show

+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|                  et|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|[{"ett":"15930500...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|[{"ett":"15930632...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+

2.4、from_json 函数清洗JSON嵌套数据

  • 清洗 jsonDF3 中的 et
  • 将JSON字符串{"cm":"ap";"et":"cc";"id":"dd"}结构化
		val jsonDF4 = jsonDF3.select($"id",
		$"ap",
		$"ln",
		$"sv",
		$"os",
		$"g",
		$"mid",
		$"nw",
		$"l",
		$"vc",
		$"hw",
		$"ar",
		$"uid",
		$"t",
		$"la",
		from_json($"et",
		ArrayType(StructType(StructField("ett",
		StringType)::StructField("en",
		StringType)::StructField("kv",
		StringType)::Nil)))
		.alias("event"))

jsonDF4.show

+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|               event|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|[[1593050051366, ...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|[[1593063223807, ...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+--------------------+

2.5、explode 函数清洗 JSON 数据

  • 清洗 jsonDF4event
  • explode 函数需要两步完成

第一步:

		val jsonDF5 = jsonDF4.select($"id",
		$"ap",
		$"ln",
		$"sv",
		$"os",
		$"g",
		$"mid",
		$"nw",
		$"l",
		$"vc",
		$"hw",
		$"ar",
		$"uid",
		$"t",
		$"la",
		explode($"event").alias("event"))

第二步:

val jsonDF6 = jsonDF5.select($"id",
$"ap",
$"ln",
$"sv",
$"os",
$"g",
$"mid",
$"nw",
$"l",
$"vc",
$"hw",
$"ar",
$"uid",
$"t",
$"la",
$"event.ett",
$"event.en",
$"event.kv")

jsonDF6.show

+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+--------------------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|          ett|               en|                  kv|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+--------------------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593050051366|          loading|{"extend2":"","lo...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593108791764|               ad|{"activityId":"1"...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593111271266|     notification|{"ap_time":"15930...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593066033562|active_background|{"active_source":...|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593135644347|          comment|{"p_comment_id":1...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593063223807|          loading|{"extend2":"","lo...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593095105466|               ad|{"activityId":"1"...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593051718208|     notification|{"ap_time":"15930...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593100021275|          comment|{"p_comment_id":4...|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593105344120|           praise|{"target_id":9,"i...|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+--------------------+

2.6、from_json 函数、explode 函数清洗

  • 此为最后一步清理,清洗 jsonDF6 中的 kv
  • 该步骤实质上是前面介绍的 2.4、2.5 的综合使用
  • 一共分为三步

第一步:from_json 函数清洗

		val jsonDF7 = jsonDF6.select($"id",$"ap",$"ln",$"sv",$"os",$"g",
		$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"ett",$"en",
		from_json($"kv",
		ArrayType(StructType(StructField("extend2",
		StringType)::StructField("loading_time",
		StringType)::StructField("action",
		StringType)::StructField("extend1",
		StringType)::StructField("type",
		StringType)::StructField("type1",
		StringType)::StructField("activityId",
		StringType)::StructField("displayMills",
		StringType)::StructField("entry",
		StringType)::StructField("contentType",
		StringType)::StructField("ap_time",
		StringType)::StructField("content",
		StringType)::StructField("p_comment_id",
		StringType)::StructField("addtime",
		StringType)::StructField("praise_count",
		StringType)::StructField("other_id",
		StringType)::StructField("comment_id",
		StringType)::StructField("reply_count",
		StringType)::StructField("userid",
		StringType)::StructField("target_id",
		StringType)::StructField("add_time",
		StringType)::StructField("loading_way",
		StringType)::Nil)))
		.alias("kv"))

第二步:explode 函数清洗

		val jsonDF8 = jsonDF7.select($"id",
		$"ap",$"ln",$"sv",$"os",$"g",
		$"mid",$"nw",$"l",$"vc",$"hw",
		$"ar",$"uid",$"t",$"la",$"ett",$"en",
		explode($"kv").alias("kv"))

第三步:生成最终清洗完成 DF

		val jsonDF9 = jsonDF8.select($"id",$"ap",$"ln",$"sv",$"os",$"g",
		$"mid",$"nw",$"l",$"vc",$"hw",$"ar",$"uid",$"t",$"la",$"ett",$"en",
		$"kv.extend2",
		$"kv.loading_time",
		$"kv.action",
		$"kv.extend1",
		$"kv.type",
		$"kv.type1",
		$"kv.activityId",
		$"kv.displayMills",
		$"kv.entry",
		$"kv.contentType",
		$"kv.ap_time",
		$"kv.content",
		$"kv.p_comment_id",
		$"kv.addtime",
		$"kv.praise_count",
		$"kv.other_id",
		$"kv.comment_id",
		$"kv.reply_count",
		$"kv.userid",
		$"kv.target_id",
		$"kv.add_time",
		$"kv.loading_way")

jsonDF9.show

+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+-------+------------+------+-------+----+-----+----------+------------+-----+-----------+-------------+--------------------------+------------+-------------+------------+--------+----------+-----------+------+---------+-------------+-----------+
|           id| ap|    ln|    sv|   os|                 g|mid| nw|  l| vc|      hw| ar|uid|            t|   la|          ett|               en|extend2|loading_time|action|extend1|type|type1|activityId|displayMills|entry|contentType|      ap_time|                   content|p_comment_id|      addtime|praise_count|other_id|comment_id|reply_count|userid|target_id|     add_time|loading_way|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+-------+------------+------+-------+----+-----+----------+------------+-----+-----------+-------------+--------------------------+------------+-------------+------------+--------+----------+-----------+------+---------+-------------+-----------+
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593050051366|          loading|       |          14|     3|       |   2|  201|      null|        null| null|       null|         null|                      null|        null|         null|        null|    null|      null|       null|  null|     null|         null|          1|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593108791764|               ad|   null|        null|     1|   null|null| null|         1|       78522|    1|          0|         null|                      null|        null|         null|        null|    null|      null|       null|  null|     null|         null|       null|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593111271266|     notification|   null|        null|     1|   null|   1| null|      null|        null| null|       null|1593097087883|                          |        null|         null|        null|    null|      null|       null|  null|     null|         null|       null|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593066033562|active_background|   null|        null|  null|   null|null| null|      null|        null| null|       null|         null|                      null|        null|         null|        null|    null|      null|       null|  null|     null|         null|       null|
|1593136280858|app| -55.0|V2.9.6|8.0.4|C6816QZ0@gmail.com|489| 3G| es|  4| 640*960| MX|489|1593123253541|  5.2|1593135644347|          comment|   null|        null|  null|   null|null| null|      null|        null| null|       null|         null|          辑赤蹲慰鸽抿肘捎|           1|1593097573725|         973|       5|         9|         40|     7|     null|         null|       null|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593063223807|          loading|       |           0|     3|       |   1|  102|      null|        null| null|       null|         null|                      null|        null|         null|        null|    null|      null|       null|  null|     null|         null|          1|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593095105466|               ad|   null|        null|     2|   null|null| null|         1|        1966|    3|          0|         null|                      null|        null|         null|        null|    null|      null|       null|  null|     null|         null|       null|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593051718208|     notification|   null|        null|     2|   null|   3| null|      null|        null| null|       null|1593095336265|                          |        null|         null|        null|    null|      null|       null|  null|     null|         null|       null|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593100021275|          comment|   null|        null|  null|   null|null| null|      null|        null| null|       null|         null|抄应螟皮釉倔掉汉蛋蕾街羡晶|           4|1593098946009|         220|       4|         9|        151|     4|     null|         null|       null|
|1593136280858|app|-114.9|V2.7.8|8.0.4|NW0S962J@gmail.com|490| 3G| pt|  8|640*1136| MX|490|1593121224789|-44.4|1593105344120|           praise|   null|        null|  null|   null|   1| null|      null|        null| null|       null|         null|                      null|        null|         null|        null|    null|      null|       null|     8|        9|1593098545976|       null|
+-------------+---+------+------+-----+------------------+---+---+---+---+--------+---+---+-------------+-----+-------------+-----------------+-------+------------+------+-------+----+-----+----------+------------+-----+-----------+-------------+--------------------------+------------+-------------+------------+--------+----------+-----------+------+---------+-------------+-----------+

3、数据保存到hive、mysql

3.1、数据保存到 hive

将 jsonDF9 转为临时视图

		jsonDF9.createTempView("test")

创建 hive 数据库

		spark.sql("create database json")

将临时视图映射到 hive 数据库中

		spark.sql("create table json.js as select * from test")

导入成功如下图所示:

在这里插入图片描述

3.2、保存到 mysql

导包并创建连接

		import java.util.Properties
		val url = "jdbc:mysql://192.168.8.99:3306/kb09db"
	    val prop = new Properties()
	    prop.setProperty("user","root")
	    prop.setProperty("password","ok")
	    prop.setProperty("driver","com.mysql.jdbc.Driver")

jsonDF9 写入 mysql 中

		jsonDF9.write.mode("overwrite").jdbc(url,"json",prop)

导入 mysql 后查看结果如下:

在这里插入图片描述

  • 4
    点赞
  • 15
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值