文章目录
一、数据源
转换之前先看下数据结构
多行存在空值需要过滤,不同的状态(yes、maybe、invited、no)存在多个值,需要转换成(events userid status)的状态
val df = spark.read.format("csv").option("header","true").load("file:///opt/data/event\_attendees.csv")
scala> df.printSchema
root
|-- event: string (nullable = true)
|-- yes: string (nullable = true)
|-- maybe: string (nullable = true)
|-- invited: string (nullable = true)
|-- no: string (nullable = true
二、首先考虑单独两行映射
df.filter(col("yes").isNotNull).select(col("event"),col("yes")).withColumn("userid",explode(split(col("yes")," "))).drop($"yes").withColumn("status",lit("yes")).show(3)
+----------+----------+------+
| event| userid|status|
+----------+----------+------+
|1159822043|1975964455| yes|
|1159822043| 252302513| yes|
|1159822043|4226086795| yes|
+----------+----------+------+
only showing top 3 rows
三、同理将其余隔行依次映射
scala> val no = df.filter(col("no").isNotNull).select(col("event"),col("no")).withColumn("userid",explode(split(col("no")," "))).drop($"no").withColumn("status",lit("no"))
+----------+----------+------+
| event| userid|status|
+----------+----------+------+
|1159822043|3575574655| no|
|1159822043|1077296663| no|
|1186208412|1728988561| no|
+----------+----------+------+
only showing top 3 rows
no: Unit = ()
scala> val invited = df.filter(col("invited").isNotNull).select(col("event"),col("invited")).withColumn("userid",explode(split(col("invited")," "))).drop($"invited").withColumn("status",lit("invited")).show(3)
+----------+----------+-------+
| event| userid| status|
+----------+----------+-------+
|1159822043|1723091036|invited|
|1159822043|3795873583|invited|
|1159822043|4109144917|invited|
+----------+----------+-------+
![img](https://img-blog.csdnimg.cn/img_convert/238f807c8ff19582fae4885f58b0b32a.png)
![img](https://img-blog.csdnimg.cn/img_convert/70388f603ac7bc8d31c1e4f22f1b4ca6.png)
**网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。**
**[需要这份系统化资料的朋友,可以戳这里获取](https://bbs.csdn.net/topics/618545628)**
**一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!**
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!**