以下是两个具有相同列名的Dataset按照同名列相等进行join操作,join结果中同名列只会出现一个:
val df1 = Seq((1, 2, 3),(1, 1, 1)).toDF("a", "b", "c")
val df2 = Seq((1, 2, 4),(2, 2, 2)).toDF("a", "b", "c")
df1.show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 2| 3|
| 1| 1| 1|
+---+---+---+
df2.show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 2| 4|
| 2| 2| 2|
+---+---+---+
df1.join(df2, Seq("a","b"), "inner").show
+---+---+---+---+
| a| b| c| c|
+---+---+---+---+
| 1| 2| 3| 4|
+---+---+---+---+
df1.join(df2, Seq("a","b"), "outer").show
+---+---+----+----+
| a| b| c| c|
+---+---+----+----+
| 2| 2|null| 2|
| 1| 2| 3| 4|
| 1| 1| 1|null|
+---+---+----+----+
df1.join(df2, Seq("a","b"), "right").show
+---+---+----+---+
| a| b| c| c|
+---+---+----+---+
| 1| 2| 3| 4|
| 2| 2|null| 2|
+---+---+----+---+
df1.join(df2, Seq("a","b"), "left").show
+---+---+---+----+
| a| b| c| c|
+---+---+---+----+
| 1| 2| 3| 4|
| 1| 1| 1|null|
+---+---+---+----+
df1.join(df2, Seq("a","b"), "leftsemi").show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 2| 3|
+---+---+---+
df1.join(df2, Seq("a","b"), "leftanti").show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 1| 1|
+---+---+---+
以下是具有不同列名的Dataset按照不同名列相等进行join操作,join结果中两个不同名列都会出现(因为列名不同不能只保留某一列):
val df1 = Seq((1, 2, 3),(1, 1, 1)).toDF("a", "b", "c")
val df2 = Seq((1, 2, 4),(2, 2, 2)).toDF("a1", "b1", "c1")
df1.show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 2| 3|
| 1| 1| 1|
+---+---+---+
df2.show
+---+---+---+
| a1| b1| c1|
+---+---+---+
| 1| 2| 4|
| 2| 2| 2|
+---+---+---+
df1.join(df2, col("a") === col("a1") && col("b") === col("b1"), "outer").show
+----+----+----+----+----+----+
| a| b| c| a1| b1| c1|
+----+----+----+----+----+----+
|null|null|null| 2| 2| 2|
| 1| 2| 3| 1| 2| 4|
| 1| 1| 1|null|null|null|
+----+----+----+----+----+----+
若是具有部分相同、部分不同列名的两个Dataset按照部分相同、部分不同列相等进行join操作,有以下几种方式:
val df1 = Seq((1, 2, 3),(1, 1, 1)).toDF("a", "b", "c")
val df2 = Seq((1, 2, 4),(2, 2, 2)).toDF("a", "b1", "d")
df1.show
+---+---+---+
| a| b| c|
+---+---+---+
| 1| 2| 3|
| 1| 1| 1|
+---+---+---+
df2.show
+---+---+---+
| a| b1| d|
+---+---+---+
| 1| 2| 4|
| 2| 2| 2|
+---+---+---+
//join条件:df1("a") == df2("a") && df1("b") == df2("b1")
//若是直接join会报错:org.apache.spark.sql.AnalysisException: Reference 'a' is ambiguous, could be:...
df1.join(df2, col("a") === col("a") && col("b") === col("b1"), "outer").show
//可以改为这样:
df1.join(df2, df1("a") === df2("a") && col("b") === col("b1"), "outer").show
+----+----+----+----+----+----+
| a| b| c| a| b1| d|
+----+----+----+----+----+----+
|null|null|null| 2| 2| 2|
| 1| 2| 3| 1| 2| 4|
| 1| 1| 1|null|null|null|
+----+----+----+----+----+----+
//当然也可以将其中一个Dataset的列改名,改为都相同或都不同,再用上面的方法join
df1.join(df2.withColumnRenamed("b1", "b"), Seq("a", "b"), "outer").show
+---+---+----+----+
| a| b| c| d|
+---+---+----+----+
| 2| 2|null| 2|
| 1| 2| 3| 4|
| 1| 1| 1|null|
+---+---+----+----+
//还可以用Dataset的as方法(与alias方法等效),给Dataset命名,然后消除歧义。(Dataset的别名类似SQL中表的别名)
df1.alias("df1")
.join(df2.as("df2"), col("df1.a") === col("df2.a") && col("b") === col("b1"), "outer")
.show
+----+----+----+----+----+----+
| a| b| c| a| b1| d|
+----+----+----+----+----+----+
|null|null|null| 2| 2| 2|
| 1| 2| 3| 1| 2| 4|
| 1| 1| 1|null|null|null|
+----+----+----+----+----+----+
//如果只想保留df2的a列:
val t = df1.alias("df1")
.join(df2.as("df2"), col("df1.a") === col("df2.a") && col("b") === col("b1"), "outer")
.drop(col("df1.a")).show
+----+----+----+----+----+
| b| c| a| b1| d|
+----+----+----+----+----+
|null|null| 2| 2| 2|
| 2| 3| 1| 2| 4|
| 1| 1|null|null|null|
+----+----+----+----+----+
补充:
Dataset的as方法(与alias方法等效):为Dataset对象起别名,Dataset的别名类似SQL中表的别名。
val df = Seq((1, 2),(1, 1)).toDF("a", "b")
df.select("a").show
+---+
| a|
+---+
| 1|
| 1|
+---+
df.select("df.a").show
//报错:org.apache.spark.sql.AnalysisException: cannot resolve '`df.a`' given input columns: [a, b];
df.as("df").select("df.a").show
+---+
| a|
+---+
| 1|
| 1|
+---+
注意:
以上Dataset的as方法是指的:def as(alias: Symbol): Dataset[T]。
另外Dataset还有一个as方法:def as[U](implicit arg0: Encoder[U]): Dataset[U],这篇文章和此方法无关。