本篇接着谈谈那些稍微复杂的API。
1) flatMapValues：针对Pair RDD中的每个值应用一个返回迭代器的函数，然后对返回的每个元素都生成一个对应原键的键值对记录
这个方法我最开始接触时候，总是感觉很诧异，不是太理解，现在回想起来主要原因是我接触的第一个flatMapValues的例子是这样的，代码如下：
|
1
2
3
4
|
val
rddPair
:
RDD[(String, Int)]
=
sc.parallelize(List((
"x01"
,
2
), (
"x02"
,
5
), (
"x03"
,
8
), (
"x04"
,
3
), (
"x01"
,
12
), (
"x03"
,
9
)),
1
)
val
rddFlatMapVals
1
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> x to (
6
) }
/* 结果：(x01,2),(x01,3),(x01,4),(x01,5),(x01,6),(x02,5),(x02,6),(x04,3),(x04,4),(x04,5),(x04,6) */
println(
"====flatMapValues 1====:"
+ rddFlatMapVals
1
.collect().mkString(
","
))
|
这个实例使用的是scala里Range这种数据类型，Range数据类型是一个数字的范围，细细讲它的理论也没啥意思，我们看下面的例子吧，代码如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
|
val
list
:
List[Int]
=
List(
1
,
2
,
3
,
4
,
5
,
6
)
val
len
:
Int
=
list.size -
1
val
r
:
Range
=
0
to len
for
(ind <- r){
print(list(ind) +
";"
)
// 1;2;3;4;5;6;
}
println(
""
)
for
(ind <-
0
to len){
print(list(ind) +
";"
)
// 1;2;3;4;5;6;
}
println(
""
)
|
由以上代码我们可以看到0 to 3就是指代0，1，2，3这三个数字，所以我们可以在for循环里指代这个范围。
其实flatMapValues接受的外部方法的返回类型是一个Seq类型，Seq类型在scala里就是一个序列，一个有序的序列，我们可以把他理解成数组，我们来看看下面的例子：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
def
flatMapRange(par
:
Int)
:
Range
=
{
par to
6
}
def
flatMapList(par
:
Int)
:
List[Int]
=
{
List(par +
1000
)
}
def
flatMapSeq(par
:
Int)
:
Seq[Int]
=
{
Seq(par +
6000
)
}
val
rddFlatMapVals
2
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapRange(x) }
/* 结果：(x01,2),(x01,3),(x01,4),(x01,5),(x01,6),(x02,5),(x02,6),(x04,3),(x04,4),(x04,5),(x04,6) */
println(
"====flatMapValues 2====:"
+ rddFlatMapVals
2
.collect().mkString(
","
))
val
rddFlatMapVals
3
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapList(x) }
/* 结果：(x01,1002),(x02,1005),(x03,1008),(x04,1003),(x01,1012),(x03,1009) */
println(
"====flatMapValues 3====:"
+ rddFlatMapVals
3
.collect().mkString(
","
))
val
rddFlatMapVals
4
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapSeq(x) }
/* 结果：(x01,6002),(x02,6005),(x03,6008),(x04,6003),(x01,6012),(x03,6009) */
println(
"====flatMapValues 4====:"
+ rddFlatMapVals
4
.collect().mkString(
","
))
|
谈到flatMapValues这个方法，让我不得不回忆起另外一个与之类似的方法flatMap，我们来看看这个方法的实例吧，代码如下：
|
1
2
3
4
5
6
7
8
9
|
val
rddFlatMap
1
:
RDD[(String,Int)]
=
rddPair.flatMap(x
=
> List((x.
_
1
,x.
_
2
+
3000
)))
// 结果：(x01,3002),(x02,3005),(x03,3008),(x04,3003),(x01,3012),(x03,3009)
println(
"=====flatMap 1======:"
+ rddFlatMap
1
.collect().mkString(
","
))
val
rddFlatMap
2
:
RDD[Int]
=
rddPair.flatMap(x
=
> List(x.
_
2
+
8000
))
// 结果:8002,8005,8008,8003,8012,8009
println(
"=====flatMap 2======:"
+ rddFlatMap
2
.collect().mkString(
","
))
val
rddFlatMap
3
:
RDD[String]
=
rddPair.flatMap(x
=
> List(x.
_
1
+
"@!@"
+ x.
_
2
))
// 结果：x01@!@2,x02@!@5,x03@!@8,x04@!@3,x01@!@12,x03@!@9
println(
"=====flatMap 3======:"
+ rddFlatMap
3
.collect().mkString(
","
))
|
由此可见flatMap方法里的参数也是一个Seq，而且他们之间可以相互替代使用，只不过flatMapValues是让二元组里的第一个元素保持不变的情况下进行计算的（及key值不发生变化）。不过spark不会无缘无故的定义一个flatMapValues,它其实和spark里的分区紧密相关，关于spark的分区知识我会在后面文章里谈谈的。
2) rightOuterJoin，leftOuterJoin，rddCogroup及右连接，左连接和分组函数
我们首先看看他们的使用吧，代码如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
val
rdd
:
RDD[(String,Int)]
=
sc.makeRDD(List((
"x01"
,
2
),(
"x02"
,
5
),(
"x03"
,
9
),(
"x03"
,
21
),(
"x04"
,
76
)))
val
other
:
RDD[(String,Int)]
=
sc.makeRDD(List((
"x01"
,
4
),(
"x02"
,
6
),(
"x03"
,
11
)))
val
rddRight
:
RDD[(String,(Option[Int],Int))]
=
rdd.rightOuterJoin(other)
/* 结果：(x02,(Some(5),6)),(x03,(Some(9),11)),(x03,(Some(21),11)),(x01,(Some(2),4)) */
println(
"====rightOuterJoin====:"
+ rddRight.collect().mkString(
","
))
val
rddLeft
:
RDD[(String,(Int,Option[Int]))]
=
rdd.leftOuterJoin(other)
/* 结果： (x02,(5,Some(6))),(x04,(76,None)),(x03,(9,Some(11))),(x03,(21,Some(11))),(x01,(2,Some(4))) */
println(
"====leftOuterJoin====:"
+ rddLeft.collect().mkString(
","
))
val
rddSome
=
rddLeft.filter(x
=
> x.
_
2
.
_
2
.isEmpty
==
false
)
// 过滤掉None的记录
/* 结果: (x02,(5,Some(6))),(x03,(9,Some(11))),(x03,(21,Some(11))),(x01,(2,Some(4)))*/
println(
"====rddSome===:"
+ rddSome.collect().mkString(
","
))
val
rddCogroup
:
RDD[(String, (Iterable[Int], Iterable[Int]))]
=
rdd.cogroup(other)
/* 结果: (x02,(CompactBuffer(5),CompactBuffer(6))),(x04,(CompactBuffer(76),CompactBuffer())),(x03,(CompactBuffer(9, 21),CompactBuffer(11))),(x01,(CompactBuffer(2),CompactBuffer(4)))*/
println(
"===cogroup====:"
+ rddCogroup.collect().mkString(
","
))
|
这三个方法很好理解，就和关系数据库里的左右连接，分组一样，不过它们的返回值在我刚学习spark时候很是疑惑了半天，这里就好好说下它们的返回值，这其实就是学习下scala的数据类型了。
首先是Some数据类型，Some并不是一个直接操作的数据类型，它属于Option这个数据结构的，其实None也是Option里的数据结构，Some里面只能放一个元素，例如Some(1),Some((1,2)),为什么scala里还要这么繁琐的定义一个Option，并在其中还定义一个Some和一个None的结构呢？我们首先看看下面代码：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
def
optionSome()
:
Unit
=
{
/*
* =======option for 1=========
0:3
2:8
3:11
=======option for 1=========
=======option for 2=========
0:3
1:None
2:8
3:11
=======option for 2=========
*/
val
list
:
List[Option[Int]]
=
List(Some(
3
),None,Some(
8
),Some(
11
))
println(
"=======option for 1========="
)
for
(i <-
0
until list.size){
if
(!list(i).isEmpty){
println(i +
":"
+ list(i).get)
}
}
println(
"=======option for 1========="
)
println(
"=======option for 2========="
)
for
(j <-
0
until list.size){
val
res
=
list(j)
match
{
case
None
=
> println(j +
":None"
)
case
_
=
> println(j +
":"
+ list(j).get)
}
}
println(
"=======option for 2========="
)
}
|
Option数据结构其实想要表达的是一个数据集合，这个数据集合里要么有值，要么没值，这点在左右连接查询里就非常有用，其实左右连接最后的结果就是要么关联上了要么没有关联上。
分组cogroup返回的结构是CompactBuffer，CompactBuffer并不是scala里定义的数据结构，而是spark里的数据结构，它继承自一个迭代器和序列，所以它的返回值是一个很容易进行循环遍历的集合，这点很符合cogroup的返回值类型。
好了，这篇内容就写完了，下一篇文章我将要简单聊聊spark分区，后面应该暂时会停停spark的学习，要搞搞前端的一些技术，这都是因为工作需要了。
最后我将完整示例代码给大家分享下，代码如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
|
package
cn.com.sparktest
import
org.apache.spark.SparkConf
import
org.apache.spark.SparkContext
import
org.apache.spark.SparkContext.
_
import
org.apache.spark.rdd.RDD
import
scala.collection.immutable.List
import
org.apache.spark.util.collection.CompactBuffer
object
ScalaTest {
val
conf
:
SparkConf
=
new
SparkConf().setAppName(
"spark scala"
).setMaster(
"local[2]"
)
val
sc
:
SparkContext
=
new
SparkContext(conf)
def
aggrFtnOne(par
:
((Int, Int), Int))
:
(Int, Int)
=
{
/*
*aggregate的初始值为(0,0):
====aggrFtnOne Param===:((0,0),1)
====aggrFtnOne Param===:((1,1),2)
====aggrFtnOne Param===:((3,2),3)
====aggrFtnOne Param===:((6,3),4)
====aggrFtnOne Param===:((10,4),5)*/
/*
*aggregate的初始值为(1,1):
====aggrFtnOne Param===:((1,1),1)
====aggrFtnOne Param===:((2,2),2)
====aggrFtnOne Param===:((4,3),3)
====aggrFtnOne Param===:((7,4),4)
====aggrFtnOne Param===:((11,5),5)
* */
println(
"====aggrFtnOne Param===:"
+ par.toString())
val
ret
:
(Int, Int)
=
(par.
_
1
.
_
1
+ par.
_
2
, par.
_
1
.
_
2
+
1
)
ret
}
def
aggrFtnTwo(par
:
((Int, Int), (Int, Int)))
:
(Int, Int)
=
{
/*aggregate的初始值为(0,0):::::((0,0),(15,5))*/
/*aggregate的初始值为(1,1):::::((1,1),(16,6))*/
println(
"====aggrFtnTwo Param===:"
+ par.toString())
val
ret
:
(Int, Int)
=
(par.
_
1
.
_
1
+ par.
_
2
.
_
1
, par.
_
1
.
_
2
+ par.
_
2
.
_
2
)
ret
}
def
foldFtn(par
:
(Int, Int))
:
Int
=
{
/*fold初始值为0：
=====foldFtn Param====:(0,1)
=====foldFtn Param====:(1,2)
=====foldFtn Param====:(3,3)
=====foldFtn Param====:(6,4)
=====foldFtn Param====:(10,5)
=====foldFtn Param====:(0,15)
* */
/*
* fold初始值为1:
=====foldFtn Param====:(1,1)
=====foldFtn Param====:(2,2)
=====foldFtn Param====:(4,3)
=====foldFtn Param====:(7,4)
=====foldFtn Param====:(11,5)
=====foldFtn Param====:(1,16)
* */
println(
"=====foldFtn Param====:"
+ par.toString())
val
ret
:
Int
=
par.
_
1
+ par.
_
2
ret
}
def
reduceFtn(par
:
(Int,Int))
:
Int
=
{
/*
* ======reduceFtn Param=====:1:2
======reduceFtn Param=====:3:3
======reduceFtn Param=====:6:4
======reduceFtn Param=====:10:5
*/
println(
"======reduceFtn Param=====:"
+ par.
_
1
+
":"
+ par.
_
2
)
par.
_
1
+ par.
_
2
}
def
sparkRDDHandle()
:
Unit
=
{
val
rddInt
:
RDD[Int]
=
sc.parallelize(List(
1
,
2
,
3
,
4
,
5
),
1
)
val
rddAggr
1
:
(Int, Int)
=
rddInt.aggregate((
0
,
0
))((x, y)
=
> (x.
_
1
+ y, x.
_
2
+
1
), (x, y)
=
> (x.
_
1
+ y.
_
1
, x.
_
2
+ y.
_
2
))
println(
"====aggregate 1====:"
+ rddAggr
1
.toString())
// (15,5)
val
rddAggr
2
:
(Int, Int)
=
rddInt.aggregate((
0
,
0
))((x, y)
=
> aggrFtnOne(x, y), (x, y)
=
> aggrFtnTwo(x, y))
// 参数可以省略元组的括号
println(
"====aggregate 2====:"
+ rddAggr
2
.toString())
// (15,5)
val
rddAggr
3
:
(Int, Int)
=
rddInt.aggregate((
1
,
1
))((x, y)
=
> aggrFtnOne((x, y)), (x, y)
=
> aggrFtnTwo((x, y)))
// 参数使用元组的括号
println(
"====aggregate 3====:"
+ rddAggr
3
.toString())
// (17,7)
val
rddAggr
4
:
(Int, Int)
=
rddInt.aggregate((
1
,
0
))((x, y)
=
> (x.
_
1
* y, x.
_
2
+
1
), (x, y)
=
> (x.
_
1
* y.
_
1
, x.
_
2
+ y.
_
2
))
println(
"====aggregate 4====:"
+ rddAggr
4
.toString())
// (120,5)
val
rddFold
1
:
Int
=
rddInt.fold(
0
)((x, y)
=
> x + y)
println(
"====fold 1====:"
+ rddFold
1
)
// 15
val
rddFold
2
:
Int
=
rddInt.fold(
0
)((x, y)
=
> foldFtn(x, y))
// 参数可以省略元组的括号
println(
"====fold 2=====:"
+ rddFold
2
)
// 15
val
rddFold
3
:
Int
=
rddInt.fold(
1
)((x, y)
=
> foldFtn((x, y)))
// 参数使用元组的括号
println(
"====fold 3====:"
+ rddFold
3
)
// 17
val
rddReduce
1
:
Int
=
rddInt.reduce((x,y)
=
> x + y)
println(
"====rddReduce 1====:"
+ rddReduce
1
)
// 15
val
rddReduce
2
:
Int
=
rddInt.reduce((x,y)
=
> reduceFtn(x,y))
println(
"====rddReduce 2====:"
+ rddReduce
2
)
// 15
}
def
combineFtnOne(par
:
Int)
:
(Int,Int)
=
{
/*
* ====combineFtnOne Param====:2
====combineFtnOne Param====:5
====combineFtnOne Param====:8
====combineFtnOne Param====:3
*/
println(
"====combineFtnOne Param====:"
+ par)
val
ret
:
(Int,Int)
=
(par,
1
)
ret
}
def
combineFtnTwo(par
:
((Int,Int),Int))
:
(Int,Int)
=
{
/*
====combineFtnTwo Param====:((2,1),12)
====combineFtnTwo Param====:((8,1),9)
* */
println(
"====combineFtnTwo Param====:"
+ par.toString())
val
ret
:
(Int,Int)
=
(par.
_
1
.
_
1
+ par.
_
2
,par.
_
1
.
_
2
+
1
)
ret
}
def
combineFtnThree(par
:
((Int,Int),(Int,Int)))
:
(Int,Int)
=
{
/*
* 无结果打印
*/
println(
"@@@@@@@@@@@@@@@@@@"
)
println(
"====combineFtnThree Param===:"
+ par.toString())
val
ret
:
(Int,Int)
=
(par.
_
1
.
_
1
+ par.
_
2
.
_
1
,par.
_
1
.
_
2
+ par.
_
2
.
_
2
)
ret
}
def
flatMapRange(par
:
Int)
:
Range
=
{
par to
6
}
def
flatMapList(par
:
Int)
:
List[Int]
=
{
List(par +
1000
)
}
def
flatMapSeq(par
:
Int)
:
Seq[Int]
=
{
Seq(par +
6000
)
}
def
sparkPairRDD()
:
Unit
=
{
val
rddPair
:
RDD[(String, Int)]
=
sc.parallelize(List((
"x01"
,
2
), (
"x02"
,
5
), (
"x03"
,
8
), (
"x04"
,
3
), (
"x01"
,
12
), (
"x03"
,
9
)),
1
)
/* def combineByKey[C](createCombiner: Int => C, mergeValue: (C, Int) => C, mergeCombiners: (C, C) => C): RDD[(String, C)] */
val
rddCombine
1
:
RDD[(String,(Int,Int))]
=
rddPair.combineByKey(x
=
> (x,
1
), (com
:
(Int, Int), x)
=
> (com.
_
1
+ x, com.
_
2
+
1
), (com
1
:
(Int, Int), com
2
:
(Int, Int))
=
> (com
1
.
_
1
+ com
2
.
_
1
, com
1
.
_
2
+ com
2
.
_
2
))
println(
"====combineByKey 1====:"
+ rddCombine
1
.collect().mkString(
","
))
// (x02,(5,1)),(x03,(17,2)),(x01,(14,2)),(x04,(3,1))
val
rddCombine
2
:
RDD[(String,(Int,Int))]
=
rddPair.combineByKey(x
=
> combineFtnOne(x), (com
:
(Int, Int), x)
=
> combineFtnTwo(com,x), (com
1
:
(Int, Int), com
2
:
(Int, Int))
=
> combineFtnThree(com
1
,com
2
))
println(
"=====combineByKey 2====:"
+ rddCombine
2
.collect().mkString(
","
))
// (x02,(5,1)),(x03,(17,2)),(x01,(14,2)),(x04,(3,1))
val
rddKeys
:
RDD[String]
=
rddPair.keys
/*结果:x01,x02,x03,x04,x01,x03 注意调用keys方法时候不能加上括号，否则会报错*/
println(
"====keys====:"
+ rddKeys.collect().mkString(
","
))
val
rddVals
:
RDD[Int]
=
rddPair.values
/*结果：2,5,8,3,12,9 注意调用values方法时候不能加上括号，否则会报错*/
println(
"=====values=====:"
+ rddVals.collect().mkString(
","
))
val
rddFlatMapVals
1
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> x to (
6
) }
/* 结果：(x01,2),(x01,3),(x01,4),(x01,5),(x01,6),(x02,5),(x02,6),(x04,3),(x04,4),(x04,5),(x04,6) */
println(
"====flatMapValues 1====:"
+ rddFlatMapVals
1
.collect().mkString(
","
))
val
rddFlatMapVals
2
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapRange(x) }
/* 结果：(x01,2),(x01,3),(x01,4),(x01,5),(x01,6),(x02,5),(x02,6),(x04,3),(x04,4),(x04,5),(x04,6) */
println(
"====flatMapValues 2====:"
+ rddFlatMapVals
2
.collect().mkString(
","
))
val
rddFlatMapVals
3
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapList(x) }
/* 结果：(x01,1002),(x02,1005),(x03,1008),(x04,1003),(x01,1012),(x03,1009) */
println(
"====flatMapValues 3====:"
+ rddFlatMapVals
3
.collect().mkString(
","
))
val
rddFlatMapVals
4
:
RDD[(String,Int)]
=
rddPair.flatMapValues { x
=
> flatMapSeq(x) }
/* 结果：(x01,6002),(x02,6005),(x03,6008),(x04,6003),(x01,6012),(x03,6009) */
println(
"====flatMapValues 4====:"
+ rddFlatMapVals
4
.collect().mkString(
","
))
val
rddFlatMap
1
:
RDD[(String,Int)]
=
rddPair.flatMap(x
=
> List((x.
_
1
,x.
_
2
+
3000
)))
// 结果：(x01,3002),(x02,3005),(x03,3008),(x04,3003),(x01,3012),(x03,3009)
println(
"=====flatMap 1======:"
+ rddFlatMap
1
.collect().mkString(
","
))
val
rddFlatMap
2
:
RDD[Int]
=
rddPair.flatMap(x
=
> List(x.
_
2
+
8000
))
// 结果:8002,8005,8008,8003,8012,8009
println(
"=====flatMap 2======:"
+ rddFlatMap
2
.collect().mkString(
","
))
val
rddFlatMap
3
:
RDD[String]
=
rddPair.flatMap(x
=
> List(x.
_
1
+
"@!@"
+ x.
_
2
))
// 结果：x01@!@2,x02@!@5,x03@!@8,x04@!@3,x01@!@12,x03@!@9
println(
"=====flatMap 3======:"
+ rddFlatMap
3
.collect().mkString(
","
))
}
def
optionSome()
:
Unit
=
{
/*
* =======option for 1=========
0:3
2:8
3:11
=======option for 1=========
=======option for 2=========
0:3
1:None
2:8
3:11
=======option for 2=========
*/
val
list
:
List[Option[Int]]
=
List(Some(
3
),None,Some(
8
),Some(
11
))
println(
"=======option for 1========="
)
for
(i <-
0
until list.size){
if
(!list(i).isEmpty){
println(i +
":"
+ list(i).get)
}
}
println(
"=======option for 1========="
)
println(
"=======option for 2========="
)
for
(j <-
0
until list.size){
val
res
=
list(j)
match
{
case
None
=
> println(j +
":None"
)
case
_
=
> println(j +
":"
+ list(j).get)
}
}
println(
"=======option for 2========="
)
}
def
pairRDDJoinGroup()
:
Unit
=
{
val
rdd
:
RDD[(String,Int)]
=
sc.makeRDD(List((
"x01"
,
2
),(
"x02"
,
5
),(
"x03"
,
9
),(
"x03"
,
21
),(
"x04"
,
76
)))
val
other
:
RDD[(String,Int)]
=
sc.makeRDD(List((
"x01"
,
4
),(
"x02"
,
6
),(
"x03"
,
11
)))
val
rddRight
:
RDD[(String,(Option[Int],Int))]
=
rdd.rightOuterJoin(other)
/* 结果：(x02,(Some(5),6)),(x03,(Some(9),11)),(x03,(Some(21),11)),(x01,(Some(2),4)) */
println(
"====rightOuterJoin====:"
+ rddRight.collect().mkString(
","
))
val
rddLeft
:
RDD[(String,(Int,Option[Int]))]
=
rdd.leftOuterJoin(other)
/* 结果： (x02,(5,Some(6))),(x04,(76,None)),(x03,(9,Some(11))),(x03,(21,Some(11))),(x01,(2,Some(4))) */
println(
"====leftOuterJoin====:"
+ rddLeft.collect().mkString(
","
))
val
rddSome
=
rddLeft.filter(x
=
> x.
_
2
.
_
2
.isEmpty
==
false
)
// 过滤掉None的记录
/* 结果: (x02,(5,Some(6))),(x03,(9,Some(11))),(x03,(21,Some(11))),(x01,(2,Some(4)))*/
println(
"====rddSome===:"
+ rddSome.collect().mkString(
","
))
val
rddCogroup
:
RDD[(String, (Iterable[Int], Iterable[Int]))]
=
rdd.cogroup(other)
/* 结果: (x02,(CompactBuffer(5),CompactBuffer(6))),(x04,(CompactBuffer(76),CompactBuffer())),(x03,(CompactBuffer(9, 21),CompactBuffer(11))),(x01,(CompactBuffer(2),CompactBuffer(4)))*/
println(
"===cogroup====:"
+ rddCogroup.collect().mkString(
","
))
}
def
scalaBasic(){
val
its
:
Iterable[Int]
=
Iterable(
1
,
2
,
3
,
4
,
5
)
its.foreach { x
=
> print(x +
","
) }
// 1,2,3,4,5,
val
tuple
2
Param
1
:
Tuple
2
[String,Int]
=
Tuple
2
(
"x01"
,
12
)
// 标准定义二元组
val
tuple
2
Param
2
:
(String,Int)
=
(
"x02"
,
29
)
// 字面量定义二元组
/* 结果: x01:12*/
println(
"====tuple2Param1====:"
+ tuple
2
Param
1
.
_
1
+
":"
+ tuple
2
Param
1
.
_
2
)
/* 结果: x02:29 */
println(
"====tuple2Param2====:"
+ tuple
2
Param
2
.
_
1
+
":"
+ tuple
2
Param
2
.
_
2
)
val
tuple
6
Param
1
:
Tuple
6
[String,Int,Int,Int,Int,String]
=
Tuple
6
(
"xx01"
,
1
,
2
,
3
,
4
,
"x1x"
)
// 标准定义6元组
val
tuple
6
Param
2
:
(String,Int,Int,Int,Int,String)
=
(
"xx02"
,
1
,
2
,
3
,
4
,
"x2x"
)
// 字面量定义6元组
/* 结果: xx01:1:2:3:4:x1x */
println(
"====tuple6Param1====:"
+ tuple
6
Param
1
.
_
1
+
":"
+ tuple
6
Param
1
.
_
2
+
":"
+ tuple
6
Param
1
.
_
3
+
":"
+ tuple
6
Param
1
.
_
4
+
":"
+ tuple
6
Param
1
.
_
5
+
":"
+ tuple
6
Param
1
.
_
6
)
/* 结果: xx02:1:2:3:4:x2x */
println(
"====tuple6Param2====:"
+ tuple
6
Param
2
.
_
1
+
":"
+ tuple
6
Param
2
.
_
2
+
":"
+ tuple
6
Param
2
.
_
3
+
":"
+ tuple
6
Param
2
.
_
4
+
":"
+ tuple
6
Param
2
.
_
5
+
":"
+ tuple
6
Param
2
.
_
6
)
val
list
:
List[Int]
=
List(
1
,
2
,
3
,
4
,
5
,
6
)
val
len
:
Int
=
list.size -
1
val
r
:
Range
=
0
to len
for
(ind <- r){
print(list(ind) +
";"
)
// 1;2;3;4;5;6;
}
println(
""
)
for
(ind <-
0
to len){
print(list(ind) +
";"
)
// 1;2;3;4;5;6;
}
println(
""
)
}
def
main(args
:
Array[String])
:
Unit
=
{
scalaBasic()
optionSome()
sparkRDDHandle()
sparkPairRDD()
pairRDDJoinGroup()
}
}
|
2381
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
