RDD基本命令

sc.master

'local[*]'

create RDD demo

intRDD=sc.parallelize([3,1,2,5,6])
intRDD.collect()

[3, 1, 2, 5, 6]

stringRDD=sc.parallelize(["Apple","Orange","Banana","Grape","Apple"])
stringRDD.collect()

['Apple', 'Orange', 'Banana', 'Grape', 'Apple']

单RDD转换

map

• nameFunction and noNameFunction

def addone(x):
    return (x+1)
intRDD.map(addone).collect()

[4, 2, 3, 6, 7]

intRDD.map(lambda x: x+1).collect()

[4, 2, 3, 6, 7]

stringRDD.map(lambda x:'fruit:'+x).collect()

['fruit:Apple', 'fruit:Orange', 'fruit:Banana', 'fruit:Grape', 'fruit:Apple']

filter

intRDD.filter(lambda x: x<3).collect()

[1, 2]

intRDD.filter(lambda x:1<x and x<5).collect()

[3, 2]

stringRDD.filter(lambda x: "ra" in x).collect()

['Orange', 'Grape']

distinct

intRDD.distinct().collect()

[1, 5, 2, 6, 3]

stringRDD.distinct().collect()

['Orange', 'Apple', 'Banana', 'Grape']

randomSplit

sRDD=intRDD.randomSplit([0.4,0.6])
sRDD[0].collect()

[1, 2]

sRDD[1].collect()

[3, 5, 6]

groupby

gRDD=intRDD.groupBy(
        lambda x:'even' if (x%2==0) else 'odd').collect()

print('even')
print(list(gRDD[0][1]))
print('odd')
print(gRDD[1][1])

even
[2, 6]
odd
<pyspark.resultiterable.ResultIterable object at 0x7f9ba805d438>

多个RDD转换运算

intRDD1=sc.parallelize([3,1,2,5,5])
intRDD2=sc.parallelize([5,6])
intRDD3=sc.parallelize([2,7])

并集union

intRDD1.union(intRDD2).union(intRDD3).collect()

[3, 1, 2, 5, 5, 5, 6, 2, 7]

交集intersection

intRDD1.intersection(intRDD2).collect()

[5]

差集 subtract

intRDD1.subtract(intRDD2).collect()

[1, 2, 3]

笛卡尔积乘积 cartesian

intRDD1.cartesian(intRDD2).collect()

[(3, 5),
 (3, 6),
 (1, 5),
 (1, 6),
 (2, 5),
 (2, 6),
 (5, 5),
 (5, 5),
 (5, 6),
 (5, 6)]

动作 运算

• first() 读取第一项数据
• take(2) 取出前两项数据
• takeOrdered(3) 从小到大排序，取出前三项数据
• takeOrdered(3,key=lambda x:-x) 从大到小排序，取出前三项

统计功能

• stats()
• min()
• max()
• stdev()
• count()
• sum()
• mean()

RDD key-value transformation

kvRDD1=sc.parallelize([(3,4),(3,6),(5,6),(1,2)])
kvRDD2=sc.parallelize([(3,8)])

kvRDD1.collect()

[(3, 4), (3, 6), (5, 6), (1, 2)]

kvRDD2.collect()

[(3, 8)]

join

kvRDD1.join(kvRDD2).collect()

[(3, (4, 8)), (3, (6, 8))]

leftOuterJoin

kvRDD1.leftOuterJoin(kvRDD2).collect()

[(1, (2, None)), (3, (4, 8)), (3, (6, 8)), (5, (6, None))]

rightOuterJoin

kvRDD1.rightOuterJoin(kvRDD2).collect()

[(3, (4, 8)), (3, (6, 8))]

subtractByKey

kvRDD1.subtractByKey(kvRDD2).collect()

[(1, 2), (5, 6)]

RDD key-value Action

key-value first

kvFirst=kvRDD1.first()
print(kvFirst[0])
print(kvFirst[1])

3
4

key count

kvRDD1.countByKey()

defaultdict(int, {1: 1, 3: 2, 5: 1})

create key-value map –>collectAsMap

KV=kvRDD1.collectAsMap()
KV

{1: 2, 3: 6, 5: 6}

print(type(KV))
print(KV[3])

<class 'dict'>
6

input key to get value

kvRDD1.lookup(3)

[4, 6]