pythonsparkfilter_过滤RDD基于条件并在Spark python中提取匹配的数据

我有这样的数据,

cl_id cn_id cn_value

10004, 77173296 ,390.0

10004, 77173299 ,376.0

10004, 77173300 ,0.0

20005, 77173296 ,0.0

20005, 77173299 ,6.0

2005, 77438800 ,2.0

Cl_id ID：10004,20005

按10004过滤

10004, 77173296 ,390.0

10004, 77173299 ,376.0

按20005过滤

20005, 77173296 ,0.0

20005, 77173299 ,6.0

现在我想要返回RDD之类的,

10004,cn_id,x1(77173296.value,77173300.value) ==> 10004,77173296,390.0,376.0

20005,cn_id,x1(77173296.value,77173300.value) ==> 20005,77173296,0.0,6.0

我想对这个return_RDD执行一些操作：

def cal_for(rdd_list):

#list.map(position1).filter(cn_id for this formula)-> calculate that formula -> store in a separate RDD -> Return that RDD

rdd_list = rdd_list.map(lambda line:line.split(','))

new_list = rdd_list.map(lambda x: (x[0]+', '+x[1],float(x[2])))

new_list = rdd_list.filter(lambda x: x[1] == '77173296' && x[1] == '77173299')

## then get the RDD containing respective cn_values for cn_id 77173296 & cn_id 77173299

## and apply the following formula whre a=77173296.value b=77173299.value for cl_id 1004

try:

# want to process RDD with this Formula

return ((float(a)/float(a+b))*100)

except ZeroDivisionError:

return 0

#return or save cal_RDD

不是过滤RDD两次,修改和重新组合生成的RDD,只需按id分组,然后映射值以进行所需的任何更改.如果要根据某些条件进一步限制结果,请在映射时执行过滤.

我真的不能给你一个更准确的答案：

a)看起来你还没有真正尝试过这个,而且b)我不完全确定你想要什么.