pythonsparkfilter_python – 基于Spark中另一个RDD进行过滤

我想只保留在第二个表中引用了具有一个签名ID的雇员.

Employee table

LastName DepartmentID

Rafferty 31

Jones 33

Heisenberg 33

Robinson 34

Smith 34

Department table

DepartmentID

31

33

我已经尝试了以下代码不起作用：

employee = [['Raffery',31], ['Jones',33], ['Heisenberg',33], ['Robinson',34], ['Smith',34]]

department = [31,33]

employee = sc.parallelize(employee)

department = sc.parallelize(department)

employee.filter(lambda e: e[1] in department).collect()

Py4JError: An error occurred while calling o344.__getnewargs__. Trace:

py4j.Py4JException: Method __getnewargs__([]) does not exist

有任何想法吗？我在使用Spark 1.1.0与Python.但是,我会接受Scala或Python的答案.