pyspark 条件,pyspark加入多个条件

I want ask if you have any idea about how i can specify lot of conditions in

pyspark when i use .join()

Example :

with hive :

query= "select a.NUMCNT,b.NUMCNT as RNUMCNT ,a.POLE,b.POLE as RPOLE,a.ACTIVITE,b.ACTIVITE as RACTIVITE FROM rapexp201412 b \

join rapexp201412 a where (a.NUMCNT=b.NUMCNT and a.ACTIVITE = b.ACTIVITE and a.POLE =b.POLE )\

But in pyspark I don't know how to make it because the following:

df_rapexp201412.join(df_aeveh,df_rapexp2014.ACTIVITE==df_rapexp2014.ACTIVITE and df_rapexp2014.POLE==df_aeveh.POLE,'inner')

does not work!!

解决方案

Quoting from spark docs:

join(other, on=None, how=None) Joins with another DataFrame, using the

given join expression.

The following performs a full outer join between df1 and df2.

Parameters: other – Right side of the join on – a string for join

column name, a list of column names, , a join expression (Column) or a

list of Columns. If on is a string or a list of string indicating the

name of the join column(s), the column(s) must exist on both sides,

and this performs an inner equi-join. how – str, default ‘inner’. One

of inner, outer, left_outer, right_outer, semijoin.

>>> df.join(df2, df.name == df2.name, 'outer').select(df.name, df2.height).collect()

[Row(name=None, height=80), Row(name=u'Alice', height=None), Row(name=u'Bob', height=85)]

>>> cond = [df.name == df3.name, df.age == df3.age]

>>> df.join(df3, cond, 'outer').select(df.name, df3.age).collect()

[Row(name=u'Bob', age=5), Row(name=u'Alice', age=2)]

So you need to use the "condition as a list" option like in the last example.