python同名函数相互冲突_Python内置函数与spark冲突[duplicate]

以下是一些玩具数据的处理方法:

spark.version

# u'2.2.0'

import pyspark.sql.functions as func

df = spark.createDataFrame(

[(0.0, 0.2, 3.45631),

(0.4, 1.4, 2.82945),

(0.5, 1.9, 7.76261),

(0.6, 0.9, 2.76790),

(1.2, 1.0, 9.87984)],

["col1", "col2", "col3"])

df.show()

# +----+----+-------+

# |col1|col2| col3|

# +----+----+-------+

# | 0.0| 0.2|3.45631|

# | 0.4| 1.4|2.82945|

# | 0.5| 1.9|7.76261|

# | 0.6| 0.9| 2.7679|

# | 1.2| 1.0|9.87984|

# +----+----+-------+

# round 'col3' in a new column:

df2 = df.withColumn("col4", func.round(df["col3"], 2)).withColumnRenamed("col4","new_col3")

df2.show()

# +----+----+-------+--------+

# |col1|col2| col3|new_col3|

# +----+----+-------+--------+

# | 0.0| 0.2|3.45631| 3.46|

# | 0.4| 1.4|2.82945| 2.83|

# | 0.5| 1.9|7.76261| 7.76|

# | 0.6| 0.9| 2.7679| 2.77|

# | 1.2| 1.0|9.87984| 9.88|

# +----+----+-------+--------+

# round & replace existing 'col3':

df3 = df.withColumn("col3", func.round(df["col3"], 2))

df3.show()

# +----+----+----+

# |col1|col2|col3|

# +----+----+----+

# | 0.0| 0.2|3.46|

# | 0.4| 1.4|2.83|

# | 0.5| 1.9|7.76|

# | 0.6| 0.9|2.77|

# | 1.2| 1.0|9.88|

# +----+----+----+

这是个人的爱好,但我也不太喜欢

col

也不是

alias

-我更喜欢

withColumn

和

withColumnRenamed

相反。不过,如果你愿意

select

和

col列

,下面是您应该如何调整自己的代码片段:

from pyspark.sql.functions import col

df4 = df.select(col("col1").alias("new_col1"),

col("col2").alias("new_col2"),

func.round(df["col3"],2).alias("new_col3"))

df4.show()

# +--------+--------+--------+

# |new_col1|new_col2|new_col3|

# +--------+--------+--------+

# | 0.0| 0.2| 3.46|

# | 0.4| 1.4| 2.83|

# | 0.5| 1.9| 7.76|

# | 0.6| 0.9| 2.77|

# | 1.2| 1.0| 9.88|

# +--------+--------+--------+