python dataframe drop null值_在DataFrame中使用None / null值替换空字符串

I have a Spark 1.5.0 DataFrame with a mix of null and empty strings in the same column. I want to convert all empty strings in all columns to null (None, in Python). The DataFrame may have hundreds of columns, so I'm trying to avoid hard-coded manipulations of each column.

See my attempt below, which results in an error.

from pyspark.sql import SQLContext

sqlContext = SQLContext(sc)

## Create a test DataFrame

testDF = sqlContext.createDataFrame([Row(col1='foo', col2=1), Row(col1='', col2=2), Row(col1=None, col2='')])

testDF.show()

## +----+----+

## |col1|col2|

## +----+----+

## | foo| 1|

## | | 2|

## |null|null|

## +----+----+

## Try to replace an empty string with None/null

test