dataframe删除全部为空列_如何在PySpark DataFrame中删除具有空值的所有列？

I have a large dataset of which I would like to drop columns that contain null values and return a new dataframe. How can I do that?

The following only drops a single column or rows containing null.

df.where(col("dt_mvmt").isNull()) #doesnt work because I do not have all the columns names or for 1000's of columns

df.filter(df.dt_mvmt.isNotNull()) #same reason as above

df.na.drop() #drops rows that contain null, instead of columns that contain null

For example

a | b | c

1 | | 0

2 | 2 | 3

In the above case it will drop the whole column B because one of its values is empty.

解决方案

Here is one possible approach for dropping all columns that have NULL values: See here for the source on the code of counting NULL values per column.

import pyspark.sql.functions as F

# Sample data