sparksql_删除行_去除行

#income列缺失太多，基本无用了，现在要去掉这一列数据
#thresh=3 表示一行中非NONE的数据少于3个则去除该行

#income列缺失太多，基本无用了，现在要去掉这一列数据
df_miss_no_income = df_miss.select([c for c in df_miss.columns if c != 'income'])
df_miss_no_income.show()
+---+------+------+----+------+
| id|weight|height| age|gender|
+---+------+------+----+------+
|  1| 143.5|   5.6|  28|     M|
|  2| 167.2|   5.4|  45|     M|
|  3|  null|   5.2|null|  null|
|  4| 144.5|   5.9|  33|     M|
|  5| 133.2|   5.7|  54|     F|
|  6| 124.1|   5.2|null|     F|
|  7| 129.2|   5.3|  42|     M|
+---+------+------+----+------+

To drop the observations instead you can use the .dropna(...) method.

#某些行缺失的数据也比较多，现在去除掉这些行
#thresh=3 表示一行中非NONE的数据少于3个则去除该行
df_miss_no_income.dropna(thresh=3).show()
​
#只要含有NONE则去除该行
df_miss_no_income.dropna().show()
+---+------+------+----+------+
| id|weight|height| age|gender|
+---+------+------+----+------+
|  1| 143.5|   5.6|  28|     M|
|  2| 167.2|   5.4|  45|     M|
|  4| 144.5|   5.9|  33|     M|
|  5| 133.2|   5.7|  54|     F|
|  6| 124.1|   5.2|null|     F|
|  7| 129.2|   5.3|  42|     M|
+---+------+------+----+------+