pyspark.sql.utils.IllegalArgumentException: u'requirement failed'

19/01/10 17:18:35 WARN KMeans: The input data is not directly cached, which may hurt performance if its parent RDDs are also uncached.
Traceback (most recent call last):
File “/home/work/XXX/XXX/XXX/XXX/clu/lib/k_means_stratification_v3_23_01_bei.py”, line 356, in
run()
File “/home/work/XXX/XXX/XXX/XXX/clu/lib/k_means_stratification_v3_23_01_bei.py”, line 351, in run
clustering(sc_context=spark_context, df_filtered=df_nonDirectBusiness, temp_path=temp_path + ‘nonDirectBusiness/’, columns=cols,options=options,property_dict=property_dict)
File “/home/work/XXX/XXX/XXX/XXX/clu/lib/k_means_stratification_v3_23_01_bei.py”, line 240, in clustering
temp_k = get_elbow(temp_vc2, temp_min_k, temp_max_k, 3)
File “/home/work/XXX/XXX/XXX/XXX/clu/lib/k_means_stratification_v3_23_01_bei.py”, line 40, in get_elbow
model = kmeans_def.fit(rdd_data)
File “/home/work/local/spark-2.1.1/python/lib/pyspark.zip/pyspark/ml/base.py”, line 64, in fit
File “/home/work/local/spark-2.1.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py”, line 236, in _fit
File “/home/work/local/spark-2.1.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py”, line 233, in _fit_java
File “/home/work/local/spark-2.1.1/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py”, line 1133, in call
File “/home/work/local/spark-2.1.1/python/lib/pyspark.zip/pyspark/sql/utils.py”, line 79, in deco
pyspark.sql.utils.IllegalArgumentException: u’requirement failed

出现这种问题是因为传进去的rdd_data中含有nan，找出数据中的nan并进行修改即可