spark_df = spark.createDataFrame(df)
pandas df转换为spark df时报错
Can not merge type <class 'pyspark.sql.types.LongType'> and <class 'pyspark.sql.types.StringType'>
原因:df中有空值,去掉空值,强转类型 即可
df['item_id'].astype(int)
df['item_geohash'].astype(str)
df['item_category'].astype(str)
df =df.replace(np.NaN, '')