pyspark dataframe 字段类型转换 pandas和pyspark的dataframe互转

知识点：

1、使用pyspark读取csv：spark.read.format("csv").load('/user/data.csv',header=True, inferSchema="true")

2、dataframe补充空值：fillna()

3、dataframe字段表示方式："APP_HOBY_CASH_LOAN"或df.APP_HOBY_CASH_LOAN或data_df["APP_HOBY_CASH_LOAN"]

 

pyspark dataframe使用astype实现dataframe字段类型转换

1. # 两种读取csv文件的方式

2. data_df = spark.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load("/user/data.csv")

3. data_df = spark.read.format("csv").load('/user/data.csv',header=True, inferSchema="true")

4. # 补充空值方法

5. data_df = data_df.fillna(0)

6. # 打印表结构 第一种执行较快

7. print(df_from_java.columns)

8. data_df.printSchema()

9. print(df_from_java.schema())

10. # 第一种方式

11. data_df = df.withColumn("APP_HOBY_CASH_LOAN", df.APP_HOBY_CASH_LOAN.cast('float'))

12. # 第二种方式

13. from pyspark.sql.types import IntegerType

14. data_df = data_df.withColumn("APP_HOBY_CASH_LOAN", data_df["APP_HOBY_CASH_LOAN"].cast(IntegerType()))

pandas dataframe使用astype实现dataframe字段类型转换

1. # -*- coding: UTF-8 -*-

2. import pandas as pd

3. df = pd.DataFrame([{'col1':'a', 'col2':'1'}, {'col1':'b', 'col2':'2'}])

4.  

5. print df.dtypes

6.  

7. df['col2'] = df['col2'].astype('int')

8. print '-----------'

9. print df.dtypes

10.  

11. df['col2'] = df['col2'].astype('float64')

12. print '-----------'

13. print df.dtypes

pandas和pyspark的dataframe互转

1. #!/usr/bin/python

2. # -*- coding: utf-8 -*-

3. from pyspark import SparkConf

4. from pyspark.sql import SparkSession

5. import pandas as pd

6.  
7.  

8. # pyspark dataframe转pandas dataframe

9. conf = SparkConf()

10. conf.set("spark.yarn.queue", "root") \

11. .set("spark.app.name", "lbs_coordinate")

12. spark = SparkSession.builder.config(conf=conf).enableHiveSupport().getOrCreate()

13. pyspark_df = spark.sql("select * from tmp.test_table")

14. pandas_pd=pyspark_df.toPandas()

15.  
16.  

17. # pandas dataframe转pyspark dataframe

18. pandas_df = pd.DataFrame([{'col1':'a', 'col2':'1'}, {'col1':'b', 'col2':'2'}])

19. pyspark_df = spark.createDataFrame(pandas_df)