pyspark.sql.DataFrame与pandas.DataFrame之间的相互转换

pyspark.sql.DataFrame与pandas.DataFrame之间的相互转换

代码示例：

# -*- coding: utf-8 -*-
import pandas as pd
from pyspark.sql import SparkSession
from pyspark.sql import SQLContext
from pyspark import SparkContext

# 配置spark 运行参数
import os
os.environ["SPARK_HOME"] = "/Users/a6/Applications/spark-2.1.0-bin-hadoop2.6"
# 初始化spark DataFrame
sc = SparkContext()
if __name__ == "__main__":
    print "1、初始化pandas DataFrame"
    # 初始化pandas DataFrame
    df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], index=['row1', 'row2'], columns=['c1', 'c2', 'c3'])

    # 打印数据
    print df

    spark = SparkSession\
        .builder\
        .appName("testDataFrame")\
        .getOrCreate()

    sentenceData = spark.createDataFrame([(0.0, "I like Spark"),
                                          (1.0, "Pandas is useful"),
                                          (2.0, "They are coded by Python ")],
                                         ["label", "sentence"])
    # 显示数据
    sentenceData.select("label").show()

    print "2、将pandas.DataFrame 转换成 spark.DataFrame"
    # spark.DataFrame 转换成 pandas.DataFrame
    sqlContext = SQLContext(sc)
    spark_df = sqlContext.createDataFrame(df)

    # 显示数据
    spark_df.select("c1").show()

    print "3、将spark.DataFrame 转换成 pandas.DataFrame"
    # pandas.DataFrame 转换成 spark.DataFrame
    pandas_df = sentenceData.toPandas()

    # 打印数据
    print pandas_df

运行结果如下：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/05/21 19:47:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/05/21 19:47:22 WARN Utils: Your hostname, localhost resolves to a loopback address: 127.0.0.1; using 10.2.33.229 instead (on interface en0)
18/05/21 19:47:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/05/21 19:47:22 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
1、初始化pandas DataFrame
      c1  c2  c3
row1   1   2   3
row2   4   5   6
+-----+
|label|
+-----+
|  0.0|
|  1.0|
|  2.0|
+-----+

2、将pandas.DataFrame 转换成  spark.DataFrame
+---+
| c1|
+---+
|  1|
|  4|
+---+

3、将spark.DataFrame 转换成  pandas.DataFrame
   label                   sentence
0    0.0               I like Spark
1    1.0           Pandas is useful
2    2.0  They are coded by Python 

Process finished with exit code 0

参考：https://blog.csdn.net/zhurui_idea/article/details/72981715