pyspark dataframe基本操作

1 创建dataframe

1.1 读取文件创建

from pyspark.sql import SparkSession #sparkSession为同统一入口

#创建spakr对象
spark = SparkSession\
    .builder\
    .appName('readfile')\
    .getOrCreate()

# 1.读取csv文件
# 1.读取csv文件
logFilePath = 'births_train.csv'
log_df = spark.read.csv(logFilePath, 
                        encoding='utf-8', 
                        header=True, 
                        inferSchema=True,
                        sep=',')

logFilePath:这是我自定义的一个参数,为文件路径
encoding:文件编码格式,默认为utf-8
header:是否将文件第一行作为表头,True即将文件第一行作为表头
inferSchema:是否自动推断列类型
sep:列分割符

log_df.show()

展示结果如下图

1.2 手动创建

这种方式一般为测试的时候用,适用于数据量很小的时候

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FirstApp").getOrCreate()

employees = [(1, "John", 25), (2, "Ray", 35), (3,"Mike", 24), (4, "Jane", 28), (5, "Kevin", 26), 
             (6, "Vincent", 35), (7,"James", 38), (8, "Shane", 32), (9, "Larry", 29), (10, "Kimberly", 29),
             (11, "Alex", 28), (12, "Garry", 25), (13, "Max",31)]
employees=spark.createDataFrame(employees, schema=["emp_id","name","age"])

这里创建了三列
employees为数据内容,schema为表头,这种方式比较简单,类型为spark推断类型

可能有的同学会见到如下表头的创建方式,类型可以自己指定

from pyspark.sql import SparkSession #sparkSession为同统一入口
from pyspark.sql.types import *

#创建spakr对象
spark = SparkSession\
    .builder\
    .appName('readfile')\
    .getOrCreate()

employees = [(1, "John", 25), (2, "Ray", 35), (3,"Mike", 24), (4, "Jane", 28), (5, "Kevin", 26), 
             (6, "Vincent", 35), (7,"James", 38), (8, "Shane", 32), (9, "Larry", 29), (10, "Kimberly", 29),
             (11, "Alex", 28), (12, "Garry", 25), (13, "Max",31)]
schema = StructType([StructField('emp_id',IntegerType(),True),
                    StructField('name',StringType(),True),
                    StructField('age',IntegerType(),True)])
df = spark.createDataFrame(employees,schema=schema)

StructType:即指定一个列类型的对象,里面包含列类型数组
Str