pyspark学习_RDD转为DataFrame

heiqizero

已于 2024-01-25 23:08:12 修改

阅读量238

点赞数 3

分类专栏： spark 文章标签： spark python

于 2024-01-25 23:07:45 首次发布

本文链接：https://blog.csdn.net/qq_37239381/article/details/135855897

版权

spark 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

#方法1：反射机制推断RDD模式
people.txt 
Tom 12
Jack 13
Janny 14
from pyspark.sql import SparkSession,Row 

spark = SparkSession.builder.getOrCreate()
lines = spark.sparkContext.textFile("people.txt")
people = lines.map(lambda x:x.split(",")).map(lambda x:Row(name=x[0],age=int(x[1])))
peopleDF = spark.createDataFrame(people) #RDD注册为DataFrame
peopleDF.createOrReplaceTempView("people") #DataFrame注册people表
result = spark.sql("select * from peopleDF")
result.rdd.map(lambda x:'name:'+x['name']+' '+'age:'+str(x['age'])).foreach(print)
#方法2：编程方式定义RDD模式
from pyspark.sql import SparkSession,Row
from pyspark.sql.types import StructType,StructField,StringType,IntegerType

spark = SparkSession.builder.getOrCreate()
#创建schema
schema = StructType([StructField('name',StringType(),True),StructField('age',IntegerType(),True)])
#创建content
content = spark.sparkContext.textFile("people.txt").map(lambda x:x.split(" ")).map(lambda x:Row(x[0],x[1]))
peopleDF = spark.createDataFrame(content,schema)
peopleDF.createOrReplaceTempView("people")
result = spark.sql("select * from peopleDF")
result.rdd.map(lambda x:'name:'+x['name']+' '+'age:'+str(x['age'])).foreach(print)
#方法3：toDF()方法
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
peopleDF = spark.sparkContext.textFile("people.txt").map(lambda x:x.split(",")).toDF(schema=['name','age'])
peopleDF.show()