Spark_DataFrame vs SQL

Spark DataFrame vs SQL 的小练习

a.初始化Spark Session

import	findspark
findspark.init()
from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

b.构建数据集与序列化

stringJSONRDD = spark.sparkContext.parallelize((""" 
  { "id": "123",
    "name": "Katie",
    "age": 19,
    "eyeColor": "brown"
  }""",
   """{
    "id": "234",
    "name": "Michael",
    "age": 22,
    "eyeColor": "green"
  }""", 
  """{
    "id": "345",
    "name": "Simone",
    "age": 23,
    "eyeColor": "blue"
  }""")
)

# 构建DataFrame
swimmersJSON = spark.read.json(stringJSONRDD)

# 创建临时表
swimmersJSON.createOrReplaceTempView("swimmersJSON")
# DataFrame信息
swimmersJSON.show()

spark.sql("select * from swimmersJSON").show()

# 执行SQL请求
spark.sql("select * from swimmersJSON").collect()

# 输出数据表的格式
swimmersJSON.printSchema()
# 执行SQL
spark.sql("select count(1) from swimmersJSON")

spark.sql("select count(1) from swimmersJSON").show()

c.DataFrame的请求方式 vs SQL的写法

# DataFrame的写法
swimmersJSON.select("id", "age").filter("age = 22").show()

# SQL的写法
spark.sql("select id, age from swimmersJSON where age = 22").show()
# DataFrame的写法
swimmersJSON.select("name", "eyeColor").filter("eyeColor like 'b%'").show()
# SQL的写法
spark.sql("select name, eyeColor from swimmersJSON where eyeColor like 'b%'").