Spark SQL 操作实战

最新推荐文章于 2024-06-17 15:42:55 发布
最新推荐文章于 2024-06-17 15:42:55 发布
阅读量371 收藏 11
点赞数 5
分类专栏： 大数据 Python 文章标签： spark sql
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/lyzsun8295/article/details/139096710
版权

Spark SQL 基础

Spark SQL 是 Apache Spark 处理结构化数据的模块。

配置spark环境

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q www-us.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz  
!tar xf spark-2.4.8-bin-hadoop2.7.tgz
!pip install -q findspark

import os
os.environ["JAVA_HOME"]="/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"]="/content/spark-2.4.8-bin-hadoop2.7"

import findspark
findspark.init()

初始化SparkSession

初始化

 SparkSession 用于创建数据帧，将数据帧注册为表，执行 SQL 查询，缓存表及读取 Parquet 文件。

from pyspark.sql import SparkSession

spark = SparkSession \
       .builder \
       .appName("Python Spark SQL basic example") \
       .config("spark.some.config.option", "some-value") \
       .getOrCreate()

创建数据帧

从 RDD 创建

from pyspark.sql.types import *
from pyspark.sql import *
from pyspark.sql.functions import *

推断 Schema

sc = spark.sparkContext

!wget https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/people.txt

lines = sc.textFile("people.txt")

parts = lines.map(lambda l: l.split(","))

people = parts.map(lambda p: Row(name=p[0],age=int(p[1])))

peopledf = spark.createDataFrame(people)

指定 Schema

people = parts.map(lambda p: Row(name=p[0], age=int(p[1].strip())))

schemaString = "name age"

fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]

schema = StructType(fields)

spark.createDataFrame(people, schema).show()

+-------+---+
|   name|age|
+-------+---+
|Michael| 29|
|   Andy| 30|
| Justin| 19|
+-------+---+

从Spark数据源创建

JSON

!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/people.json
!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/employees.json
!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/customers.json

df = spark.read.json("customers.json")

df.show()

+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+

df2 = spark.read.load("people.json", format="json")

Parquet文件

!wget -q https://github.com/apache/spark/raw/master/examples/src/main/resources/users.parquet

df3 = spark.read.load("users.parquet")

文本文件

df4 = spark.read.text("people.txt")

查阅数据信息

查阅spark Dataframe的信息

df.dtypes   #返回 df 的列名与数据类型

[('address', 'struct<city:string,state:string,street:string,zip:string>'),
 ('age', 'bigint'),
 ('firstName', 'string'),
 ('lastName', 'string'),
 ('phoneNumber', 'struct<number:string,type:string>')]

df.show()   #显示 df 的内容

+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+

df.head(3)   #返回前 n 行数据

[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Brighton', state='MI', street='4 B Blue Ridge Blvd', zip='48116'), age=25, firstName='Josephine', lastName='Darakjy', phoneNumber=Row(number='210-319-103', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]

df.first()   #返回第 1 行数据

Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile'))

df.take(3)   #返回前 n 行数据

[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Brighton', state='MI', street='4 B Blue Ridge Blvd', zip='48116'), age=25, firstName='Josephine', lastName='Darakjy', phoneNumber=Row(number='210-319-103', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]

df.schema   #返回 df 的 Schema

StructType(List(StructField(address,StructType(List(StructField(city,StringType,true),StructField(state,StringType,true),StructField(street,StringType,true),StructField(zip,StringType,true))),true),StructField(age,LongType,true),StructField(firstName,StringType,true),StructField(lastName,StringType,true),StructField(phoneNumber,StructType(List(StructField(number,StringType,true),StructField(type,StringType,true))),true)))

df.describe().show()   #汇总统计数据

+-------+-----------------+---------+----------+
|summary|              age|firstName|  lastName|
+-------+-----------------+---------+----------+
|  count|                4|        4|         4|
|   mean|             28.5|     null|      null|
| stddev|6.027713773341708|     null|      null|
|    min|               22|    Boris|Butterburg|
|    max|               35|Josephine|     Smith|
+-------+-----------------+---------+----------+

df.columns   #返回 df 的列名

['address', 'age', 'firstName', 'lastName', 'phoneNumber']

df.count()   #返回 df 的行数

4

df.distinct().count()   #返回 df 中不重复的行数

4

df.printSchema()   #返回 df的 Schema

root
 |-- address: struct (nullable = true)
 |    |-- city: string (nullable = true)
 |    |-- state: string (nullable = true)
 |    |-- street: string (nullable = true)
 |    |-- zip: string (nullable = true)
 |-- age: long (nullable = true)
 |-- firstName: string (nullable = true)
 |-- lastName: string (nullable = true)
 |-- phoneNumber: struct (nullable = true)
 |    |-- number: string (nullable = true)
 |    |-- type: string (nullable = true)

df.explain()   #返回逻辑与实体方案

== Physical Plan ==
*(1) FileScan json [address#452,age#453L,firstName#454,lastName#455,phoneNumber#456] Batched: false, Format: JSON, Location: InMemoryFileIndex[file:/content/customers.json], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<address:struct<city:string,state:string,street:string,zip:string>,age:bigint,firstName:str...

重复值

dropDuplicates函数

df = df.dropDuplicates()

查询

自定义函数F

from pyspark.sql import functions as F

Select

df.select("firstName").show()   #显示 firstName 列的所有条目

+---------+
|firstName|
+---------+
|Josephine|
|     Jane|
|    Boris|
|  Johnson|
+---------+

df.select("firstName","lastName").show()

+---------+----------+
|firstName|  lastName|
+---------+----------+
|Josephine|   Darakjy|
|     Jane|Butterburg|
|    Boris|    Chemel|
|  Johnson|     Smith|
+---------+----------+

df.show()

+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+

df.select("firstName", "age", df.phoneNumber \
            .alias("contactInfo")) \
            .select("contactInfo", "firstName", "age") \
            .show()   #显示 firstName、age 的所有条目和类型

+--------------------+---------+---+
|         contactInfo|firstName|age|
+--------------------+---------+---+
|[210-319-103, mob...|Josephine| 25|
|[210-332-578, mob...|     Jane| 22|
|[210-322-007, mob...|    Boris| 32|
|[210-303-029, mob...|  Johnson| 35|
+--------------------+---------+---+

df.select(df["firstName"],df["age"]+ 1).show()   # 显示 firstName 和 age 列的所有 记录，并对 age 记录添加1

+---------+---------+
|firstName|(age + 1)|
+---------+---------+
|Josephine|       26|
|     Jane|       23|
|    Boris|       33|
|  Johnson|       36|
+---------+---------+

df.select(df['age'] > 24).show()   #显示所有小于24岁的记录

+----------+
|(age > 24)|
+----------+
|      true|
|     false|
|      true|
|      true|
+----------+

When

df.select("firstName", F.when(df.age > 30, 1) \
            .otherwise(0)) \
            .show()   #显示 firstName，且大于30岁显示 1，小于30岁显示0

+---------+--------------------------------------+
|firstName|CASE WHEN (age > 30) THEN 1 ELSE 0 END|
+---------+--------------------------------------+
|Josephine|                                     0|
|     Jane|                                     0|
|    Boris|                                     1|
|  Johnson|                                     1|
+---------+--------------------------------------+

df[df.firstName.isin("Jane","Boris")].collect()   # 显示符合指定条件的 firstName 列 的记录

[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]

Like

df.select("firstName", df.lastName.like("Smith")) \
  .show()   # 显示 lastName 列中包含 Smith 的 firstName 列的记录

+---------+-------------------+
|firstName|lastName LIKE Smith|
+---------+-------------------+
|Josephine|              false|
|     Jane|              false|
|    Boris|              false|
|  Johnson|               true|
+---------+-------------------+

Startswith - Endswith

df.select("firstName", df.lastName \
  .startswith("Sm")) \
  .show()   # 显示 lastName 列中以 Sm 开头的 firstName 列的记录

+---------+------------------------+
|firstName|startswith(lastName, Sm)|
+---------+------------------------+
|Josephine|                   false|
|     Jane|                   false|
|    Boris|                   false|
|  Johnson|                    true|
+---------+------------------------+

df.select(df.lastName \
  .endswith("th")) \
  .show()   # 显示以 th 结尾的 lastName

+----------------------+
|endswith(lastName, th)|
+----------------------+
|                 false|
|                 false|
|                 false|
|                  true|
+----------------------+

Substring

df.select(df.firstName.substr(1, 3) \
 .alias("name")) \
 .collect()   #返回 firstName 的子字符串

[Row(name='Jos'), Row(name='Jan'), Row(name='Bor'), Row(name='Joh')]

Between

df.select(df.age.between(22, 24)) \
        .show()   #显示介于22岁至24岁之间的 age 列的记录

+-----------------------------+
|((age >= 22) AND (age <= 24))|
+-----------------------------+
|                        false|
|                         true|
|                        false|
|                        false|
+-----------------------------+

添加、修改、删除列

添加列

df.select(df.phoneNumber.number).show()

+------------------+
|phoneNumber.number|
+------------------+
|       210-319-103|
|       210-332-578|
|       210-322-007|
|       210-303-029|
+------------------+

df = df.withColumn('city',df.address.city) \
             .withColumn('postalCode',df.address.zip) \
             .withColumn('state',df.address.state) \
             .withColumn('streetAddress',df.address.street) \
             .withColumn('telePhoneNumber', df.phoneNumber.number) \
             .withColumn('telePhoneType', df.phoneNumber.type)

df.show()

+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+
|             address|age|firstName|  lastName|         phoneNumber|       city|postalCode|state|       streetAddress|telePhoneNumber|telePhoneType|
+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|    210-319-103|       mobile|
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|New Orleans|     70116|   LA|  6649 N Blue Gum St|    210-332-578|       mobile|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|    210-322-007|       mobile|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...| Bridgeport|     08112|   NJ|     5 W Blue Ave St|    210-303-029|       mobile|
+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+

修改列

df = df.withColumnRenamed('telePhoneNumber', 'phoneNumber')

删除列

df = df.drop("address", "phoneNumber")
# 等价于
# df = df.drop(df.address).drop(df.phoneNumber)

分组

groupBy操作

df.groupBy("age")\
      .count() \
      .show()   #按 age 列分组，统计每组人数

+---+-----+
|age|count|
+---+-----+
| 22|    1|
| 32|    1|
| 25|    1|
| 35|    1|
+---+-----+

筛选

filter筛选

df.filter(df["age"]>24).show()   #按 age 列筛选，保留年龄大于24岁的

+---+---------+--------+----------+----------+-----+--------------------+-------------+
|age|firstName|lastName|      city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+--------+----------+----------+-----+--------------------+-------------+
| 25|Josephine| Darakjy|  Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 32|    Boris|  Chemel|Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|   Smith|Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+--------+----------+----------+-----+--------------------+-------------+

排序

sort与orderBy操作

peopledf.sort(peopledf.age.desc()).collect()

[Row(age=30, name='Andy'),
 Row(age=29, name='Michael'),
 Row(age=19, name='Justin')]

df.sort("age", ascending=False).collect()

[Row(age=35, firstName='Johnson', lastName='Smith', city='Bridgeport', postalCode='08112', state='NJ', streetAddress='5 W Blue Ave St', telePhoneType='mobile'),
 Row(age=32, firstName='Boris', lastName='Chemel', city='Bridgeport', postalCode='08014', state='NJ', streetAddress='8 W Cerritos Ave #54', telePhoneType='mobile'),
 Row(age=25, firstName='Josephine', lastName='Darakjy', city='Brighton', postalCode='48116', state='MI', streetAddress='4 B Blue Ridge Blvd', telePhoneType='mobile'),
 Row(age=22, firstName='Jane', lastName='Butterburg', city='New Orleans', postalCode='70116', state='LA', streetAddress='6649 N Blue Gum St', telePhoneType='mobile')]

df.orderBy(["age","city"],ascending=[0,1]).collect()

[Row(age=35, firstName='Johnson', lastName='Smith', city='Bridgeport', postalCode='08112', state='NJ', streetAddress='5 W Blue Ave St', telePhoneType='mobile'),
 Row(age=32, firstName='Boris', lastName='Chemel', city='Bridgeport', postalCode='08014', state='NJ', streetAddress='8 W Cerritos Ave #54', telePhoneType='mobile'),
 Row(age=25, firstName='Josephine', lastName='Darakjy', city='Brighton', postalCode='48116', state='MI', streetAddress='4 B Blue Ridge Blvd', telePhoneType='mobile'),
 Row(age=22, firstName='Jane', lastName='Butterburg', city='New Orleans', postalCode='70116', state='LA', streetAddress='6649 N Blue Gum St', telePhoneType='mobile')]

替换缺失值

replace操作

df.na.fill(50).show()   #用一个值替换空值

+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+

df.na.drop().show()   #去除 df 中为空值的行

+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+

df.na.replace(10, 20).show()

+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+

重分区

repartition重分区

df.repartition(10)\
      .rdd \
      .getNumPartitions()   #将 df 拆分为10个分区

10

df.coalesce(1).rdd.getNumPartitions()   #将 df 合并为1个分区

1

运行 SQL 查询

将数据帧注册为视图

peopledf.createGlobalTempView("people")

df.createTempView("customer")

df.createOrReplaceTempView("customer")

查询视图

df5 = spark.sql("SELECT * FROM customer").show()

+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+

peopledf2 = spark.sql("SELECT * FROM global_temp.people").show()

+---+-------+
|age|   name|
+---+-------+
| 29|Michael|
| 30|   Andy|
| 19| Justin|
+---+-------+

输出

数据结构

rdd1 = df.rdd   #将 df 转换为 RDD

df.toJSON().first()   #将 df 转换为 RDD 字符串

'{"age":25,"firstName":"Josephine","lastName":"Darakjy","city":"Brighton","postalCode":"48116","state":"MI","streetAddress":"4 B Blue Ridge Blvd","telePhoneType":"mobile"}'

df.toPandas()   #将 df 的内容转为 Pandas 的数据帧
agefirstNamelastNamecitypostalCodestatestreetAddresstelePhoneType
025JosephineDarakjyBrighton48116MI4 B Blue Ridge Blvdmobile
122JaneButterburgNew Orleans70116LA6649 N Blue Gum Stmobile
232BorisChemelBridgeport08014NJ8 W Cerritos Ave #54mobile
335JohnsonSmithBridgeport08112NJ5 W Blue Ave Stmobile

保存至文件

df.select("firstName", "city")\
      .write \
      .save("nameAndCity.parquet")

df.select("firstName", "age") \
      .write \
      .save("namesAndAges.json",format="json")

终止SparkSession

终止spark session

spark.stop()
一缕阳光lyz
关注
  • 5
    点赞
  • 踩
  • 11
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Spark sql 实战案例
TU_JCN的博客
07-11 911
1、sparksql 操作hivesql 添加依赖 <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.11</artifactId> <version>2.3.3</version> </dependency>
Spark 入门实战系列
01-18
Spark 入门实战系列，适合初学者，文档包括十部分内容，质量很好，为了感谢文档作者，也为了帮助更多的人入门，传播作者的心血，特此友情转贴: 1.Spark及其生态圈简介.pdf 2.Spark编译与部署（上）--基础环境搭建....
spark—SQL实战案例
weixin_44604159的博客
10-05 3461
对sparkSQL的学习进行案例练习
Java-Spark系列6-Spark SQL编程实战
热门推荐
只是甲的博客
09-26 2万+
文章目录一.Spark DataFrame概述1.1 创建DataFrame1.1.1 通过json文件创建DataFrame1.1.2 通过CSV文件创建DataFrame1.1.3 通过hive table创建DataFrame1.1.4 通过jdbc数据源创建DataFrame二.Spark SQL实战2.1 DataFrame的统计信息2.2 DataFrame的select操作2.3 DataFrame对列的操作2.3 过滤数据2.4 简单的聚合操作2.4.1 简单聚合2.5 自定义函数2.6 表
Spark SQL调优实战
u013984781的博客
02-05 1592
spark sql参数调优
Spark SQL详细知识点
qq_45831157的博客
07-27 1015
Spark SQL是Spark框架中用于结构化数据(structured data)处理的Spark模块。SparkSQL知识点超详细文档！！！可以让基础小白快速入门。有丰富的小案例，边学边练！！
Spark SQL操作Hive实战
叶小刀
10-24 2659
在目前企业级 Spark大数据开发中,大多数情况下都是采用Hive来作为数据仓库的。Spark提供了对Hive的支持, Spark通过HiveContext可以直接操作Hive中的数据。基于HiveContext,、我们可以使用sql/hql两种方式来编写SQL语句 对Hive进行操作,包括:创建表、删除表、往表中导入数据、以及对表中的数据进行CRUD(増、删、改、査)操作。 下面就开始动...
pyspark系列6-Spark SQL编程实战
只是甲的博客
05-06 5616
一.Spark DataFrame概述 从上一篇博客，我们可以知道因为Python是弱类型，所以PySpark SQL的数据抽象就只有DataFrame,这里我们再来复习一下DataFrame。 在Spark语义中，DtatFrame是一个分布式的行集合，可以想象为一个关系型数据库的表，或一个带有列头的Excel表格。它和RDD一样，有这样一些特点: Immuatable: 一旦RDD、DataFrame被创建，就不能更改，只能通过tranformation生成新的RDD、DataFrame Lazy
spark sql实战—拆分数据
zg_hover的专栏
05-23 1万+
拆分的数据 有时在进行数据时我们需要把一列数据分割成多列数据，把一个字段值，分割成多个值。本节介绍如何通过spark sql提供的函数来进行数据的分割。 1. 数据拆分概述 数据拆分操作 在进行数据处理时，通常我们需要对数据进行拆分。比如：把一列拆分成多行，多列，把一行拆分成多行，多列等。 在spark-sql中提供了多个函数用来进行数据拆分。 数据拆分的函数 split explode postexplode substring 2. 数据的拆分 2.1 通过explode系列函数进行拆分 把一个数
Spark Sql实战--合并数据
zg_hover的专栏
05-23 7521
数据的合并 概述 本文介绍如何通过spark sql对数据进行各种的合并操作，包括：列合并，行合并，相同key的合并等等。 在实际的数据处理场景中，数据的合并操作非常常用，这里介绍如何通过spark sql来完成常用的合并操作。 数据准备 例子数据 准备以下数据： name,address,age,id,time david,shenzhen,31,1,201903 eason,shenzhen,27,2,201904 jarry,wuhan,35,3,201904 aarry2,wuhan1,34,4
Spark SQL新特性
wangleigiser的博客
06-27 599
Spark SQL内核调优
spark入门及实战文档
04-09
包含spark快速入门、spark-sql介绍和应用、spark-streaming介绍和应用、spark项目实战等
Spark机器学习实战介绍
04-25
4. Spark MLlib 库提供了一个灵活的 API，能够与其他 Spark 库集成，例如 Spark SQL、Spark Streaming 等。 Spark MLlib 库的组件包括： 1. Spark MLlib 的 Core 组件，提供了基本的机器学习算法和工具。 2. Spark...
Spark SQL数据加载和保存实例讲解
12-16
Spark SQL重要是操作DataFrame，DataFrame本身提供了save和load的操作， Load：可以创建DataFrame， Save：把DataFrame中的数据保存到文件或者说与具体的格式来指明我们要读取的文件的类型以及与具体的格式来指出...
大数据Spark企业级实战
05-19
《大数据Spark企业级实战》详细解析了企业级Spark开发所需的几乎所有技术内容，涵盖Spark的架构设计、Spark的集群搭建、Spark内核的解析、Spark SQL、MLLib、GraphX、Spark Streaming、Tachyon、SparkR、Spark多语言...
【大数据】Spark使用大全:下载安装、RDD操作、JAVA编程、SQL
Joker_ZJN的博客
06-13 1009
一文详解Spark的使用
【kyuubi k8s】kyuubi发布k8s执行spark sql
最新发布
你的博客
06-17 340
准备好kyuubi的包，spark包，jdk包，还有上篇中重点提到的aws包：awsicelib目录下的jar，还有一点将k8s的config也要打包进来，否则无法通过k8s的认证，没法提交任务。依据上一篇kyuubi与spark集成，并发布spark sql到k8s集群，上一篇的将kyuubi和spark环境放在本地某台服务器上的，为了高可用，本篇将其打包镜像，并发布到k8s。我的busycommand，10s钟检测一次，如果进程没了，重新启动kyuubi。k8s中生成2个executors。
windows安装spark
weixin_41046245的博客
06-17 236
在 Windows 上安装 Spark 并进行配置需要一些步骤，包括安装必要的软件和配置环境变量。
教案：在 Spark 上使用 Horovod 进行分布式训练
weixin_41046245的博客
06-17 167
Horovod 简介分布式深度学习的概念Horovod 的作用与优势Spark 简介Spark 的基本概念Spark 集群架构Horovod 与 Spark 的集成包的功能数据处理、模型训练和评估的一体化流程。
Spark SQL参考文献
06-08
4. 《Spark SQL实战》：https://book.douban.com/subject/26954456/ 5. 《Spark大数据处理技术》：https://book.douban.com/subject/27037654/ 6. 《Spark大数据分析实战》：...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值