DataFrame算子

算子分类
      |__ RDD同源
      |    |__ 数据转换
      |    |             |__ map
      |    |             |__ mapPartitions
      |    |             |__ flatMap
      |    |             |__ filter
      |    |__ 数据聚合
      |    |             |__ groupByKey
      |    |             |__ reduce
      |    |__ 数据准备
      |    |             |__ union
      |    |             |__ sample
      |    |__ 数据预处理
      |    |             |__ coalesce
      |    |             |__ repartition
      |    |__ 结构收集
      |                  |__ first
      |                  |__ take
      |                  |__ collect
      |__ Schema同源
          |__ 探索类算子
          |              |__ 查看数据模式
          |              |                |__ columns
          |              |                |__ schema
          |              |                |__ printSchema
          |              |__ 查看数据"模样"
          |              |                |__ show
          |              |__ 查看数据分布
          |              |                |__ describe
          |              |__ 查看数据执行计划
          |                               |__ explain
          |__ 清洗类算子
          |              |__ 删除DataFrame中列数据
          |              |                |__ drop
          |              |__ 去重
          |              |                |__ distinct
          |              |__ 按照指定列去重
          |              |                |__ drop
          |              |__ null处理
          |                               |__ na
          |__ 转换类算子
          |              |__ 按照列名对数据做投影
          |              |                |__ select
          |              |__ 以SQL语句为参数生成、提取数据
          |              |                |__ selectExpr
          |              |__ 以SQL语句为参数做数据过滤
          |              |                |__ where
          |              |__ 字段重命名
          |              |                |__ withColumnRenamed
          |              |__ 生成新的数据列
          |              |                |__ withColumn
          |              |__ 展开数组类的数据列
          |                               |__ explode
          |__ 分析类算子
          |              |__ 两个DataFrame间做数据关联
          |              |                |__ join
          |              |__ 按照列分组
          |              |                |__ groupBy
          |              |__ 分组后做数据聚合
          |              |                |__ agg
          |              |__ 排序
          |                               |__ sort
          |                               |__ orederBy
          |__ 持久化类算子
                        |__ write

算子分类

1 RDD同源类算子

1.1 数据转换

map

mapPartitions

flatMap

filter

1.2 数据聚合

groupByKey

reduce

1.3 数据准备

union

sample

1.4 数据预处理

coalesce

repartition

1.5 结构收集

first

take

collect

2. Schema同源类算子

2.1 探索类算子

2.1.0 查看数据模式

columns

schema

printSchema

2.1.1 查看数据"模样"

show

2.1.2 查看数据分布

describe

2.1.3 查看数据执行计划

explain

2.2 清洗类算子

2.2.1 删除DataFrame中列数据

drop

2.2.2 去重

distinct

2.2.3 按照指定列去重

drop

2.2.4 null处理

na

2.3 转换类算子

2.3.0 按照列名对数据做投影

select

2.3.1 以SQL语句为参数生成、提取数据

selectExpr

2.3.2 以SQL语句为参数做数据过滤

where

2.3.3 字段重命名

withColumnRenamed

2.3.4 生成新的数据列

withColumn

2.3.5 展开数组类的数据列

explode

2.4 分析类算子

2.4.0 两个DataFrame间做数据关联

join

2.4.1 按照列分组

groupBy

2.4.2 分组后做数据聚合

agg

2.4.3 排序

sort

orederBy

2.5 持久化类算子

write