Hive窗口函数使用

最新推荐文章于 2024-02-11 16:48:58 发布

小财迷，嘻嘻

最新推荐文章于 2024-02-11 16:48:58 发布

阅读量214

点赞数 1

分类专栏： hive 文章标签： hive 大数据

本文链接：https://blog.csdn.net/weixin_48185778/article/details/108677737

版权

hive 专栏收录该内容

12 篇文章 0 订阅

订阅专栏

1、简介

1、窗口函数具有什么功能?
分组和排序
但是这些功能我们的groupby和order by不是就可以实现了吗?看差别:
场景:统计各班人数
当我们使用group by来分组统计的时候
在这里插入图片描述
当我们使用窗口函数的时候

很明显当我们使用group by分组时改变了原来数据库的行数,即生成了一张新表,但是使用窗口函数的时候并不会影响到我们的行数,这样我们在进行既需要分组又需要显示详细信息的时候可以使用该函数

2、那么为什么叫窗口函数?
这是因为该函数在分组后的结果被称为"窗口"
窗口函数的语法:

<窗口函数> over (partition by <用于分组的列名>
order by <用于排序的列名>)

窗口函数中的partition by是对表中需要分组的列来进行分组的

在知乎上看到的大佬文，分享出来~
Hive窗口函数：https://zhuanlan.zhihu.com/p/113245904

补充常用窗口函数：
在这里插入图片描述

2、如何使用窗口函数

数据准备

Michael|1000|99|5000|full|2014-01-29
Will|1000|100|4000|full|2013-10-02
Wendy|1000|101|4000|part|2014-10-02
Steven|1000|102|6400|part|2012-11-03
Lucy|1000|103|5500|full|2010-01-03
Lily|1001|104|5000|part|2014-11-29
Jess|1001|105|6000|part|2014-12-02
Mike|1001|106|6400|part|2013-11-03
Wei|1002|107|7000|part|2010-04-03
Yun|1002|108|5500|full|2014-01-29
Richard|1002|109|8000|full|2013-09-01


create table if not exists employee_contract(
name string, 		--姓名
dept_num int,		--部门编号
employee_id int,	--员工编号
salary int,			--薪资
type string,		--员工类型，兼职还是全职
start_date date		--入职日期
)
row format delimited
fields terminated by '|'
stored as textfile;

--加载本地数据
load data local inpath '/tmp/hivedemo/data/employee_contract.txt' overwrite into table employee_contract;

语法

<窗口函数> over (partition by <用于分组的列名>
order by <用于排序的列名>)

partition by：在over窗口中进行分区，对么一列进行分区统计，窗口的大小就是分区的大小
order by：会对输入的数据强制排序

2.1 窗口功能之序列

窗口函数的序列功能在工作中还是比较常用的。

常用序列函数如下：

ROW_NUMBER()：对所有数值输出不同的序号，序号唯一连续
RANK()：对相同数值，输出相同的序号，下一个序号跳过（1,1,3）
DENSE_RANK()：对相同数值，输出相同的序号，下一个序号连续（1,1,2）
NLITE(n)：将有序的数据集合平均分配到n个桶中, 将桶号分配给每一行，根据桶号，选取前或后 n分之几的数据
PERCENT_RANK()：(目前排名- 1)/(总行数- 1)，值相对于一组值的百分比排名

--给所有的员工薪资排序，从低到高，针对所有员工
--row_number()
select name,dept_num,salary,
row_number() over(order by salary) as rn
from employee_contract;

--rank
select name,dept_num,salary,
rank() over(order by salary) as r
from employee_contract;

--dense_rank
select name,dept_num,salary,
dense_rank() over(order by salary) as dr
from employee_contract;

--nlite
select name,dept_num,salary,
ntile(2) over(partition by dept_num order by salary) as n
from employee_contract;

--按部门，对每个部门的员工薪资进行升序排列
select name,dept_num,salary,
row_number() over(partition by dept_num order by salary) as rn
from employee_contract;

--同时使用多个窗口函数
select name,dept_num,salary,
row_number() over () as row_num, 
rank() over (partition by dept_num order by salary) as rank, 
dense_rank() over (partition by dept_num order by salary) as dense_rank,
percent_rank() over(partition by dept_num order by salary) as percent_rank, 
ntile(2) over(partition by dept_num order by salary) as ntile 
from employee_contract 
order by dept_num, salary;

--工作中会遇到的场景
--按部门分组，获取每个部门薪资最低的员工(薪资相同算同一排名)
select name,dept_num,salary from(
select name,dept_num,salary,
dense_rank() over(partition by dept_num order by salary) as dr
from employee_contract) t
where t.dr=1;

2.2 窗口功能之聚合

和mysql的使用相同，不赘述。

count()
sum()
avg()
max()
min()

从Hive 2.1.0开始在OVER子句中支持聚合函数

SELECT rank() OVER (ORDER BY sum(b))  FROM T GROUP BY a;

select name,dept_num,salary,
COUNT(*) over (partition by dept_num) as row_cnt,
-- COUNT(DISTINCT *) over (partition by dept_num) as row_cnt_dis,
SUM(salary) over(partition by dept_num order by dept_num) as deptTotal,
SUM(salary) over(order by dept_num) as runningTotal1, 
SUM(salary) over(order by dept_num, name rows unbounded preceding) as runningTotal2,
avg(salary) over(partition by dept_num) as avgDept,
min(salary) over(partition by dept_num) as minDept,
max(salary) over(partition by dept_num) as maxDept
from employee_contract
order by dept_num, name;

2.3 窗口功能之分析

CUME_DIST：小于等于当前值的行数/分组内总行数
LEAD／LAG(col,n)：某一列进行往前/后第n行值(n可选，默认为1)
FIRST_VALUE：对该列到目前为止的首个值
LAST_VALUE：到目前行为止的最后一个值
cume_dist:小于等于当前的行数/分组内总行数

select name, dept_num, salary,
lead(salary, 2) over(partition by dept_num order by salary) as lead,
lag(salary, 2, 0) over(partition by dept_num order by salary) as lag,
first_value(salary) over (partition by dept_num order by salary) as first_value,
last_value(salary) over (partition by dept_num order by salary) as last_value_default,
last_value(salary) over (partition by dept_num order by salary rows between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING) as last_value
from employee_contract
order by dept_num, salary;

lead(col,n,i):对col列向下n行错位匹配，没有值的就用i填补

select name, dept_num, salary,
lead(salary, 2,0) over(partition by dept_num order by salary) as lead
from employee_contract; 
+----------+-----------+---------+-------+--+
|   name   | dept_num  | salary  | lead  |
+----------+-----------+---------+-------+--+
| Wendy    | 1000      | 4000    | 5000  |
| Will     | 1000      | 4000    | 5500  |
| Michael  | 1000      | 5000    | 6400  |
| Lucy     | 1000      | 5500    | 0     |
| Steven   | 1000      | 6400    | 0     |
| Lily     | 1001      | 5000    | 6400  |
| Jess     | 1001      | 6000    | 0     |
| Mike     | 1001      | 6400    | 0     |
| Yun      | 1002      | 5500    | 8000  |
| Wei      | 1002      | 7000    | 0     |
| Richard  | 1002      | 8000    | 0     |
+----------+-----------+---------+-------+--+
--解释：salary的第三个值放在lead的第一个，也就是错两位匹配，没有值用0填补

lag(col,n,i):与lead相反，对col列向上n行错位匹配，没有值的就用i填补

select name, dept_num, salary,
lag(salary, 2, 0) over(partition by dept_num order by salary) as lag
from employee_contract;
+----------+-----------+---------+-------+--+
|   name   | dept_num  | salary  |  lag  |
+----------+-----------+---------+-------+--+
| Wendy    | 1000      | 4000    | 0     |
| Will     | 1000      | 4000    | 0     |
| Michael  | 1000      | 5000    | 4000  |
| Lucy     | 1000      | 5500    | 4000  |
| Steven   | 1000      | 6400    | 5000  |
| Lily     | 1001      | 5000    | 0     |
| Jess     | 1001      | 6000    | 0     |
| Mike     | 1001      | 6400    | 5000  |
| Yun      | 1002      | 5500    | 0     |
| Wei      | 1002      | 7000    | 0     |
| Richard  | 1002      | 8000    | 5500  |
+----------+-----------+---------+-------+--+
--解释：salary的第一个值放在lag的第三个，也就是错两位匹配，没有值用0填补

first_value(col):对该列到目前为止的首个值

select name, dept_num, salary,
first_value(salary) over (partition by dept_num order by salary) as first_value
from employee_contract;

+----------+-----------+---------+--------------+--+
|   name   | dept_num  | salary  | first_value  |
+----------+-----------+---------+--------------+--+
| Wendy    | 1000      | 4000    | 4000         |
| Will     | 1000      | 4000    | 4000         |
| Michael  | 1000      | 5000    | 4000         |
| Lucy     | 1000      | 5500    | 4000         |
| Steven   | 1000      | 6400    | 4000         |
| Lily     | 1001      | 5000    | 5000         |
| Jess     | 1001      | 6000    | 5000         |
| Mike     | 1001      | 6400    | 5000         |
| Yun      | 1002      | 5500    | 5500         |
| Wei      | 1002      | 7000    | 5500         |
| Richard  | 1002      | 8000    | 5500         |
+----------+-----------+---------+--------------+--+

last_value(col):到目前行为止的最后一个值

select name, dept_num, salary,
last_value(salary) over (partition by dept_num order by salary) as last_value
from employee_contract;

+----------+-----------+---------+-------------+--+
|   name   | dept_num  | salary  | last_value  |
+----------+-----------+---------+-------------+--+
| Wendy    | 1000      | 4000    | 4000        |
| Will     | 1000      | 4000    | 4000        |
| Michael  | 1000      | 5000    | 5000        |
| Lucy     | 1000      | 5500    | 5500        |
| Steven   | 1000      | 6400    | 6400        |
| Lily     | 1001      | 5000    | 5000        |
| Jess     | 1001      | 6000    | 6000        |
| Mike     | 1001      | 6400    | 6400        |
| Yun      | 1002      | 5500    | 5500        |
| Wei      | 1002      | 7000    | 7000        |
| Richard  | 1002      | 8000    | 8000        |
+----------+-----------+---------+-------------+--+

cume_dist():小于等于当前的行数/分组内总行数

select name, dept_num, salary,
cume_dist() over (partition by dept_num order by salary) as cd
from employee_contract;

+----------+-----------+---------+---------------------+--+
|   name   | dept_num  | salary  |         cd          |
+----------+-----------+---------+---------------------+--+
| Wendy    | 1000      | 4000    | 0.4                 |
| Will     | 1000      | 4000    | 0.4                 |
| Michael  | 1000      | 5000    | 0.6                 |
| Lucy     | 1000      | 5500    | 0.8                 |
| Steven   | 1000      | 6400    | 1.0                 |
| Lily     | 1001      | 5000    | 0.3333333333333333  |
| Jess     | 1001      | 6000    | 0.6666666666666666  |
| Mike     | 1001      | 6400    | 1.0                 |
| Yun      | 1002      | 5500    | 0.3333333333333333  |
| Wei      | 1002      | 7000    | 0.6666666666666666  |
| Richard  | 1002      | 8000    | 1.0                 |
+----------+-----------+---------+---------------------+--+
--解释：在1000部门内，薪资<4000的占0.4，薪资<5000的占0.6。

3、窗口子句

窗口字句用于进一步细分结果并应用分析函数
支持两类窗口子句：行类型窗口、范围类型窗口
RANK、NTILE、DENSE_RANK、CUME_DIST、PERCENT_RANK、LEAD、LAG和ROW_NUMBER函数不支持与窗口子句一起使用

3.1 行窗口

根据当前行之前或之后的行号确定的窗口。

语法：rows between 开始行 and 结束行

unbounded preceding:第一行
m preceding:当前行往前m行
current row：当前行
n preceding:当前行往后n行
unbounded following：最后一行

在这里插入图片描述

SELECT
name, dept_num AS dept, salary AS sal,
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) win1, --当前行往前2行到当前行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN 2 PRECEDING AND UNBOUNDED FOLLOWING) win2, --当前行往前2行到最后一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN 1 PRECEDING AND 2 FOLLOWING) win3,  --当前行前一行到当前行后一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN 2 PRECEDING AND 1 PRECEDING) win4,  --当前行前2行到当前行前2行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING) win5,  --当前行往后一行到当前行往后2行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN CURRENT ROW AND CURRENT ROW) win6,  --当前行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING) win7,  --当前行到下一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) win8,  --当前行到最后一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) win9,  --第一行到当前行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING) win10,  --第一行到当前行的下一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) win11,  --第一行到最后一行
MAX(salary) OVER (PARTITION BY dept_num ORDER BY name ROWS 2 PRECEDING) win12 --前两行
FROM employee_contract  ORDER BY dept, name;

3.2 范围窗口

取分组内的值在指定范围区间内的行
该范围值/区间必须是数字或日期类型
目前只支持一个ORDER BY列
工作中很少用

语法：range between 开始行 and 结束行

SUM(close) RANGE BETWEEN 500 PRECEDING AND 1000 FOLLOWING
-- 假设当前close值为3000，语句将包含分区内范围从2500到4000的行

--示例
SELECT name, dept_num AS dept, salary AS sal,
MAX(salary) OVER (PARTITION BY dept_num ORDER BY salary ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) win1,
salary - 1000 as sal_r_start,salary as sal_r_end,
MAX(salary) OVER (PARTITION BY dept_num ORDER BY salary RANGE BETWEEN 1000 PRECEDING AND CURRENT ROW) win13
FROM employee_contract ORDER BY dept, name;