hive 行转列/列转行 多行转一行/一行转多行

mysql跟hive列转行/行转列一样,但是多行转一行,一行转多行就不太一样了

链接:mysql 行转列,多行转一行,列转行,一行转多列

hive

启动hive时 ./hive -S (去除MR打印日志)
hive命令行 set hive.exec.mode.local.auto=true; 设置本地模式
数据准备:

create table student_score(s_id int,s_name string,s_sub string,s_score bigint);
insert into student_score values
(1,'张三','数学',90),
(2,'张三','语文',85),
(3,'张三','英语',92),
(4,'李四','数学',88),
(5,'李四','语文',91),
(6,'李四','英语',99),
(7,'王五','数学',100),
(8,'王五','语文',82),
(9,'王五','英语',88);

行转列

思路,先使用case when 分成数学,语文,英语 三列

--注意,hive不支持列名为中文
select s_name,
case s_sub when '数学' then s_score else 0 end shuxue,
case s_sub when '语文' then s_score else 0 end yuwen,
case s_sub when '英语' then s_score else 0 end yingyu
from student_score ;

再根据 名字 group by

select s_name,
sum(case s_sub when '数学' then s_score else 0 end) shuxue,
sum(case s_sub when '语文' then s_score else 0 end) yuwen,
max(case s_sub when '英语' then s_score else 0 end) yingyu
from student_score group by s_name;

列转行

先创建上图中的表

create table student_score2 as 
select s_name,
sum(case s_sub when '数学' then s_score else 0 end) shuxue,
sum(case s_sub when '语文' then s_score else 0 end) yuwen,
max(case s_sub when '英语' then s_score else 0 end) yingyu
from student_score group by s_name;

列转行sql:

select s_name,'数学' as s_sub, shuxue as s_score from student_score2
union all select s_name,'语文' as s_sub, yuwen as s_score from student_score2
union all select s_name,'英语' as s_sub, yingyu as s_score from student_score2;

结果

多行转一行

生成格式:

张三           数学:90 语文:85 英语:92

先说函数:
1 collect_xx函数:
collect_set(column)
collect_list(column)

set去重,list不去重,通过group by,将相通s_name的s_sub列转成数组.

2 concat_ws函数
单行中的列聚合:concat_ws(sp,str1,str2,…) sp是分隔符

ps:单行列聚合还有一个函数式concat(str1,str2,…) 跟concat_ws区别是前者如果有一个null,就后返回null并且只是字符串拼接,后者会把null去除,可以设定分隔符拼接.
多行中的列聚合 concat_ws(sp,arr1,arr2,…) 结合数组参数collect_set/collect_list使用,并且只能是字符串数组,所以其他类型要转化成string.
完整sql:

select s_name,
concat_ws('|',collect_set(concat(s_sub,cast(s_score as string)))) as all_score
from student_score group by s_name;

ps:mysql是使用group_concat函数完成多行转一行的,hive中有些版本没有group_concat函数

一行转多行:

先创建上图中的表

create table studnent_score3 as 
select s_name,
concat_ws('|',collect_set(concat(s_sub,cast(s_score as string)))) as all_score
from student_score group by s_name;

lateral view 侧视图
lateral view + UDTF 使用 UDTF一进多出函数
explode() 函数是UDTF函数,接收一个数组或者map函数
split(column,sp) split 接收一个字符串函数,指定分隔符返回一个数组
这里使用 lateral view explode(split(列名,分隔符)) 表名(随便写) as 列名(新列名)

sql:

select s_name,score from student_score3 
lateral view explode(split(all_score,'\\|')) a as score;

详解一下,lateral view 相当于开了一个视图,explode()函数只有查一个列时候才有效,比如

只查explode(split(all_score,‘\|’)) 是可以的,但是加上s_name是不行的,因为s_name不知道如何处理.
lateral view explode(split(all_score,‘\|’) a 就相当于开了一个a视图,只有一列 as score.
后面视图中列score 与前面的s_name 再进行笛卡尔积,就展现出了正确的结果,并且lateral view 和可以连着用,lateral veiw 1 lateral view 2
如

select name ,alie,blie from student 
lateral view explode(split(a,',')) a as alie 
lateral view explode(split(b,',')) b as blie;