大数据之SparkHive在生产实际中的如何优雅的解决数据倾斜的问题

问题： 原数据匹配事件名称、定位信息，因某个字段对应的数据条数特别多，任务运行时间较长，在shuffle时总是失败。

报错信息：

ShuffleMapStage has failed the maxinum allowable number of times
Caused by :io.netty.utile.internal.OutOfDirectMemoryError:failed to allocate 16777216 bytes of direct memory(used:3741319168,max:3750756352)

好家伙数据量还挺大，任务运行这么久还要报错，顿时来气，那就上刀子一步到位，把数据倾斜问题解决了吧。

**方案：通过关联条件字段的拼接随机数进行加盐，扩容小表的匹配字段,最后记得去除添加的随机数，还原数据，才是正确结果**

睁大眼睛好好看，关联表那就加上随机数，具体实施：

1. 业务表：

select
concat_ws('_',event_id,cast(rand() * 1000% 6 as int)) as event_id,
concat_ws('_',s_ip_string,cast(rand() * 1000 % 6 as int)) as s_ip_string,
concat_ws('_',d_ip_string,cast(rand() * 1000 % 6 as int)) as d_ip_string,
s_port,
d_port,
c_time
from atable

2. 规则表：

select
concat_ws('_',event_id,num) as event_id,
event_name,
from btable
lateral view explode(split('1,2,3,4,5,6,7,8,9,0',',')) tp as num
#tp是临时表名

3. 定位信息表：

select
concat_ws('_',ip,num) as ip,
company,
city
from ctable
lateral view explode(split('1,2,3,4,5,6,7,8,9,0',',')) tp as num

随机数添加好了，shuffle的时候就可以很均匀的分配任务到executor了吧

最后记得记得记得，重要事说三遍！！
还没结束呢，需要处理一下！

select
split(event_id,’_’)[0] as event_id,
event_name,
split(s_ip_string,’_’)[0] as s_ip_string,
s_city,
s_company,
split(d_ip_string,’_’)[0] as d_ip_string,
d_city,
d_company,
d_port,
s_port,
time
from 
a

小朋友，你学fei了吗？