2021-01-06

最新推荐文章于 2024-04-26 16:47:03 发布

q_15403863

最新推荐文章于 2024-04-26 16:47:03 发布

阅读量323

点赞数

文章标签： javascript java 开发语言

本文链接：https://blog.csdn.net/qq_15403863/article/details/112277292

版权

<div id="img-content" class="rich_media_wrp">

<h2 class="rich_media_title" id="activity-name">



漫谈千亿级数据优化实践：数据倾斜
</h2>
<div id="meta_content" class="rich_media_meta_list">
<span class="rich_media_meta rich_media_meta_text">
大数据技术与架构
</span>

<span class="rich_media_meta rich_media_meta_nickname" id="profileBt">
<a href="javascript:void(0);" id="js_name">
数据仓库与Python大数据 </a>
<div id="js_profile_qrcode" class="profile_container" style="display:none;">
<div class="profile_inner">
<strong class="profile_nickname">数据仓库与Python大数据</strong>
<img class="profile_avatar" id="js_profile_qrcode_img" src="" alt="">

<p class="profile_meta">
<label class="profile_meta_label">功能介绍</label>
<span class="profile_meta_value">专注于大数据、数据仓库与数据科学，数据分析与数据挖掘，Hadoop/Spark/Flink、Python与面试经验和职场感悟</span>
</p>

</div>
<span class="profile_arrow_wrp" id="js_profile_arrow_wrp">
<i class="profile_arrow arrow_out"></i>
<i class="profile_arrow arrow_in"></i>
</span>
</div>
</span>
<em id="publish_time" class="rich_media_meta rich_media_meta_text">2020-12-02</em>
</div>

<div id="js_tags" class="article-tag__list" style="display: none;" data-len="0">

<div class="article-tag-card__title">收录于话题</div>
<div class="article-tags">
</div>
</div>

<p style="text-align: center;" data-mpa-powered-by="yiban.io"><img class="rich_pages __bg_gif" data-ratio="0.11310592459605028" data-src="https://mmbiz.qpic.cn/mmbiz_gif/1OYP1AZw0W21OGUNeTZDxFaljU3KfppQHic2LcWGEWobktvIlre4Xkr6aauuTznqP2wFtzyBKmclyp4PvhXHq9w/640?wx_fmt=gif" data-type="gif" data-w="557" style="width: 557px !important; height: auto !important; visibility: visible !important;" _width="557px" src="https://mmbiz.qpic.cn/mmbiz_gif/1OYP1AZw0W21OGUNeTZDxFaljU3KfppQHic2LcWGEWobktvIlre4Xkr6aauuTznqP2wFtzyBKmclyp4PvhXHq9w/640?wx_fmt=gif&tp=webp&wxfrom=5&wx_lazy=1" data-order="0" alt="图片" data-fail="0"></p><p style="text-align: center;"><a target="_blank" href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496298&idx=1&sn=d6082fc82882dc5e5255640f51865233&chksm=cf37c067f840497130834aaf9fcb6686cd86c7a52373fedc006e9b903eb6989c0af43faf8f61&scene=21#wechat_redirect" textvalue="你已选中了添加链接的内容" data-itemshowtype="0" tab="innerlink" data-linktype="1" hasload="1"><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img class="rich_pages " data-ratio="0.2824858757062147" data-s="300,640" data-src="https://mmbiz.qpic.cn/mmbiz_png/1OYP1AZw0W2FPhfWfjQRaywhbAEr6UUNDqd4M9r3c5hgt6ia9CeqdKpMsruP0xvJJKAaAOvWygTOTcsiaGibNyZbQ/640?wx_fmt=png" data-type="png" data-w="1062" style="margin: 0px; width: 677px !important; height: auto !important; visibility: visible !important;" _width="677px" src="https://mmbiz.qpic.cn/mmbiz_png/1OYP1AZw0W2FPhfWfjQRaywhbAEr6UUNDqd4M9r3c5hgt6ia9CeqdKpMsruP0xvJJKAaAOvWygTOTcsiaGibNyZbQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1" crossorigin="anonymous" alt="图片" data-fail="0"></span></a></p><h2 style="margin-top: 2.5rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.75rem;padding-bottom: 0.5rem;border-bottom: 1px double rgba(0, 0, 0, 0.1);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">0x00 前言</h2><blockquote style="margin-top: 0px;margin-bottom: 1.25rem;padding-left: 1rem;border-left-width: 0.25rem;border-left-color: rgb(233, 236, 239);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><p style="margin-bottom: 1.25rem;">数据倾斜是大数据领域绕不开的拦路虎，当你所需处理的数据量到达了上亿甚至是千亿条的时候，数据倾斜将是横在你面前一道巨大的坎。</p><p style="margin-bottom: 1.25rem;">迈的过去，将会海阔天空！迈不过去，就要做好准备：很可能有几周甚至几月都要头疼于数据倾斜导致的各类诡异的问题。</p></blockquote><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">郑重声明：</span></p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>话题比较大，技术要求也比较高，笔者尽最大的能力来写出自己的理解，写的不对和不好的地方大家一起交流。</p></li><li><p>有些例子不是特别严谨，一些小细节对文章理解没有影响，不要太在意。（比如我在算机器内存的时候，就不把Hadoop自身的进程算到使用内存中）</p></li></ul><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">文章结构</h3><ol style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>先大致解释一下什么是数据倾斜</p></li><li><p>再根据几个场景来描述一下数据倾斜产生的情况</p></li><li><p>详细分析一下在Hadoop和Spark中产生数据倾斜的原因</p></li><li><p>如何解决（优化）数据倾斜问题？</p></li></ol><section style="margin-right: 8px;margin-left: 8px;font-variant-numeric: normal;font-variant-east-asian: normal;font-stretch: normal;font-size: 13px;line-height: normal;font-family: 'Helvetica Neue';white-space: normal;-webkit-text-stroke-color: rgb(136, 136, 136);text-align: left;color: rgb(136, 136, 136);background-color: rgb(255, 255, 255);"><span style="color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 1.75rem;font-weight: 600;">0x01 什么是数据倾斜</span><br></section><blockquote style="margin-top: 0px;margin-bottom: 1.25rem;padding-left: 1rem;border-left-width: 0.25rem;border-left-color: rgb(233, 236, 239);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><p style="margin-bottom: 1.25rem;">简单的讲，数据倾斜就是我们在计算数据的时候，数据的分散度不够，导致大量的数据集中到了一台或者几台机器上计算，这些数据的计算速度远远低于平均计算速度，导致整个计算过程过慢。</p></blockquote><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">一、关键字：<span style="font-weight: bolder;">数据倾斜</span></h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">相信大部分做数据的童鞋们都会遇到数据倾斜，数据倾斜会发生在数据开发的各个环节中，比如：</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>用Hive算数据的时候reduce阶段卡在99.99%</p></li><li><p>用SparkStreaming做实时算法时候，一直会有executor出现OOM的错误，但是其余的executor内存使用率却很低。</p></li></ul><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">这些问题经常会困扰我们，辛辛苦苦等了几个小时的数据就是跑不出来，心里多难过啊。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">例子很多，这里先随便举两个，后文会详细的说明。</p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">二、关键字：<span style="font-weight: bolder;">千亿级</span></h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">为什么要突出这么大数据量？先说一下笔者自己最初对数据量的理解：</p><blockquote style="margin-top: 0px;margin-bottom: 1.25rem;padding-left: 1rem;border-left-width: 0.25rem;border-left-color: rgb(233, 236, 239);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><p style="margin-bottom: 1.25rem;">数据量大就了不起了？数据量少，机器也少，计算能力也是有限的，因此难度也是一样的。凭什么数据量大就会有数据倾斜，数据量小就没有？</p></blockquote><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">这样理解也有道理，但是比较片面，举两个场景来对比：</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>公司一：总用户量1000万，5台64G内存的的服务器。</p></li><li><p>公司二：总用户量10亿，1000台64G内存的服务器。</p></li></ul><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">两个公司都部署了Hadoop集群。假设现在遇到了数据倾斜，发生什么？</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">公司一</span>的数据分时童鞋在做join的时候发生了数据倾斜，会导致有几百万用户的相关数据集中到了一台服务器上，几百万的用户数据，说大也不大，正常字段量的数据的话64G还是能轻松处理掉的。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">公司二</span>的数据分时童鞋在做join的时候也发生了数据倾斜，可能会有1个亿的用户相关数据集中到了一台机器上了（相信我，这很常见），这时候一台机器就很难搞定了，最后会很难算出结果。</p><h2 style="margin-top: 2.5rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.75rem;padding-bottom: 0.5rem;border-bottom: 1px double rgba(0, 0, 0, 0.1);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">0x02 数据倾斜长什么样</h2><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">笔者大部分的数据倾斜问题都解决了，而且也不想重新运行任务来截图，下面会分几个场景来描述一下数据倾斜的特征，方便读者辨别。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">由于Hadoop和Spark是最常见的两个计算平台，下面就以这两个平台说明：</p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">一、Hadoop中的数据倾斜</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">Hadoop中直接贴近用户使用使用的时Mapreduce程序和Hive程序，虽说Hive最后也是用MR来执行（至少目前Hive内存计算并不普及），但是毕竟写的内容逻辑区别很大，一个是程序，一个是Sql，因此这里稍作区分。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">Hadoop中的数据倾斜主要表现在、ruduce阶段卡在99.99%，一直99.99%不能结束。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">这里如果详细的看日志或者和监控界面的话会发现：</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>有一个多几个reduce卡住</p></li><li><p>各种container报错OOM</p></li><li><p>读写的数据量极大，至少远远超过其它正常的reduce</p></li></ul><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">伴随着数据倾斜，会出现任务被kill等各种诡异的表现。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">经验：Hive的数据倾斜，一般都发生在Sql中Group和On上，而且和数据逻辑绑定比较深。</span></p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">二、Spark中的数据倾斜</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">Spark中的数据倾斜也很常见，这里包括Spark Streaming和Spark Sql，表现主要有下面几种：</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>Executor lost，OOM，Shuffle过程出错</p></li><li><p>Driver OOM</p></li><li><p>单个Executor执行时间特别久，整体任务卡在某个阶段不能结束</p></li><li><p>正常运行的任务突然失败</p></li></ul><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">补充一下，在Spark streaming程序中，数据倾斜更容易出现，特别是在程序中包含一些类似sql的join、group这种操作的时候。因为Spark Streaming程序在运行的时候，我们一般不会分配特别多的内存，因此一旦在这个过程中出现一些数据倾斜，就十分容易造成OOM。</p><h2 style="margin-top: 2.5rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.75rem;padding-bottom: 0.5rem;border-bottom: 1px double rgba(0, 0, 0, 0.1);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">0x03 数据倾斜的原理</h2><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">一、数据倾斜产生的原因</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">我们以Spark和Hive的使用场景为例。他们在做数据运算的时候会设计到，countdistinct、group by、join等操作，这些都会触发Shuffle动作，一旦触发，所有相同key的值就会拉到一个或几个节点上，就容易发生单点问题。</p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">二、万恶的shuffle</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">Shuffle是一个能产生奇迹的地方，不管是在Spark还是Hadoop中，它们的作用都是至关重要的。关于Shuffle的原理，这里不再讲述，看看Hadoop相关的论文或者文章理解一下就ok。这里主要针对，在Shuffle如何产生了数据倾斜。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">Hadoop和Spark在Shuffle过程中产生数据倾斜的原理基本类似。如下图。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">大部分数据倾斜的原理就类似于下图，很明了，因为数据分布不均匀，导致大量的数据分配到了一个节点。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="display: block;text-align: center;"><img data-ratio="0.8984375" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/UdK9ByfMT2NKnzfwC98elakwkuvicl1u1G7eCzgicq1RbSN1wkwPxON4oUpXHerBicib3a9A7pcKCJamzqPicbvJNww/640?wx_fmt=jpeg" data-type="jpeg" data-w="640" _width="640px" class="img_loading" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" style="width: 640px !important; height: 575.203px !important;" crossorigin="anonymous"></span></p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">三、<span style="font-weight: bolder;">从数据角度来理解数据倾斜</span></h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">我们举一个例子，就说数据默认值的设计吧，假设我们有两张表：</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>user（用户信息表）：userid，register_ip</p></li><li><p>ip（IP表）：ip，register_user_cnt</p></li></ul><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">这可能是两个不同的人开发的数据表，如果我们的数据规范不太完善的话，会出现一种情况，user表中的register_ip字段，如果获取不到这个信息，我们默认为null，但是在ip表中，我们在统计这个值的时候，为了方便，我们把获取不到ip的用户，统一认为他们的ip为0。</span></p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">两边其实都没有错的，但是一旦我们做关联了会出现什么情况，这个任务会在做关联的阶段，也就是sql的on的阶段卡死。</p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">四、<span style="font-weight: bolder;">从业务计角度来理解数据倾斜</span></h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">数据往往和业务是强相关的，业务的场景直接影响到了数据的分布。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">再举一个例子，比如就说订单场景吧，我们在某一天在北京和上海两个城市多了强力的推广，结果可能是这两个城市的订单量增长了10000%，其余城市的数据量不变。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">然后我们要统计不同城市的订单情况，这样，一做group操作，可能直接就数据倾斜了。</p><h2 style="margin-top: 2.5rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.75rem;padding-bottom: 0.5rem;border-bottom: 1px double rgba(0, 0, 0, 0.1);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">0x04 如何解决</h2><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">数据倾斜的产生是有一些讨论的，解决它们也是有一些讨论的，本章会先给出几个解决数据倾斜的思路，然后对Hadoop和Spark分别给出一些解决数据倾斜的方案。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">注意：</span> 很多数据倾斜的问题，都可以用和平台无关的方式解决，比如更好的数据预处理，异常值的过滤等，因此笔者认为，解决数据倾斜的重点在于对数据设计和业务的理解，这两个搞清楚了，数据倾斜就解决了大部分了。</p><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">一、几个思路</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">解决数据倾斜有这几个思路：</p><ol style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>业务逻辑，我们从业务逻辑的层面上来优化数据倾斜，比如上面的例子，我们单独对这两个城市来做count，最后和其它城市做整合。</p></li><li><p>程序层面，比如说在Hive中，经常遇到<code style="font-family: SFMono-Regular, Menlo, Monaco, Consolas, 'Liberation Mono', 'Courier New', monospace;font-size: 14px;color: rgb(232, 62, 140);">count（distinct）</code>操作，这样会导致最终只有一个reduce，我们可以先group 再在外面包一层count，就可以了。</p></li><li><p>调参方面，Hadoop和Spark都自带了很多的参数和机制来调节数据倾斜，合理利用它们就能解决大部分问题。</p></li></ol><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">二、从业务和数据上解决数据倾斜</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">很多数据倾斜都是在数据的使用上造成的。我们举几个场景，并分别给出它们的解决方案。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);"><span style="font-weight: bolder;">数据分布不均匀：</span></p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">前面提到的“从数据角度来理解数据倾斜”和“从业务计角度来理解数据倾斜”中的例子，其实都是数据分布不均匀的类型，这种情况和计算平台无关，我们能通过设计的角度尝试解决它。</p><ul style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>有损的方法：</p></li><ul class="list-paddingleft-2" style="list-style-type: square;"><li><p>找到异常数据，比如ip为0的数据，过滤掉</p></li></ul><li><p>无损的方法：</p></li><ul class="list-paddingleft-2" style="list-style-type: square;"><li><p>对分布不均匀的数据，单独计算</p></li><li><p>先对key做一层hash，先将数据打散让它的并行度变大，再汇集</p></li></ul><li><p>数据预处理</p></li></ul><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">三、Hadoop平台的优化方法</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">列出来一些方法和思路，具体的参数和用法在官网看就行了。</p><ol style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>mapjoin方式</p></li><li><p>count distinct的操作，先转成group，再count</p></li><li><p>万能膏药：hive.groupby.skewindata=true</p></li><li><p>left semi jioin的使用</p></li><li><p>设置map端输出、中间结果压缩。（不完全是解决数据倾斜的问题，但是减少了IO读写和网络传输，能提高很多效率）</p></li></ol><h3 style="margin-top: 2.25rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.5rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">四、Spark平台的优化方法</h3><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">列出来一些方法和思路，具体的参数和用法在官网看就行了。</p><ol style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);" class="list-paddingleft-2"><li><p>mapjoin方式</p></li><li><p>设置rdd压缩</p></li><li><p>合理设置driver的内存</p></li><li><p>Spark Sql中的优化和Hive类似，可以参考Hive</p></li></ol><h2 style="margin-top: 2.5rem;margin-bottom: 1rem;font-weight: 600;line-height: 1.2;font-size: 1.75rem;padding-bottom: 0.5rem;border-bottom: 1px double rgba(0, 0, 0, 0.1);color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">0xFF 总结</h2><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">数据倾斜的坑还是很大的，如何处理数据倾斜是一个长期的过程，希望本文的一些思路能提供帮助。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">文中一些内容没有细讲，比如Hive Sql的优化，数据清洗中的各种坑，这些留待后面单独的分享，会有很多的内容。</p><p style="margin-bottom: 1.25rem;color: rgb(33, 37, 41);font-family: -apple-system, system-ui, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);">另外千亿级别的数据还会有更多的难点，不仅仅是数据倾斜的问题，这一点在后面也会有专门的分享。</p><section style="white-space: normal;max-width: 100%;overflow-wrap: break-word !important;box-sizing: border-box !important;"><section powered-by="xiumi.us" style="max-width: 100%;box-sizing: border-box;letter-spacing: 1px;font-size: 15px;overflow-wrap: break-word !important;"><section data-role="outer" label="Powered by 135editor.com"><section data-tools="135编辑器" data-id="98469"><section hm_fix="349:393"><section style="margin: 10px auto;width: 30px;"><img data-ratio="2.246376811594203" data-src="https://mmbiz.qpic.cn/mmbiz_gif/UdK9ByfMT2O2R2sWRGyQHreSAGqmpRwtrmHudAsc2zib2AiatXJ6goxkskB4JEFrld8ia4WYOCsVdVVmr5hgfDdpg/640?wx_fmt=gif" data-type="gif" data-w="138" data-width="100%" style="display: block; width: 30px !important; height: 67.3913px !important;" _width="30px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" class=" __bg_gif" data-order="1"></section><section data-role="outer" label="Powered by 135editor.com"><section data-tools="135编辑器" data-id="95753"><section data-width="50%" style="margin: 10px auto;width: 289px;height: 10px;overflow: hidden;"><img data-ratio="0.2519561815336463" data-src="https://mmbiz.qpic.cn/mmbiz_gif/UdK9ByfMT2O2R2sWRGyQHreSAGqmpRwtNhh4iaJx03bCpDFmIGtWfNCxp2qoM4UlGsfKsnYfajRKa5feialNm5uw/640?wx_fmt=gif" data-type="gif" data-w="639" data-width="100%" style="width: 289px !important; height: 74.3114px !important;" _width="289px" class="img_loading __bg_gif" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" data-order="2"></section></section></section></section></section></section><p style="font-variant-numeric: normal;font-variant-east-asian: normal;font-stretch: normal;font-size: 13px;line-height: normal;font-family: 'Helvetica Neue';-webkit-text-stroke-color: rgb(136, 136, 136);text-align: left;color: rgb(136, 136, 136);background-color: rgb(255, 255, 255);"><strong><span style="font-kerning: none;">版权声明：</span></strong></p><section style="font-variant-numeric: normal;font-variant-east-asian: normal;font-stretch: normal;font-size: 13px;font-family: 'Helvetica Neue';-webkit-text-stroke-color: rgb(136, 136, 136);text-align: left;color: rgb(136, 136, 136);background-color: rgb(255, 255, 255);line-height: 1.75em;"><span style="font-kerning: none;">本文为大数据技术与架构整理，原作者独家授权。</span>未经原作者允许转载追究侵权责任。</section><section style="font-variant-numeric: normal;font-variant-east-asian: normal;font-stretch: normal;font-size: 13px;font-family: 'Helvetica Neue';-webkit-text-stroke-color: rgb(136, 136, 136);text-align: left;color: rgb(136, 136, 136);background-color: rgb(255, 255, 255);line-height: 1.75em;"><br></section><section style="max-width: 100%;box-sizing: border-box;font-family: -apple-system-font, system-ui, 'Helvetica Neue', 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei UI', 'Microsoft YaHei', Arial, sans-serif;background-color: rgb(255, 255, 255);font-size: 16px;overflow-wrap: break-word !important;"><section powered-by="xiumi.us" style="max-width: 100%;box-sizing: border-box;text-align: center;font-size: 15px;color: rgb(255, 97, 149);overflow-wrap: break-word !important;"><section data-recommend-type="list-normal" data-recommend-tid="2" data-mpa-template="t" style="width: 100%;display: flex;justify-content: center;align-items: center;" data-mid="" data-from="yb-recommend"><section style="width: 100%;padding: 14px 7px 6px;background: rgb(255, 255, 255);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(232, 232, 235);" data-mid=""><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247496320_1" data-recommend-article-time="1606695360" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/yofhibCRConCNGoGFgeib9ogad1lWrIYicLibEXBqIAHuIpISkdibXA1XfU3ZgFLHxRHgTnwnwwj8CNYibMlT2nYokLA/0?wx_fmt=jpeg" data-recommend-article-title="Hive SQL 语法大全，宇宙最强整理（建议收藏）" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496320&idx=1&sn=9b0a3d0e6c5c6d1f0ccb39c53aa54b3c&chksm=cf37c08df840499bc3270d8f8fdf369cd67d4cbf1b39d442c38c59fa5e02ee03bcf7aede4dbf#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496320&idx=1&sn=9b0a3d0e6c5c6d1f0ccb39c53aa54b3c&chksm=cf37c08df840499bc3270d8f8fdf369cd67d4cbf1b39d442c38c59fa5e02ee03bcf7aede4dbf&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">Hive SQL 语法大全，宇宙最强整理（建议收藏）</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-30</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.4269788182831661" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/yofhibCRConCNGoGFgeib9ogad1lWrIYicLibEXBqIAHuIpISkdibXA1XfU3ZgFLHxRHgTnwnwwj8CNYibMlT2nYokLA/640?wx_fmt=jpeg" data-type="jpeg" data-w="897" style="object-fit: cover; margin: 0px; width: 51px !important; height: 21.7759px !important;" _width="51px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247496320_2" data-recommend-article-time="1606695360" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0LtGjJtHOJiax4TkJURZm1V7Z1G6ghn9xeJrLYFTwtG14tLKZicr8G0IGVzBmKRhMT54OO8T5z9KIg/0?wx_fmt=jpeg" data-recommend-article-title="企业级丨推荐系统架构体系" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496320&idx=2&sn=35f3989aad010b42857c51f6614122b2&chksm=cf37c08df840499b100b96a83f99a41ed742e782b1ac7d24a1f2f485c5bd29e861437452c003#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496320&idx=2&sn=35f3989aad010b42857c51f6614122b2&chksm=cf37c08df840499b100b96a83f99a41ed742e782b1ac7d24a1f2f485c5bd29e861437452c003&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">企业级丨推荐系统架构体系</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-30</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="1" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0LtGjJtHOJiax4TkJURZm1V7Z1G6ghn9xeJrLYFTwtG14tLKZicr8G0IGVzBmKRhMT54OO8T5z9KIg/640?wx_fmt=jpeg" data-type="jpeg" data-w="800" style="object-fit: cover; margin: 0px; width: 51px !important; height: 51px !important;" _width="51px" class="img_loading" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247496249_1" data-recommend-article-time="1606579051" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/zHbzQPKIBPjhFZFSyhsUtJ4ic1yWcPT76ewricibicvrVHCp2JiaeuUicRzpfm4DnslUhlLBKnpodMFMcOjqkctiaNpicw/0?wx_fmt=jpeg" data-recommend-article-title="做用户，绕不开画像！" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496249&idx=1&sn=bb42b22fae4dce00801c49684e6c6188&chksm=cf37c034f8404922e6aa465b5a1341bccc195ebb7daac841bd38c20800a64a94a92741a84cbc#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496249&idx=1&sn=bb42b22fae4dce00801c49684e6c6188&chksm=cf37c034f8404922e6aa465b5a1341bccc195ebb7daac841bd38c20800a64a94a92741a84cbc&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">做用户，绕不开画像！</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-28</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.4234375" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/zHbzQPKIBPjhFZFSyhsUtJ4ic1yWcPT76ewricibicvrVHCp2JiaeuUicRzpfm4DnslUhlLBKnpodMFMcOjqkctiaNpicw/640?wx_fmt=jpeg" data-type="jpeg" data-w="1280" style="object-fit: cover; margin: 0px; width: 51px !important; height: 21.5953px !important;" _width="51px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247496298_1" data-recommend-article-time="1606608960" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0iauB2FYcxw0yiacEnuqgu2puBn5CSbla7VuJDdibFIHykF5FxicrljDicneFhpIRxIycKCXuUbTtdVuA/0?wx_fmt=jpeg" data-recommend-article-title="再次分享！漫谈数仓OLAP技术哪家强？" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496298&idx=1&sn=d6082fc82882dc5e5255640f51865233&chksm=cf37c067f840497130834aaf9fcb6686cd86c7a52373fedc006e9b903eb6989c0af43faf8f61#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496298&idx=1&sn=d6082fc82882dc5e5255640f51865233&chksm=cf37c067f840497130834aaf9fcb6686cd86c7a52373fedc006e9b903eb6989c0af43faf8f61&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">再次分享！漫谈数仓OLAP技术哪家强？</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-29</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.42326732673267325" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0iauB2FYcxw0yiacEnuqgu2puBn5CSbla7VuJDdibFIHykF5FxicrljDicneFhpIRxIycKCXuUbTtdVuA/640?wx_fmt=jpeg" data-type="jpeg" data-w="404" style="object-fit: cover; margin: 0px; width: 51px !important; height: 21.5866px !important;" _width="51px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247496249_2" data-recommend-article-time="1606579051" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0Q4wd6zqLRItyaicW28wp2QiaeFPhxjHjfdqkBpqfibUPkrnQTYxLZdtohy9ua5RjMpOsKwu9wH8Kjg/0?wx_fmt=jpeg" data-recommend-article-title="浅谈“HTAP”前世今生" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496249&idx=2&sn=a28b1854de038cea9174e79742089136&chksm=cf37c034f84049221a5d6c0db337041631e71ee41069cdc700cea689fff3bd7be69b546ac5de#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247496249&idx=2&sn=a28b1854de038cea9174e79742089136&chksm=cf37c034f84049221a5d6c0db337041631e71ee41069cdc700cea689fff3bd7be69b546ac5de&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">浅谈“HTAP”前世今生</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-28</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.9981751824817519" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0Q4wd6zqLRItyaicW28wp2QiaeFPhxjHjfdqkBpqfibUPkrnQTYxLZdtohy9ua5RjMpOsKwu9wH8Kjg/640?wx_fmt=jpeg" data-type="jpeg" data-w="548" style="object-fit: cover; margin: 0px; width: 51px !important; height: 50.9106px !important;" _width="51px" class="img_loading" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247495907_2" data-recommend-article-time="1606263360" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W1ydqrzjS4F1dxm8MiaSUbhBCbHW0A8YCf5hWUcugkM7icnyrpbPNwb12E4nWeDiadP3FHbeXl7SEU9A/0?wx_fmt=jpeg" data-recommend-article-title="从美团外卖的数据仓库建设中，我学到了什么？" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247495907&idx=2&sn=2f78c0de9edda3df7254ecb8c481ac3b&chksm=cf37c2eef8404bf825309edf202f05442b5b1c93f20ff995d1f8cff0c5e9616636bc72821214#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247495907&idx=2&sn=2f78c0de9edda3df7254ecb8c481ac3b&chksm=cf37c2eef8404bf825309edf202f05442b5b1c93f20ff995d1f8cff0c5e9616636bc72821214&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">从美团外卖的数据仓库建设中，我学到了什么？</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-25</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.9931972789115646" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W1ydqrzjS4F1dxm8MiaSUbhBCbHW0A8YCf5hWUcugkM7icnyrpbPNwb12E4nWeDiadP3FHbeXl7SEU9A/640?wx_fmt=jpeg" data-type="jpeg" data-w="588" style="object-fit: cover; margin: 0px; width: 51px !important; height: 50.6667px !important;" _width="51px" class="img_loading" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section><section data-mpa-template="t" data-recommend-article-type="list-normal" data-recomment-template-id="2" data-recommend-article-id="2247495889_1" data-recommend-article-time="1606176960" data-recommend-article-cover="http://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0ic9jaYwspK3MOx0G6fAyorSSfaWUNzqYvjVDOeBYxsWh23NicBUAB9icBEaBqdR5jI6541y2ibLDE7Q/0?wx_fmt=jpeg" data-recommend-article-title="40亿条/秒！Flink流批一体在阿里双11首次落地的背后" data-recommend-article-content-url="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247495889&idx=1&sn=6e4cddcfc85a1ba92068bea350242921&chksm=cf37c2dcf8404bca4f58b24d6c2e56f87c6f651c6408bec9e0f55a72dfe8c4d2877d98088fa1#rd"><a href="http://mp.weixin.qq.com/s?__biz=Mzg3NjIyNjQwMg==&mid=2247495889&idx=1&sn=6e4cddcfc85a1ba92068bea350242921&chksm=cf37c2dcf8404bca4f58b24d6c2e56f87c6f651c6408bec9e0f55a72dfe8c4d2877d98088fa1&scene=21#wechat_redirect" data-linktype="1"><section data-recommend-content="t" style="width: 100%;padding: 5px 10px;background: rgb(249, 249, 249);border-radius: 3px;border-width: 1px;border-style: solid;border-color: rgb(239, 239, 239);margin-bottom: 8px;display: flex;justify-content: space-between;align-items: center;" data-mid=""><section style="margin-right: 12px;flex: 1;" data-mid=""><p data-recommend-title="t" style="font-size: 13px;font-weight: 400;color: #333333;line-height: 18px;margin-bottom: 20px;max-width: 100%;overflow: hidden;display: -webkit-box;-webkit-line-clamp: 2;-webkit-box-orient: vertical;" data-mid="">40亿条/秒！Flink流批一体在阿里双11首次落地的背后</p><p data-recommend-date="t" style="max-width: 100%;font-size: 11px;font-weight: 400;color: #999999;line-height: 16px;" data-mid="">2020-11-24</p></section><span class="js_jump_icon h5_image_link" data-positionback="static" style="top: auto;left: auto;margin: 0px;right: auto;bottom: auto;"><img data-ratio="0.4265279583875163" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W0ic9jaYwspK3MOx0G6fAyorSSfaWUNzqYvjVDOeBYxsWh23NicBUAB9icBEaBqdR5jI6541y2ibLDE7Q/640?wx_fmt=jpeg" data-type="jpeg" data-w="769" style="object-fit: cover; margin: 0px; width: 51px !important; height: 21.7529px !important;" _width="51px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></span> </section></a></section></section></section><p><br></p><p style="font-size: 16px;"><strong style="color: rgba(0, 0, 0, 0.8);letter-spacing: 0.544px;font-size: 17px;"><span data-darkmode-bgcolor-16008590572276="rgb(25, 25, 25)" data-darkmode-original-bgcolor-16008590572276="rgb(255, 255, 255)" data-darkmode-color-16008590572276="rgb(136, 151, 171)" data-darkmode-original-color-16008590572276="rgb(63, 74, 89)" data-style="color: rgb(63, 74, 89); font-family: 微软雅黑; font-size: 11pt; letter-spacing: 0.4pt; text-align: left; text-indent: 0pt; visibility: visible;" data-darkmode-bgcolor="rgb(36, 36, 36)" data-darkmode-original-bgcolor="rgb(255, 255, 255)" data-darkmode-color="rgb(136, 151, 171)" data-darkmode-original-color="rgb(63, 74, 89)" style="color: rgb(63, 74, 89);font-family: 微软雅黑;font-size: 11pt;letter-spacing: 0.4pt;text-align: left;text-indent: 0pt;visibility: visible;">今天的分享就到这里，谢谢大家。</span></strong></p><p style="margin-top: 10px;margin-bottom: 10px;letter-spacing: 0.544px;text-align: center;line-height: 2em;"><span style="font-size: 13px;letter-spacing: 0.5px;color: rgb(0, 128, 255);">分享、点赞、在看，人间真爱~</span></p><p><br></p><section data-mpa-template="t" mpa-from-tpl="t"><section data-mpa-template="t" mpa-from-tpl="t"><section data-id="20187" data-md5="fcdad" mpa-from-tpl="t"><section data-md5="fcdad" mpa-from-tpl="t" style="margin-right: auto;margin-left: auto;width: 200px;height: 59px;background: url("https://mmbiz.qpic.cn/mmbiz_png/1OYP1AZw0W1kW09m20UHpPEiaSm2rh5dblrZ04WzicdicK9346qKibZMx69HBAWiaQ7PN2X2gKibwic470jRC9ZW57Jjg/640?wx_fmt=png") 0% 0% / 100% 100% no-repeat;"><section data-md5="fcdad" mpa-from-tpl="t" style="background: rgba(255, 255, 255, 0.65);height: 58.9875px;"><section data-md5="fcdad" mpa-from-tpl="t" style="padding-top: 18px;"><section data-md5="fcdad" mpa-from-tpl="t" style="color: rgb(215, 19, 23);letter-spacing: 1px;line-height: 24px;text-align: center;"><p data-md5="fcdad" style="font-size: 16px;"><strong data-md5="fcdad" mpa-from-tpl="t" mpa-is-content="t"><strong>▼ </strong><strong>福利时刻</strong><strong> ▼</strong></strong><strong style="font-family: Optima-Regular, PingFangTC-light;color: rgb(40, 100, 117);letter-spacing: 0.5px;"><span style="font-size: 15px;"> </span></strong></p></section></section></section></section></section></section></section><section data-role="outer" label="Powered by 135editor.com" style="letter-spacing: 0.544px;color: rgb(29, 29, 29);font-size: 16px;font-family: 微软雅黑;text-align: center;"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><p style="font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;color: rgb(160, 160, 160);font-size: 14px;text-align: left;"><span style="letter-spacing: 0.544px;color: rgb(51, 51, 51);text-align: justify;">01. 扫码后台回复「<span style="color: rgb(255, 0, 0);letter-spacing: 0.544px;">经典</span>」,即可领取大数据数仓经典书籍。</span></p><p style="font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;color: rgb(160, 160, 160);font-size: 14px;text-align: left;"><span style="letter-spacing: 0.544px;color: rgb(51, 51, 51);text-align: justify;">02. <span style="letter-spacing: 0.544px;">扫码</span><span style="letter-spacing: 0.544px;">后台回复「</span><span style="letter-spacing: 0.544px;color: rgb(255, 0, 0);">中台</span><span style="letter-spacing: 0.544px;">」,即可领取大厂中台架构高清ppt。</span><br></span></p><p style="margin-bottom: 25px;letter-spacing: 0.544px;color: rgb(47, 47, 47);font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif;text-align: left;word-break: break-word !important;"><strong style="letter-spacing: 0.544px;"><span style="color: rgb(51, 51, 51);font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 14px;letter-spacing: 0.544px;text-align: justify;">03</span></strong><span style="letter-spacing: 0.544px;color: rgb(51, 51, 51);font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 14px;text-align: justify;">.<span style="letter-spacing: 0.544px;">扫码</span>后台回复关键词：</span><span style="letter-spacing: 0.544px;font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 14px;text-align: justify;color: rgb(255, 0, 0);">画像源码、画像ppt、用户画像</span><span style="letter-spacing: 0.544px;color: rgb(51, 51, 51);font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 14px;text-align: justify;">，都可获取宝贵干货资源与资料</span><span style="color: rgb(51, 51, 51);font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;font-size: 17px;text-align: justify;"></span></p></section></section></section></section></section></section><section powered-by="xiumi.us" style="letter-spacing: 0.544px;font-family: Optima-Regular, PingFangTC-light;"><section><section data-role="outer" label="Powered by 135editor.com" style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;text-align: center;"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><section data-role="outer" label="Powered by 135editor.com"><p style="margin-bottom: 25px;letter-spacing: 0.544px;color: rgb(47, 47, 47);font-family: -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif;text-align: left;word-break: break-word !important;"><strong><span style="font-size: 14px;color: rgb(51, 51, 51);text-align: justify;letter-spacing: 0.544px;font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;">04.更多福利：<br></span></strong></p><table width="676"><tbody style="box-sizing: border-box !important;"><tr style="box-sizing: border-box !important;"><th style="word-break: break-all;box-sizing: border-box !important;">关键词<br style="box-sizing: border-box !important;"></th><th width="63" style="word-break: break-all;box-sizing: border-box !important;">领取资源<br style="box-sizing: border-box !important;"></th></tr><tr style="box-sizing: border-box !important;"><td width="172" valign="top" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">ck安装</span></td><td width="235.66666666666666" valign="top" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">clickhouse安装pdf文档</span></td></tr><tr style="box-sizing: border-box !important;"><td width="172" valign="top" style="word-break: break-all;box-sizing: border-box !important;"><span style="font-size: 15px;box-sizing: border-box !important;">0808<br style="box-sizing: border-box !important;"></span></td><td width="235" valign="top" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">大厂实时数仓PPT合集</span></td></tr><tr style="box-sizing: border-box !important;"><td valign="top" colspan="1" rowspan="1" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">画像源码</span></td><td valign="top" colspan="1" rowspan="1" width="63" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">用户画像项目源码</span></td></tr><tr style="box-sizing: border-box !important;"><td valign="top" colspan="1" rowspan="1" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">推荐系统</span></td><td valign="top" colspan="1" rowspan="1" width="63" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">推荐系统教程视频</span></td></tr><tr style="box-sizing: border-box !important;"><td valign="top" colspan="1" rowspan="1" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">OneData</span></td><td valign="top" colspan="1" rowspan="1" width="63" style="word-break: break-all;box-sizing: border-box !important;"><span style="letter-spacing: 0.544px;font-size: 15px;box-sizing: border-box !important;">阿里OneData体系PPT</span></td></tr></tbody></table><p><span style="letter-spacing: 0.544px;"></span></p></section></section></section></section></section></section><p style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;line-height: 25.6px;color: rgb(62, 62, 62);text-align: center;"><span style="font-size: 20px;font-style: italic;text-align: justify;color: rgb(0, 122, 170);">Q:</span><span style="color: rgb(73, 59, 59);"> 关于数据仓库，你还想了解什么？</span></p><p style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;line-height: 25.6px;color: rgb(62, 62, 62);text-align: center;"><span style="color: rgb(73, 59, 59);">欢迎关注我们一起进步</span></p><p style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;line-height: 25.6px;color: rgb(62, 62, 62);text-align: center;"><span style="color: rgb(136, 136, 136);font-size: 14px;">觉得不错，请把这篇文章分享给你的朋友哦</span></p><p style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;line-height: 25.6px;color: rgb(62, 62, 62);text-align: center;"><span style="color: rgb(136, 136, 136);font-size: 14px;">投稿请联系小助手：iom1128『紫霞仙子』</span></p></section></section><section style="letter-spacing: 0.544px;font-size: 16px;font-family: 微软雅黑;"><section><section><section data-role="outer" label="Powered by 135editor.com" style="font-family: -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;"><section data-role="outer" label="Powered by 135editor.com"><section data-tools="135编辑器" data-id="87907"><section style="text-align: center;"><section style="width: 320px;background-image: url("https://mmbiz.qpic.cn/mmbiz_gif/ziadDDQxbCJFA2XMwm7VE1RTcS94u8LqVeqfdxn1PO6F1BpxAGcLibhiaWAhbsTQJ1xol1JV15CNzELATwTkcmVDQ/640?wx_fmt=gif");background-size: contain;background-position: 50% 50%;background-repeat: no-repeat;display: inline-block;"><section style="margin-right: 110px;padding-top: 20px;padding-bottom: 40px;padding-left: 20px;width: 140px;text-align: right;"><img border="0" data-ratio="1" data-type="jpeg" data-w="430" height="" title="微信图片.jpg" width="" data-src="https://mmbiz.qpic.cn/mmbiz_jpg/1OYP1AZw0W1Ne8Cp4hOhvDric3tpj15MCZFC2cEGXspjvQd8vsSKB0YII4TX5cX1S8KiawnbnYibR3SJfBgCSgtFg/640?wx_fmt=jpeg" style="box-sizing: border-box !important; visibility: visible !important; width: 120px !important; height: 120px !important;" _width="120px" class="img_loading" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"><span style="font-size: 14px;text-align: left;letter-spacing: 1.5px;"> </span><em style="letter-spacing: 0.544px;text-align: center;font-family: mp-quote, -apple-system-font, BlinkMacSystemFont, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;color: rgb(160, 160, 160);font-size: 14px;"> </em></section></section></section></section></section></section></section></section></section><p style="letter-spacing: 0.544px;text-align: center;"><em style="color: rgb(160, 160, 160);font-size: 14px;letter-spacing: 0.544px;"><span style="letter-spacing: 0.544px;">！<strong>关注不迷路~ 各种福利、资源定期分享</strong>！</span></em></p><p style="letter-spacing: 0.544px;text-align: center;"><br></p><section powered-by="xiumi.us" style="letter-spacing: 0.544px;"><section><section><pre style="font-size: inherit;color: inherit;line-height: inherit;"><section data-tools="135编辑器" data-id="94250" style="letter-spacing: 0.544px;white-space: normal;border-width: 0px;border-style: none;border-color: initial;"><section data-tools="135编辑器" data-id="91842" style="border-width: 0px;border-style: none;border-color: initial;"><section style="text-align: right;width: auto;"><section style="display: inline-block;clear: both;"><section data-brushtype="text" style="padding: 18px 15px 20px 10px;color: rgb(86, 146, 214);text-align: center;letter-spacing: 1.5px;background-image: url("https://mmbiz.qpic.cn/mmbiz_png/Pn4Sm0RsAuhpplm16ibb8iaib7RoGQ5iaHEdy66AHd7QqL7A2s5icSBE0aw4iaKOKPnXGYxQPhG7VMpbbYV6VJprSh7w/640?wx_fmt=png");background-size: 100% 100%;background-repeat: no-repeat;"><section style="display: flex;justify-content: center;align-items: center;"><section style="margin-left: 2px;width: 20px;"><img data-ratio="0.8936170212765957" data-type="png" data-w="47" data-src="https://mmbiz.qpic.cn/mmbiz_png/Pn4Sm0RsAuhpplm16ibb8iaib7RoGQ5iaHEdvAd0o9e1LlUGA2k0Yib222agOxzweXhahA9GuzJcGBg0dA4DzlibxRqw/640?wx_fmt=png" style="margin-bottom: -6px; box-sizing: border-box !important; visibility: visible !important; width: 20px !important; height: 17.8723px !important;" _width="20px" src="https://img-blog.csdnimg.cn/2022010622203353343.gif" crossorigin="anonymous"></section><section data-brushtype="text" style="font-size: 14px;color: rgb(51, 51, 51);"><span style="font-family: 楷体, 楷体_GB2312, SimKai;">[在看、收藏、转发]，真爱三连！</span><strong style="text-align: right;font-size: 15px;letter-spacing: 1px;font-family: -apple-system-font, system-ui, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;"><span style="color: rgb(0, 0, 0);"><strong><strong style="font-family: Avenir, -apple-system-font, 微软雅黑, sans-serif;letter-spacing: 0.544px;white-space: pre-line;color: rgb(51, 104, 206);"></strong></strong></span></strong></section></section></section></section></section></section></section></pre></section></section></section></section></section></section></section>
</div>

if ("" == 1 && document.getElementById('js_content')) {
document.getElementById('js_content').addEventListener("selectstart",function(e){ e.preventDefault(); });
}

(function(){
if (navigator.userAgent.indexOf("WindowsWechat") != -1){
var link = document.createElement('link');
var head = document.getElementsByTagName('head')[0];
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = "//res.wx.qq.com/mmbizwap/zh_CN/htmledition/style/page/appmsg_new/winwx502975.css";
head.appendChild(link);
}
})();
</script>

</div>
<div id="js_tags_preview_toast" class="article-tag__error-tips" style="display: none;">预览时标签不可点</div>

<div id="js_album_keep_read" class="appmsg_card_context album_read_card" style="display: none;">
<div class="album_read_hd weui-flex" id="js_album_directory">
<div class="album_read_source"><span>收录于话题 </span><span>#<span id="js_album_keep_read_title"></span></span></div>
<span class="weui-btn__word-wrp"><span id="js_album_keep_read_size"></span>个<i class="weui_right_arrow"></i></span>
</div>
<div class="album_read_bd weui-flex">
<span class="album_read_nav_item album_read_nav_prev weui-flex__item" id="js_album_prev">
<span class="album_read_nav_inner">
<span class="album_read_nav_btn">上一篇</span>
<span class="album_read_nav_title" id="js_album_keep_read_pre_title"></span>
</span>
</span>
<span class="album_read_nav_item album_read_nav_next weui-flex__item" id="js_album_next">
<span class="album_read_nav_inner">
<span class="album_read_nav_btn">下一篇</span>
<span class="album_read_nav_title" id="js_album_keep_read_next_title"></span>
</span>
</span>
</div>
</div>

<span style="display:none;" class="media_tool_meta meta_extra meta_like js_media_tool_meta" id="like3">

<button class="sns_opr_btn sns_like_btn" id="js_like_btn"><span class="sns_opr_gap" id="js_like_wording">在看</span><span class="sns_opr_num sns_opr_gap" id="likeNum3"></span></button>
</span>
</div>
</div>

<div class="like_comment_wrp" id="js_like_comment" style="display: none;">
<div class="like_comment_inner">
<div class="like_comment_hd" style="display:none" id="js_like_title"></div>
<div class="like_comment_bd">
<div class="like_comment_tips" id="js_comment_area">
<i class="weui-icon-success"></i><i class="icon-success-primary"></i>已同步到看一看<a href="javascript:;" class="like_comment_share_link" id="js_like_comment_share">写下你的想法</a>
</div>
<div class="like_comment_extra_info" id="js_like_educate" style="display: none">
<p class="like_comment_extra_tips">
<span id="js_friend_like_area" style="display: none"></span>
<span id="js_friend_like_word">前往“发现”-“看一看”浏览“朋友在看”</span>
</p>
<p class="like_comment_pic_wrp">
<img class="like_comment_pic" src="//res.wx.qq.com/mmbizwap/zh_CN/htmledition/images/pic/appmsg/pic_like_comment52ab86.png">
</p>
<button class="weui-btn weui-btn_primary" id="js_go_wow">前往看一看</button>
</div>
</div>
</div>
</div>
<div style="display:none;" id="wow_close_inform">
<div class="weui-mask"></div>
<div class="weui-dialog">
<div class="weui-dialog__hd"><strong class="weui-dialog__title">看一看入口已关闭</strong></div>
<div class="weui-dialog__bd">
在“设置”-“通用”-“发现页管理”打开“看一看”入口 </div>
<div class="weui-dialog__ft" id="wow_close_ack">
<a href="javascript:;" class="weui-dialog__btn weui-dialog__btn_primary">我知道了</a>
</div>
</div>
</div>

<div style="display: none;" id="js_comment_panel">
<div class="like_comment_primary_wrp editing" id="js_comment_wrp">
<div class="like_comment_primary_inner">
<div class="like_comment_primary_hd">
<div class="like_comment_primary_hd_side">
<button class="like_comment_primary_cancel" id="js_comment_cancel">
取消 <i class="weui-icon-close-thin"></i>
</button>
</div>
<h4 class="like_comment_primary_title"> 发送到看一看 </h4>
<div class="like_comment_primary_hd_side">
<button class="like_comment_primary_btn" id="js_comment_confirm" disabled="disabled">发送</button>
</div>
</div>
<div class="like_comment_primary_bd">
<div class="like_comment_media_title" id="js_panel_like_title">
漫谈千亿级数据优化实践：数据倾斜
</div>
<div class="like_comment_textarea_wrp">
<textarea class="like_comment_textarea weui-textarea" placeholder="写下你的想法..." id="js_comment_text"></textarea>
</div>
</div>
<span class="like_comment_msg" id="js_like_comment_msg" style="visibility: hidden;">最多200字，当前共<span id="js_like_current_cnt"></span>字</span>
</div>
</div>
<div class="like_comment_primary_mask" id="js_mask_2"></div>
</div>

</div>
</div>

q_15403863

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
2021-01-06

<div id="page-content" class="rich_media_area_primary"> <div class="rich_media_area_primary_inner"> <div id="img-content" class="rich_media_wrp"> ...
复制链接

扫一扫

2021-01-06

“相关推荐”对你有帮助么？