HIVE向量化执行引擎优化功能的bug解决

最新推荐文章于 2024-09-20 15:12:59 发布

小陈Chen

最新推荐文章于 2024-09-20 15:12:59 发布

阅读量1k

点赞数

本文链接：https://blog.csdn.net/weixin_46323688/article/details/108343310

版权

在项目中，HIVE的向量化执行引擎在数据从dwd层到dws层聚合时出现数据无法从integer转byte数组的错误。经过排查，发现是HIVE的一个优化功能bug。通过优化脚本并过滤数据，问题依然存在。最终，通过在代码中添加特定行以规避该BUG，成功解决了数据写入问题，实现了正常的数据流转。

摘要由CSDN通过智能技术生成

项目场景：

项目任务调度 dwd层事件明细表-->dws层流量聚合表脚本开发初期使用时一切正常 *脚本测试代码如下:*

<span style="color:#000000"><code class="language-c">export HIVE_HOME<span style="color:#669900">=</span><span style="color:#669900">/</span>opt<span style="color:#669900">/</span>apps<span style="color:#669900">/</span>apache<span style="color:#669900">-</span>hive<span style="color:#669900">-</span><span style="color:#98c379">3.1</span><span style="color:#98c379">.2</span><span style="color:#669900">-</span>bin<span style="color:#669900">/</span>

datestr<span style="color:#669900">=</span>`date <span style="color:#669900">-</span>d<span style="color:#669900">'-1 day'</span> <span style="color:#669900">+</span><span style="color:#669900">%</span>Y<span style="color:#669900">-</span><span style="color:#669900">%</span>m<span style="color:#669900">-</span><span style="color:#669900">%</span>d`

<span style="color:#c678dd">if</span> <span style="color:#999999">[</span> $<span style="color:#98c379">1</span> <span style="color:#999999">]</span>
then
datestr<span style="color:#669900">=</span>$<span style="color:#98c379">1</span>
fi

$<span style="color:#999999">{</span>HIVE_HOME<span style="color:#999999">}</span><span style="color:#669900">/</span>bin<span style="color:#669900">/</span>hive <span style="color:#669900">-</span>e "

INSERT INTO TABLE dws<span style="color:#999999">.</span>app_pv_agg_session <span style="color:#61aeee">PARTITION</span><span style="color:#999999">(</span>dt<span style="color:#669900">=</span><span style="color:#669900">'${datestr}'</span><span style="color:#999999">)</span>
SELECT
    guid<span style="color:#999999">,</span>
    sessionid as session_id<span style="color:#999999">,</span>
    <span style="color:#61aeee">min</span><span style="color:#999999">(</span>ts<span style="color:#999999">)</span> as start_time<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>ts<span style="color:#999999">)</span> as end_time<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>first_page<span style="color:#999999">)</span> as first_page<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>last_page<span style="color:#999999">)</span> as last_page<span style="color:#999999">,</span>
    <span style="color:#61aeee">count</span><span style="color:#999999">(</span><span style="color:#98c379">1</span><span style="color:#999999">)</span> as pv_cnt<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>isnew<span style="color:#999999">)</span> as isnew<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>hour<span style="color:#999999">)</span> as hour<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>province<span style="color:#999999">)</span> as province<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>city<span style="color:#999999">)</span> as city<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>district<span style="color:#999999">)</span> as district<span style="color:#999999">,</span>
    <span style="color:#61aeee">max</span><span style="color:#999999">(</span>devicetype<span style="color:#999999">)</span> as device_type
FROM
    <span style="color:#999999">(</span>
        SELECT
            guid<span