HQL调优实操1 —— reduce卡住

最新推荐文章于 2022-04-25 22:18:40 发布
最新推荐文章于 2022-04-25 22:18:40 发布
阅读量290 收藏
点赞数
分类专栏： HQL命令的底层执行逻辑解析 文章标签： HQL调优 reducer 慢 卡
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ZZh1301051836/article/details/118391598
版权

1 问题

任务执行缓慢。卡顿日志如下：

2021-07-01 10:19:04 log INFO - Starting Job = job_1625097332819_16902, Tracking URL = http://c2-xxx-hadoopxxx.bj:8088/proxy/application_xxxxxxxx_16902/
2021-07-01 10:19:04 log INFO - Kill Command = /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop job -kill job_1625097332819_16902
2021-07-01 10:19:24 log INFO - Hadoop job information for Stage-1: number of mappers: 810; number of reducers: 200
2021-07-01 10:19:25 log INFO - 2021-07-01 10:19:25,490 Stage-1 map = 0%, reduce = 0%
2021-07-01 10:19:48 log INFO - 2021-07-01 10:19:48,356 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 83.01 sec
2021-07-01 10:19:49 log INFO - 2021-07-01 10:19:49,402 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 174.33 sec
2021-07-01 10:19:50 log INFO - 2021-07-01 10:19:50,451 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 221.48 sec
2021-07-01 10:19:51 log INFO - 2021-07-01 10:19:51,488 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 335.56 sec
2021-07-01 10:19:52 log INFO - 2021-07-01 10:19:52,527 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 531.72 sec
2021-07-01 10:19:53 log INFO - 2021-07-01 10:19:53,582 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 809.4 sec
2021-07-01 10:19:54 log INFO - 2021-07-01 10:19:54,651 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 1198.47 sec
2021-07-01 10:19:55 log INFO - 2021-07-01 10:19:55,691 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 1534.1 sec
2021-07-01 10:19:56 log INFO - 2021-07-01 10:19:56,733 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 1762.57 sec
2021-07-01 10:19:57 log INFO - 2021-07-01 10:19:57,777 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 2180.07 sec
2021-07-01 10:19:58 log INFO - 2021-07-01 10:19:58,816 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 2598.88 sec
2021-07-01 10:19:59 log INFO - 2021-07-01 10:19:59,857 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 2949.73 sec
2021-07-01 10:20:00 log INFO - 2021-07-01 10:20:00,905 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 3638.16 sec
2021-07-01 10:20:01 log INFO - 2021-07-01 10:20:01,947 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 4049.69 sec
2021-07-01 10:20:02 log INFO - 2021-07-01 10:20:02,986 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 4272.81 sec
2021-07-01 10:20:04 log INFO - 2021-07-01 10:20:04,035 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 4553.64 sec
2021-07-01 10:20:05 log INFO - 2021-07-01 10:20:05,085 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 4941.65 sec
2021-07-01 10:20:06 log INFO - 2021-07-01 10:20:06,124 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 5443.33 sec
2021-07-01 10:20:07 log INFO - 2021-07-01 10:20:07,169 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 6004.05 sec
2021-07-01 10:20:08 log INFO - 2021-07-01 10:20:08,210 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 6370.51 sec
2021-07-01 10:20:09 log INFO - 2021-07-01 10:20:09,249 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 6869.53 sec
2021-07-01 10:20:10 log INFO - 2021-07-01 10:20:10,288 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 7293.59 sec
2021-07-01 10:20:11 log INFO - 2021-07-01 10:20:11,329 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 7739.03 sec
2021-07-01 10:20:12 log INFO - 2021-07-01 10:20:12,372 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 8230.34 sec
2021-07-01 10:20:13 log INFO - 2021-07-01 10:20:13,415 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 8593.03 sec
2021-07-01 10:20:14 log INFO - 2021-07-01 10:20:14,455 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 8884.0 sec
2021-07-01 10:20:15 log INFO - 2021-07-01 10:20:15,495 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 9217.56 sec
2021-07-01 10:20:17 log INFO - 2021-07-01 10:20:17,576 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 9502.69 sec
2021-07-01 10:20:19 log INFO - 2021-07-01 10:20:19,666 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 9907.57 sec
2021-07-01 10:20:22 log INFO - 2021-07-01 10:20:22,809 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 10226.92 sec
2021-07-01 10:20:27 log INFO - 2021-07-01 10:20:27,016 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 10641.08 sec
2021-07-01 10:20:34 log INFO - 2021-07-01 10:20:34,631 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 11264.2 sec
2021-07-01 10:20:38 log INFO - 2021-07-01 10:20:38,800 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 11534.41 sec
2021-07-01 10:20:46 log INFO - 2021-07-01 10:20:46,099 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 11949.88 sec
2021-07-01 10:20:49 log INFO - 2021-07-01 10:20:49,223 Stage-1 map = 84%, reduce = 4%, Cumulative CPU 12164.8 sec
2021-07-01 10:20:50 log INFO - 2021-07-01 10:20:50,266 Stage-1 map = 84%, reduce = 9%, Cumulative CPU 12301.65 sec
2021-07-01 10:20:51 log INFO - 2021-07-01 10:20:51,306 Stage-1 map = 84%, reduce = 14%, Cumulative CPU 12450.18 sec
2021-07-01 10:20:52 log INFO - 2021-07-01 10:20:52,351 Stage-1 map = 84%, reduce = 18%, Cumulative CPU 12578.82 sec
2021-07-01 10:20:53 log INFO - 2021-07-01 10:20:53,391 Stage-1 map = 85%, reduce = 22%, Cumulative CPU 12710.59 sec
2021-07-01 10:20:54 log INFO - 2021-07-01 10:20:54,435 Stage-1 map = 85%, reduce = 25%, Cumulative CPU 12802.93 sec
2021-07-01 10:20:55 log INFO - 2021-07-01 10:20:55,486 Stage-1 map = 85%, reduce = 27%, Cumulative CPU 12897.56 sec
2021-07-01 10:20:59 log INFO - 2021-07-01 10:20:59,648 Stage-1 map = 85%, reduce = 28%, Cumulative CPU 13117.54 sec
2021-07-01 10:21:04 log INFO - 2021-07-01 10:21:04,854 Stage-1 map = 86%, reduce = 28%, Cumulative CPU 13317.26 sec
2021-07-01 10:21:19 log INFO - 2021-07-01 10:21:19,433 Stage-1 map = 86%, reduce = 29%, Cumulative CPU 13803.17 sec
2021-07-01 10:21:48 log INFO - 2021-07-01 10:21:48,576 Stage-1 map = 87%, reduce = 29%, Cumulative CPU 14941.58 sec
2021-07-01 10:22:48 log INFO - 2021-07-01 10:22:48,970 Stage-1 map = 87%, reduce = 29%, Cumulative CPU 16916.36 sec
2021-07-01 10:23:16 log INFO - 2021-07-01 10:23:16,107 Stage-1 map = 88%, reduce = 29%, Cumulative CPU 17681.74 sec
2021-07-01 10:24:16 log INFO - 2021-07-01 10:24:16,702 Stage-1 map = 88%, reduce = 29%, Cumulative CPU 19030.01 sec
2021-07-01 10:24:39 log INFO - 2021-07-01 10:24:39,607 Stage-1 map = 89%, reduce = 29%, Cumulative CPU 19543.91 sec
2021-07-01 10:25:05 log INFO - 2021-07-01 10:25:05,628 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 20126.07 sec
2021-07-01 10:26:06 log INFO - 2021-07-01 10:26:06,084 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 21018.0 sec
2021-07-01 10:27:06 log INFO - 2021-07-01 10:27:06,474 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 21879.94 sec
2021-07-01 10:28:07 log INFO - 2021-07-01 10:28:07,046 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 22815.09 sec
2021-07-01 10:29:07 log INFO - 2021-07-01 10:29:07,398 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 23958.32 sec
2021-07-01 10:30:07 log INFO - 2021-07-01 10:30:07,695 Stage-1 map = 89%, reduce = 30%, Cumulative CPU 25010.02 sec
2021-07-01 10:30:11 log INFO - 2021-07-01 10:30:11,860 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 25097.77 sec
2021-07-01 10:31:12 log INFO - 2021-07-01 10:31:12,237 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 26114.11 sec
2021-07-01 10:32:12 log INFO - 2021-07-01 10:32:12,518 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 27115.18 sec
2021-07-01 10:33:12 log INFO - 2021-07-01 10:33:12,871 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 28257.23 sec
2021-07-01 10:34:13 log INFO - 2021-07-01 10:34:13,258 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 29457.73 sec
2021-07-01 10:35:13 log INFO - 2021-07-01 10:35:13,700 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 30644.47 sec
2021-07-01 10:36:13 log INFO - 2021-07-01 10:36:13,863 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 31783.64 sec
2021-07-01 10:37:14 log INFO - 2021-07-01 10:37:14,015 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 32519.44 sec
2021-07-01 10:38:14 log INFO - 2021-07-01 10:38:14,147 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 33398.86 sec
2021-07-01 10:39:14 log INFO - 2021-07-01 10:39:14,256 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 34157.51 sec
2021-07-01 10:40:14 log INFO - 2021-07-01 10:40:14,463 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 35916.48 sec
2021-07-01 10:41:14 log INFO - 2021-07-01 10:41:14,682 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 38235.93 sec
2021-07-01 10:42:14 log INFO - 2021-07-01 10:42:14,796 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 40046.69 sec
2021-07-01 10:43:15 log INFO - 2021-07-01 10:43:15,041 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 40877.78 sec
2021-07-01 10:44:15 log INFO - 2021-07-01 10:44:15,070 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 41510.4 sec
2021-07-01 10:45:15 log INFO - 2021-07-01 10:45:15,312 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 42102.11 sec
2021-07-01 10:46:15 log INFO - 2021-07-01 10:46:15,404 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 42606.47 sec
2021-07-01 10:47:15 log INFO - 2021-07-01 10:47:15,506 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 43235.72 sec
2021-07-01 10:48:15 log INFO - 2021-07-01 10:48:15,632 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 44009.25 sec
2021-07-01 10:49:15 log INFO - 2021-07-01 10:49:15,718 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 44663.71 sec
2021-07-01 10:50:15 log INFO - 2021-07-01 10:50:15,798 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 45339.24 sec
2021-07-01 10:51:15 log INFO - 2021-07-01 10:51:15,925 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 46219.58 sec
2021-07-01 10:52:01 log INFO - 2021-07-01 10:52:01,519 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 47109.44 sec
2021-07-01 10:53:01 log INFO - 2021-07-01 10:53:01,753 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 48380.21 sec
2021-07-01 10:54:01 log INFO - 2021-07-01 10:54:01,920 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 49759.68 sec
2021-07-01 10:55:01 log INFO - 2021-07-01 10:55:01,995 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 50736.85 sec
2021-07-01 10:56:02 log INFO - 2021-07-01 10:56:02,004 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 51564.75 sec
2021-07-01 10:56:35 log INFO - 2021-07-01 10:56:35,153 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 52149.04 sec
2021-07-01 10:57:35 log INFO - 2021-07-01 10:57:35,269 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 52934.9 sec
2021-07-01 10:58:35 log INFO - 2021-07-01 10:58:35,426 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 54443.17 sec
2021-07-01 10:59:35 log INFO - 2021-07-01 10:59:35,620 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 56427.32 sec
2021-07-01 11:00:35 log INFO - 2021-07-01 11:00:35,780 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 58277.61 sec
2021-07-01 11:00:36 log INFO - 2021-07-01 11:00:36,816 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 58306.46 sec
2021-07-01 11:01:36 log INFO - 2021-07-01 11:01:36,994 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 59500.57 sec
2021-07-01 11:02:37 log INFO - 2021-07-01 11:02:37,186 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 60221.41 sec
2021-07-01 11:03:37 log INFO - 2021-07-01 11:03:37,307 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 60730.51 sec
2021-07-01 11:04:37 log INFO - 2021-07-01 11:04:37,494 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 61053.13 sec
2021-07-01 11:05:19 log INFO - 2021-07-01 11:05:19,068 Stage-1 map = 94%, reduce = 30%, Cumulative CPU 61357.76 sec
2021-07-01 11:06:19 log INFO - 2021-07-01 11:06:19,283 Stage-1 map = 94%, reduce = 30%, Cumulative CPU 61765.52 sec
2021-07-01 11:07:03 log INFO - 2021-07-01 11:07:03,927 Stage-1 map = 95%, reduce = 30%, Cumulative CPU 62209.08 sec
2021-07-01 11:07:04 log INFO - 2021-07-01 11:07:04,966 Stage-1 map = 95%, reduce = 31%, Cumulative CPU 62214.65 sec
2021-07-01 11:08:05 log INFO - 2021-07-01 11:08:05,107 Stage-1 map = 95%, reduce = 31%, Cumulative CPU 62675.88 sec
2021-07-01 11:08:33 log INFO - 2021-07-01 11:08:33,193 Stage-1 map = 96%, reduce = 31%, Cumulative CPU 62905.03 sec
2021-07-01 11:09:16 log INFO - 2021-07-01 11:09:16,730 Stage-1 map = 96%, reduce = 32%, Cumulative CPU 63276.93 sec
2021-07-01 11:09:21 log INFO - 2021-07-01 11:09:21,906 Stage-1 map = 97%, reduce = 32%, Cumulative CPU 63305.22 sec
2021-07-01 11:10:07 log INFO - 2021-07-01 11:10:07,499 Stage-1 map = 98%, reduce = 32%, Cumulative CPU 63655.26 sec
2021-07-01 11:11:07 log INFO - 2021-07-01 11:11:07,681 Stage-1 map = 98%, reduce = 32%, Cumulative CPU 64020.02 sec
2021-07-01 11:11:13 log INFO - 2021-07-01 11:11:13,916 Stage-1 map = 98%, reduce = 33%, Cumulative CPU 64044.0 sec
2021-07-01 11:11:28 log INFO - 2021-07-01 11:11:28,456 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 64116.01 sec
2021-07-01 11:12:28 log INFO - 2021-07-01 11:12:28,651 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 64356.53 sec
2021-07-01 11:13:28 log INFO - 2021-07-01 11:13:28,772 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 64584.49 sec
2021-07-01 11:14:28 log INFO - 2021-07-01 11:14:28,914 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 64822.4 sec
2021-07-01 11:15:29 log INFO - 2021-07-01 11:15:29,062 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 65080.55 sec
2021-07-01 11:16:29 log INFO - 2021-07-01 11:16:29,172 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 65326.9 sec
2021-07-01 11:17:29 log INFO - 2021-07-01 11:17:29,215 Stage-1 map = 99%, reduce = 33%, Cumulative CPU 65529.33 sec

2 资源管理界面查看详情

点击链接（此处已做处理，打不开哟）：http://c2-xxx-hadoopxxx.bj:8088/proxy/application_xxxxxxxx_16902/，打开后如下图所示，依次找到正在执行的reduce tasks。
在这里插入图片描述
在这里插入图片描述
发现，所有reduce tasks的执行时间都一样，一样的慢。而reduce们都卡在了copy阶段，也就是从map拉取数据到reduce的阶段。

3 思考过程

如下为个人顺序思考过程，有不合适、不正确的地方还请务必留言指导，小编必非常感谢。

  1. 所有reduce tasks的执行时间都一样，并不是有的很短，有的很长，排除key不均匀导致的数据倾斜问题。

  2. 卡顿发生在copy阶段，猜测可能是map结果的数据量太大，导致的拉取数据缓慢。
    map结果数据量太大，猜测关联键中出现空值或者出现1对多的情况。
    回到原始HQL代码，发现结果为用户明细数据。并且,关联键无空值，无1对多的情况，所以数据量虽大，但问题并不出自代码。

select uid,
	...
from
(
	select uid
	...
	group by uid
)t1
left join
(
	select uid
	...
	group by uid
)
t2 on t1.uid = t2.uid
...

3.既然是单纯的map结果数据量大，所以reducer从map拉取数据较慢。猜测reducer拉取数据时与未结束的map竞争资源 ，导致死锁。
参考网址：MapReduce任务运行慢问题排查

在mapreduce流程中，reduce启动后需要先copy map的相关shuffle数据，全部搜集完所有map的数据后才能处理。
hadoop为了加快mapreduce job的整体速度，会提前启动一些reduce，先去拉取那些已经完成的map的数据。
但这样会存在隐患，即资源有限时，reduce占用了map的资源，还在等待map的数据，map又在等待资源以运行任务，这就形成了一个死锁。在发生死锁一定时间后，hadoop就会kill掉那个reduce task。

这种情况可以通过调整参数推迟reduce task的执行：
set mapreduce.job.reduce.slowstart.completedmaps = 0.95; – 有95%的map task执行完后，reduce task才能开始copy数据。防止map和reduce竞争资源，导致死锁
mapreduce.job.reduce.slowstart.completedmaps：默认是0.05，即只要有5%的map task执行完，reduce task就开始copy数据。小编的程序中，因为reduce拉取总数据量达到38亿，所以小编干脆调整到了0.95，即map差不多执行完后，reduce再拉取数据。该程序是离线计算指标，计算时间长还可以承受。

张之海
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
HQL调优
05-08 146
1.sql优化2.参数优化3.文件类型。
hadoop 异常 reduce长时间卡住停滞不前的问题
923723914
12-10 1244
单个节点的命名问题也会导致这个问题，具体解决办法： http://blog.csdn.net/lxpbs8851/article/details/11820601 磁盘损坏的解决办法： 起因： datanode的磁盘挂掉了2个 昨天的任务跑完了之后，检查了下dfsadmin -report 监测到集群之中有个节点的 Configured Capacity 比其它的节点少了差不多...
HQL语句的调优
li1579026891的博客
12-12 829
1、去重语句(用group by 来代替distinct) Group by 也有去重的功能，具体内容如下 select distinct customer_id from test_join_order; => select customer_id from test_join_order group by customer_id; 在极大的数据量(且很多重复值)时，可以先group by去重，在count()计数，效率高于count(distinct col) create tabl
Hadoop开发遇到的问题之reduce卡住
weixin_33912638的博客
08-26 230
遇到的问题描述：在hadoop上面执行程序，程序运行之后能够正常执行。一切似乎都是正常的，然而过了一段时间之后程序便开始阻塞直到程序超时退出（如下）。 14/08/19 21:17:51 INFO mapred.JobClient: map 99% reduce 71% 14/08/19 21:17:54 INFO mapred.JobClient: map 99% reduce 75% ...
hql性能优化
jiedaodezhuti的博客
07-07 831
性能低的来源 hive性能优化时，把HiveQL当做M/R程序来读，即从M/R的运行角度来考虑优化性能，从更底层思考如何优化运算性能，而不仅仅局限于逻辑代码的替换层面。 Hadoop处理数据的显著特征： 1.数据的大规模并不是负载重点，造成运行压力过大原因是数据倾斜; 2.jobs数比较多也是作业运行效率低的原因之一，如一个几百行的表，多次关联对此汇总，产生几十个jobs，将需要花费大量时间且大部分时间用于作业分配，初始化和数据输出。M/R作业初始化的时间是比较耗时间资源的一个部分; 3.在使用SUM，CO
Hibernate-HQL.rar_HQL_hibernate hql
09-23
1. 合理使用HQL与SQL：对于复杂的数据库操作，可能需要结合使用HQL和原生SQL。 2. 避免在HQL中使用SQL函数：尽量用Hibernate提供的函数，以确保跨数据库兼容性。 3. 优化查询性能：合理设计数据库索引，避免全表扫描...
hibernate_hql.rar_HQL
09-21
《深入理解Hibernate HQL：Java数据库交互利器》 Hibernate，作为一款强大的对象关系映射（ORM）框架，极大地简化了Java开发者与数据库之间的交互。其中，Hibernate查询语言（HQL）是其核心特性之一，它为开发者...
hibernateのHQL
01-19
1、HQL是面向对象的，所以HQL与数据库无关 2、HQL最终也会生成SQL,也就是说 HQL使用范围是<SQL的范围 3、HQL是由query对象获得。 hql sql 面向对象 面向结构(表) 操作的是实体类 ，类属性 表，表列段 ...
Spring--HQL-chaxun.zip_HQL
09-19
1. **HQL的基本概念**： HQL允许开发者用类和属性名称来代替表和列名，这使得代码更具可读性和可维护性。它支持单表查询、多表关联查询、条件查询、分组与排序等操作，同时还支持函数调用和嵌套查询。 2. **HQL...
HQL常用函数大全.docx
06-10
适合大数据初学者，数据分析行业工作者登需要使用hive SQL的人使用
reduce运行到69%就卡住了。。。。肿么办？
zoe_ranxiaosu的博客
01-03 968
19/01/03 14:17:22 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 19/01/03 14:17:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTrac...
hql 优化
kayla的博客
10-13 4390
优化时，把hql当做map reduce程序来读，会有意想不到的惊喜。 理解hadoop的核心能力，是hive优化的根本。这是这一年来，项目组所有成员宝贵的经验总结。  对分区插入数据(每次都会覆盖掉原来的数据): hive.map.aggr=true Map 端部分聚合，相当于Combiner hive.groupby.skewindata=true 有数据倾斜的时候进行负载均衡，当...
错误的配置使reduce tasks一直处于Pending状态
weixin_30260399的博客
03-06 290
正确设置Hadoop程序运行时的参数可以提高程序的运行效率，但是错误的设置也会带来效率的降低，甚至程序的失败。我今天就遇到了这种情况，事情是这样的： 在Hadoop的配置中有这样一个参数： mapred.reduce.slowstart.completed.maps: 这个参数控制reducer何时开始运行，也就是何时开始将mapper的输出拷贝到reducer本地。通常情况下，reducer...
HQL查询优化
我们始终是路人
03-03 442
将HQL语句放到XML文件中 XML文件如下：<query name="名字">HQL语句</query>JAVA代码Query q =session.getNamedQuery("名字");
Hive-HQL语句调优
独自优秀的数据分析师
04-25 1892
一、除去多余操作 比如获取每个员工最高薪水记录 方案一： selecet * from a inner join(select user_id,max(salary) as max_salary from a group by user_id) b on a.user_id = b.user_id and a.salary = b.max_salary 方案二：减少了join操作，性能提升 select * from (select *,row_number()over(partition by
hive卡在最后一个reduce过不去_数据倾斜
maomao_dreamhigh的博客
12-15 3896
sql语句 insert overwrite table jk_ld_xyxj_1214_25002 partition (stat_month='202110') select a.msisdn, a.25002_area_type, a.longitude, a.latitude from (select msisdn,25002_area_type,longitude,latitude,row_number() over (partition by msisdn order by longitude
记一次HQL优化过程
optimus
08-25 328
记一次相对较大的“数据量”尿崩问题的解决过程~~ 故事背景 故事背景是这样子的，小伙伴写了个狗屎一样的HQL，数据量大了之后直接被压挂了，当我看到他的SQL的一刹那，我觉得我可以拯救下这个可怜的孩子，从此踏上了一条慢慢调参路！ 首先，SQL的大致逻辑是，表A关联客户信息表B两次。简单表述SQL如下： A left join B on a.id_1 = b.id left join B on a.id_2 = b.id 他原来的写法就不恢复了，脑回路太清奇，原谅我想不起来他原来是怎么写的。 出发 好了，我们
Hibernate查询优化 HQL提高效率
热门推荐
xiaoli
04-11 1万+
1、Hibernate查询性能优化 1、1
Hibernate性能调优
LJBOO的专栏
06-28 706
1 Hibernate 连接池    hibernate自带一个连接池，但是这个连接池的性能不好，BUG也比较多，所以hibernate借助第三方的连接池来配置。通常用的比较多的是prxool。目前在j2ee中最通用的框架ssh中，hibernate的配置如下：  在spring 的applicationContext.xm.配置文件里配置数据源连接信息: <!-- 配置数据源 -->
hql转化为map reduce
最新发布
05-05
HQL（Hive Query Language）是Hive中的一种查询语言，它可以将类SQL语句转换为MapReduce任务。因此，将HQL语句转换为MapReduce任务是可能的，但需要一些额外的工作。 以下是将HQL转换为MapReduce的一般步骤： 1. 确定输入和输出：确定输入和输出的格式和位置。这些信息通常在Hive表中定义，因此需要查看表定义。 2. 确定数据操作：查看HQL查询，确定需要执行哪些数据操作，例如过滤，排序，聚合等。 3. 将操作转换为MapReduce任务：使用Hadoop MapReduce API编写代码来执行每个操作。这通常涉及到Mapper和Reducer类的编写。Mapper类将输入数据转换为键值对，Reducer类对Mapper的输出进行处理。 4. 将查询转换为MapReduce作业：将上述步骤组合成一个完整的MapReduce作业。 以下是一个简单的示例，将HQL查询转换为MapReduce作业： 假设有一个名为“users”的表，其中包含以下列：id，name，age和gender。现在想要计算所有男性用户的平均年龄。 HQL查询： SELECT AVG(age) FROM users WHERE gender='male'; 将查询转换为MapReduce作业： 1. 确定输入和输出：输入数据为Hive表“users”，输出数据为一个单独的数字。 2. 确定数据操作：需要过滤所有性别为男性的用户，并计算平均年龄。 3. 将操作转换为MapReduce任务：编写一个Mapper类，将输入数据转换为键值对，并过滤出性别为男性的用户。编写一个Reducer类，对Mapper输出的年龄值进行求和，并计算平均值。 4. 将查询转换为MapReduce作业：将Mapper和Reducer类组合起来，使用Hadoop MapReduce API创建一个作业，并将它们与输入和输出连接起来。 请注意，以上步骤是一个简单示例，实际情况可能更加复杂。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

张之海

若有帮助，客官打赏一分吧

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值