Kylin查询遇到的问题

博客讲述了在使用Kylin进行大数据查询时遇到的问题,即由于扫描大量数据导致HBase的协处理器内存压力过大。解决方法是调整HBase协处理器的内存设置,从而避免此类问题。
摘要由CSDN通过智能技术生成

这个问题是直接扫描出来一千多万调数据,导致hbas的协处理器扛不住了,把协处理器的内存调高就可以了

2019-06-11 12:09:32,892 ERROR [pool-11-thread-627] v2.CubeHBaseEndpointRPC:426 : <sub-thread for Query 22f6f45c-4d00-d3ba-c693-674b19f2d2fa GTScanRequest 3b1c138e>Error when visiting cubes by endpoint
org.apache.kylin.common.exceptions.KylinTimeoutException: coprocessor timeout after scanning 15170300 rows
	at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.getCoprocessorException(CubeHBaseEndpointRPC.java:505)
	at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$300(CubeHBaseEndpointRPC.java:77)
	at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$3.update(CubeHBaseEndpointRPC.java:355)
	at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$3.update(CubeHBaseEndpointRPC.java:323)
	at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1810)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2019-06-11 12:09:32,935 ERROR [Query 22f6f45c-4d00-d3ba-c693-674b19f2d2fa-4296] service.QueryService:510 : Exception while executing query
java.sql.SQLException: Error while executing SQL "SELECT "s598b2_0"."周" AS "f1", "s598b2_0"."时长分类" AS "f2", count("s598b2_0"."用户ID") AS "f0" FROM (select 
case 
 when aa.pjsc < 30 then '0.5小时以下' 
 when aa.pjsc >= 30  and aa.pjsc < 60  then '0.5到1小时' 
 when aa.pjsc >= 60  and aa.pjsc < 120  then '1到2小时' 
 when aa.pjsc >= 120 and aa.pjsc < 180  then '2到3小时' 
else '3小时以上'
end 时长分类,
userid 用户id,
ljsc 累计时长,
dtx 周,
country 国家,
province 省 ,
city 市,
vendor 信息商,
broker 券商,
product 产品,
home 机房,
operator 运营商
from 
(select
  country ,
  province  ,
  city ,
  vendor ,
  broker ,
  product ,
  home ,
  operator ,
  week(cast(dt as DATE)) dtx ,
  userid ,
  sum(duration) ljsc,
  sum(duration)/count(userid) pjsc,
  count(userid) 
  from
  sse_AAUDIT_ONLINE
  where trade = 1 and CAST(quarter AS varchar) =CAST(YEAR(CURRENT_DATE) AS varchar)+'0'+CAST(QUARTER(CURRENT_DATE) AS varchar) 
 group BY
  country,
  province,
  city,
  vendor,
  broker,
  product,
  home,
  operator,
  week(cast(dt as DATE)),
  userid) aa) "s598b2_0" WHERE "s598b2_0"."券商"='富贵证券' GROUP BY "s598b2_0"."周", "s598b2_0"."时长分类"": coprocessor timeout after scanning 15170300 rows
	at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
	at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
	at org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:938)
	at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:641)
	at org.apache.kylin.rest.service.QueryService.query(QueryService.java:208)
	at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:468)
	at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:429)
	at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:367)
	at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
	at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHan
  • 1
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 6
    评论
评论 6
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值