ERROR streaming.StreamExecution: Query topic_lcc [id =xx, runId=xx] terminated with error

204 篇文章 480 订阅 ¥49.90 ¥99.00
在将工作从开发测试集群转移到测试集群后,Spark Structured SQL任务遇到错误。尽管代码在原始环境中运行正常,但在测试环境中启动时出现错误。报错信息为'ERROR streaming.StreamExecution: Query topic_lcc [id =xx, runId=xx] terminated with error'。错误可能与不同环境的配置或依赖有关。" 122474374,9344249,深入理解JavaScript箭头函数,"['javascript', '前端开发', '开发语言']
摘要由CSDN通过智能技术生成

1.背景

在这里插入图片描述
spark structured sql 任务启动报错,这个是在一个开发测试集群上是好的,但是移交到测试集群,测试人员进行测试的时候报错。查看程序,感觉没什么问题。

错误如下

19/08/01 17:36:52 WARN consumer.ConsumerConfig: The configuration max.poll.records = 1 was supplied but isn't a known config.
19/08/01 17:36:52 ERROR streaming.StreamExecution: Query topic_lcc [id = fff32e45-d354-4a4f-af2c-106fa9801bd0, runId = 562ca67e-8acd-40eb-9a92-e88a3ed99266] terminated with error
	at org.apache.spark.sql.kafka010.SubscribeStrategy.createConsumer(ConsumerStrategy.scala:63)
	at org.apache.spark.sql.kafka010.KafkaOffsetReader.createConsumer(KafkaOffsetReader.scala:314)
	at
set user_write_ugi:afs://kunpeng.afs.baidu.com:9902=lbs-huiyan,lbs-huiyan; insert overwrite directory 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day={FORWARD_MONTH_END}' using CSV options('compression'='GZIP', sep='\t', escapeQuotes=false) SELECT source.mall_id AS mall_id, COUNT(1) AS total FROM (SELECT cuid, mall_id, mall_name, date_type FROM huiyan_ns.huiyan.map_huiyan_mall_basic_source WHERE event_day = '{FORWARD_MONTH_END}') source JOIN (SELECT cuid, * FROM huiyan_ns.huiyan.map_huiyan_parse_userpic WHERE event_day >= '{FORWARD_7_DAY}' AND event_day <= '{FORWARD_MONTH_END}') userpic ON userpic.cuid = source.cuid GROUP BY source.mall_id, source.mall_name, source.date_type; alter table huiyan_ns.huiyan.map_huiyan_mall_userpic_raw add partition(event_day='{FORWARD_MONTH_END}') location 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_flow_userpic_raw/event_day={FORWARD_MONTH_END}'; set user_write_ugi:afs://kunpeng.afs.baidu.com:9902=lbs-huiyan,lbs-huiyan; insert overwrite directory 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day={FORWARD_MONTH_END}' using CSV options('compression'='GZIP', sep='\t', escapeQuotes=false) SELECT TRANSFORM(mall_id, ) USING 'python3 mall_userpic_streaming.py' AS(mall_id, ) FROM huiyan_ns.huiyan.map_huiyan_mall_userpic_raw WHERE event_day = '{FORWARD_MONTH_END}' ; alter table huiyan_ns.huiyan.map_huiyan_mall_userpic add partition(event_type='monthly', event_day='{FORWARD_MONTH_END}') location 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day=${FORWARD_MONTH_END}'; 将这两段代码合并为一段
最新发布
07-22
你可以将这两段代码合并为以下形式: ```sql set user_write_ugi:afs://kunpeng.afs.baidu.com:9902=lbs-huiyan,lbs-huiyan; insert overwrite directory 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day={FORWARD_MONTH_END}' using CSV options('compression'='GZIP', sep='\t', escapeQuotes=false) SELECT source.mall_id AS mall_id, COUNT(1) AS total FROM (SELECT cuid, mall_id, mall_name, date_type FROM huiyan_ns.huiyan.map_huiyan_mall_basic_source WHERE event_day = '{FORWARD_MONTH_END}') source JOIN (SELECT cuid, * FROM huiyan_ns.huiyan.map_huiyan_parse_userpic WHERE event_day >= '{FORWARD_7_DAY}' AND event_day <= '{FORWARD_MONTH_END}') userpic ON userpic.cuid = source.cuid GROUP BY source.mall_id, source.mall_name, source.date_type; alter table huiyan_ns.huiyan.map_huiyan_mall_userpic_raw add partition(event_day='{FORWARD_MONTH_END}') location 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_flow_userpic_raw/event_day={FORWARD_MONTH_END}'; set user_write_ugi:afs://kunpeng.afs.baidu.com:9902=lbs-huiyan,lbs-huiyan; insert overwrite directory 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day={FORWARD_MONTH_END}' using CSV options('compression'='GZIP', sep='\t', escapeQuotes=false) SELECT TRANSFORM(mall_id) USING 'python3 mall_userpic_streaming.py' AS (mall_id) FROM huiyan_ns.huiyan.map_huiyan_mall_userpic_raw WHERE event_day = '{FORWARD_MONTH_END}'; alter table huiyan_ns.huiyan.map_huiyan_mall_userpic add partition(event_type='monthly', event_day='{FORWARD_MONTH_END}') location 'afs://kunpeng.afs.baidu.com:9902/user/lbs-huiyan/warehouse/huiyan.db/map_huiyan_mall_userpic/event_type=monthly/event_day=${FORWARD_MONTH_END}'; ``` 请注意,这是将两段代码合并为一段,但我无法保证这段代码的逻辑和语法是否正确。请在使用之前进行适当的检查和测试。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

九师兄

你的鼓励是我做大写作的动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值