StructuredStreamingInSQL项目实现动态更新 新版(直接替换,无需重启)。

CREATE TABLE kafkaTable(
    word string
)WITH(
    type='kafka',
    kafka.bootstrap.servers='kafka001:9092',
    processwindow='10 seconds,10 seconds',
    watermark='10 seconds',
    subscribe='test',
    group='test'
);

create SINK consoleOut(
)WITH(
    type='console',
    outputmode='complete',
    process='10s'
);

insert into consoleOut select word,count(*) from kafkaTable group by word,processwindow;


启动
分隔符未配置,默认为逗号
root
 |-- WORD: string (nullable = true)
 |-- timestamp: timestamp (nullable = true)
 |-- processwindow: struct (nullable = false)
 |    |-- start: timestamp (nullable = true)
 |    |-- end: timestamp (nullable = true)

table:KAFKATABLE
first add :e753351e-9777-448e-b649-704e59c11755
-------------------------------------------
Batch: 0
-------------------------------------------
+----+--------+
|WORD|count(1)|
+----+--------+
+----+--------+

Query made progress: e753351e-9777-448e-b649-704e59c11755
active query length:1

输入数据
-------------------------------------------
Batch: 1
-------------------------------------------
+----+--------+
|WORD|count(1)|
+----+--------+
|c   |7       |
|a   |3       |
+----+--------+

替换SQL
CREATE TABLE kafkaTable(
    word string
)WITH(
    type='kafka',
    kafka.bootstrap.servers='kafka001:9092',
    processwindow='10 seconds,10 seconds',
    watermark='10 seconds',
    subscribe='test',
    group='test'
);

create SINK consoleOut(
)WITH(
    type='console',
    outputmode='complete',
    process='10s'
);

insert into consoleOut select processwindow,word,count(*) from kafkaTable group by word,processwindow;

sql中增加一个processwindow显示

输入数据

SQL更新了呦
分隔符未配置,默认为逗号
root
 |-- WORD: string (nullable = true)
 |-- timestamp: timestamp (nullable = true)
 |-- processwindow: struct (nullable = false)
 |    |-- start: timestamp (nullable = true)
 |    |-- end: timestamp (nullable = true)

table:KAFKATABLE
Query started: 549cf074-6cf9-4934-b57d-0932230320e8
zhiqian :e753351e-9777-448e-b649-704e59c11755
new de  :549cf074-6cf9-4934-b57d-0932230320e8
Query terminated: e753351e-9777-448e-b649-704e59c11755
-------------------------------------------
Batch: 0
-------------------------------------------
+-------------+----+--------+
|PROCESSWINDOW|WORD|count(1)|
+-------------+----+--------+
+-------------+----+--------+

Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
-------------------------------------------
Batch: 1
-------------------------------------------
+------------------------------------------+----+--------+
|PROCESSWINDOW                             |WORD|count(1)|
+------------------------------------------+----+--------+
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|a   |3       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|c   |8       |
+------------------------------------------+----+--------+

Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
-------------------------------------------
Batch: 2
-------------------------------------------
+------------------------------------------+----+--------+
|PROCESSWINDOW                             |WORD|count(1)|
+------------------------------------------+----+--------+
|[2019-01-03 14:49:50, 2019-01-03 14:50:00]|a   |3       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|a   |3       |
|[2019-01-03 14:49:50, 2019-01-03 14:50:00]|c   |7       |
|[2019-01-03 14:50:00, 2019-01-03 14:50:10]|c   |1       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|c   |8       |
+------------------------------------------+----+--------+

jobid实现了更新,之前的窗口也不见了,解决了前面只能添加,不能删除的bug

github地址:https://github.com/peopleindreamdontsleep/StructuredStreamingInSQL

喜欢,就star一下吧

  • 1
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值