StructuredStreamingInSQL项目实现动态更新新版(直接替换，无需重启)。

最新推荐文章于 2023-09-22 19:55:22 发布

有梦想的人不睡觉121

最新推荐文章于 2023-09-22 19:55:22 发布

阅读量582

点赞数 1

分类专栏： spark

本文链接：https://blog.csdn.net/qq_36421826/article/details/85707473

版权

spark 专栏收录该内容

21 篇文章 0 订阅

订阅专栏

CREATE TABLE kafkaTable(
    word string
)WITH(
    type='kafka',
    kafka.bootstrap.servers='kafka001:9092',
    processwindow='10 seconds,10 seconds',
    watermark='10 seconds',
    subscribe='test',
    group='test'
);

create SINK consoleOut(
)WITH(
    type='console',
    outputmode='complete',
    process='10s'
);

insert into consoleOut select word,count(*) from kafkaTable group by word,processwindow;


启动
分隔符未配置，默认为逗号
root
 |-- WORD: string (nullable = true)
 |-- timestamp: timestamp (nullable = true)
 |-- processwindow: struct (nullable = false)
 |    |-- start: timestamp (nullable = true)
 |    |-- end: timestamp (nullable = true)

table:KAFKATABLE
first add :e753351e-9777-448e-b649-704e59c11755
-------------------------------------------
Batch: 0
-------------------------------------------
+----+--------+
|WORD|count(1)|
+----+--------+
+----+--------+

Query made progress: e753351e-9777-448e-b649-704e59c11755
active query length:1

输入数据
-------------------------------------------
Batch: 1
-------------------------------------------
+----+--------+
|WORD|count(1)|
+----+--------+
|c   |7       |
|a   |3       |
+----+--------+

替换SQL
CREATE TABLE kafkaTable(
    word string
)WITH(
    type='kafka',
    kafka.bootstrap.servers='kafka001:9092',
    processwindow='10 seconds,10 seconds',
    watermark='10 seconds',
    subscribe='test',
    group='test'
);

create SINK consoleOut(
)WITH(
    type='console',
    outputmode='complete',
    process='10s'
);

insert into consoleOut select processwindow,word,count(*) from kafkaTable group by word,processwindow;

sql中增加一个processwindow显示

输入数据

SQL更新了呦
分隔符未配置，默认为逗号
root
 |-- WORD: string (nullable = true)
 |-- timestamp: timestamp (nullable = true)
 |-- processwindow: struct (nullable = false)
 |    |-- start: timestamp (nullable = true)
 |    |-- end: timestamp (nullable = true)

table:KAFKATABLE
Query started: 549cf074-6cf9-4934-b57d-0932230320e8
zhiqian :e753351e-9777-448e-b649-704e59c11755
new de  :549cf074-6cf9-4934-b57d-0932230320e8
Query terminated: e753351e-9777-448e-b649-704e59c11755
-------------------------------------------
Batch: 0
-------------------------------------------
+-------------+----+--------+
|PROCESSWINDOW|WORD|count(1)|
+-------------+----+--------+
+-------------+----+--------+

Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
-------------------------------------------
Batch: 1
-------------------------------------------
+------------------------------------------+----+--------+
|PROCESSWINDOW                             |WORD|count(1)|
+------------------------------------------+----+--------+
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|a   |3       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|c   |8       |
+------------------------------------------+----+--------+

Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
Query made progress: 549cf074-6cf9-4934-b57d-0932230320e8
active query length:1
-------------------------------------------
Batch: 2
-------------------------------------------
+------------------------------------------+----+--------+
|PROCESSWINDOW                             |WORD|count(1)|
+------------------------------------------+----+--------+
|[2019-01-03 14:49:50, 2019-01-03 14:50:00]|a   |3       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|a   |3       |
|[2019-01-03 14:49:50, 2019-01-03 14:50:00]|c   |7       |
|[2019-01-03 14:50:00, 2019-01-03 14:50:10]|c   |1       |
|[2019-01-03 14:49:30, 2019-01-03 14:49:40]|c   |8       |
+------------------------------------------+----+--------+

jobid实现了更新，之前的窗口也不见了，解决了前面只能添加，不能删除的bug

github地址：https://github.com/peopleindreamdontsleep/StructuredStreamingInSQL

喜欢，就star一下吧