1.paimon与flink环境整合

概述

paimon 0.5 ,flink 1.17.1版本, 目标flink与paimon进行整合,测试的环境搞起

相关链接

阅读之前,可以先浏览一下

相关文章系列聚合

下面的链接是paimon官方文档地址,有兴趣的同学可以浏览一下
官方快速入门链接

jar包

整合需要添加一些jar包

上传

上传至 linux 机器

在这里插入图片描述

上传之后如下图

在这里插入图片描述

重启一下 yarn session ,如果还没有yarn session环境,请参照下面链接
yarn session环境搭建

测试

测试flink与paimon整合的环境是否有问题

yarn session

命令行连接 yarn session 环境

# 连接上 yarn session环境
[root@hadoop01 flink]# bin/sql-client.sh 
Flink SQL> CREATE CATALOG my_catalog WITH (
>     'type'='paimon',
>     'warehouse'='hdfs:///data/hive/warehouse/paimon'
> );
[INFO] Execute statement succeed.

Flink SQL> USE CATALOG my_catalog;
[INFO] Execute statement succeed.

Flink SQL> CREATE TABLE word_count (
>     word STRING PRIMARY KEY NOT ENFORCED,
>     cnt BIGINT
> );
[INFO] Execute statement succeed.

Flink SQL> CREATE TEMPORARY TABLE word_table (
>     word STRING
> ) WITH (
>     'connector' = 'datagen',
>     'fields.word.length' = '1'
> );
[INFO] Execute statement succeed.

Flink SQL> SET 'execution.checkpointing.interval' = '10 s';
[INFO] Execute statement succeed.

Flink SQL> INSERT INTO word_count SELECT word, COUNT(*) FROM word_table GROUP BY word;2023-10-19 09:44:34,955 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-10-19 09:44:34,992 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032
2023-10-19 09:44:35,081 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2023-10-19 09:44:35,082 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2023-10-19 09:44:35,106 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'.

[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: 13414ca4b36fd7adf20aa0b59e5fba0e


Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
[INFO] Execute statement succeed.

Flink SQL> RESET 'execution.checkpointing.interval';
[INFO] Execute statement succeed.

Flink SQL> SET 'execution.runtime-mode' = 'batch';
[INFO] Execute statement succeed.

Flink SQL> SELECT * FROM word_count;2023-10-19 09:45:04,314 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-10-19 09:45:04,357 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032
2023-10-19 09:45:04,358 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2023-10-19 09:45:04,358 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2023-10-19 09:45:04,361 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'.

+------+-------+
| word |   cnt |
+------+-------+
|    0 | 13114 |
|    1 | 13090 |
|    2 | 13223 |
|    3 | 12934 |
|    4 | 13266 |
|    5 | 12952 |
|    6 | 13113 |
|    7 | 13051 |
|    8 | 13070 |
|    9 | 13193 |
|    a | 13212 |
|    b | 13032 |
|    c | 13054 |
|    d | 13277 |
|    e | 13252 |
|    f | 13167 |
+------+-------+
16 rows in set

Flink SQL> SET 'execution.runtime-mode' = 'streaming';
[INFO] Execute statement succeed.

Flink SQL> SELECT `interval`, COUNT(*) AS interval_cnt FROM
>     (SELECT cnt / 10000 AS `interval` FROM word_count) GROUP BY `interval`;2023-10-19 09:50:49,672 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2023-10-19 09:50:49,695 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032
2023-10-19 09:50:49,696 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2023-10-19 09:50:49,696 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2023-10-19 09:50:49,699 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'.

+----+----------------------+----------------------+
| op |             interval |         interval_cnt |
+----+----------------------+----------------------+
| +I |                   22 |                    1 |
| -U |                   22 |                    1 |
| +U |                   22 |                    2 |
| -U |                   22 |                    2 |
| +U |                   22 |                    3 |
| -U |                   22 |                    3 |
| +U |                   22 |                    4 |
| -U |                   22 |                    4 |
| +U |                   22 |                    5 |
| -U |                   22 |                    5 |
| +U |                   22 |                    6 |
| -U |                   22 |                    6 |
| +U |                   22 |                    7 |
| -U |                   22 |                    7 |
| +U |                   22 |                    8 |
| -U |                   22 |                    8 |
| +U |                   22 |                    9 |
| -U |                   22 |                    9 |
| +U |                   22 |                   10 |
| -U |                   22 |                   10 |
| +U |                   22 |                   11 |
| -U |                   22 |                   11 |
| +U |                   22 |                   12 |
| -U |                   22 |                   12 |
| +U |                   22 |                   13 |
| -U |                   22 |                   13 |
| +U |                   22 |                   14 |
| -U |                   22 |                   14 |
| +U |                   22 |                   15 |
| -U |                   22 |                   15 |
| +U |                   22 |                   16 |
| -U |                   22 |                   16 |
| +U |                   22 |                   15 |
| +I |                   23 |                    1 |
| -U |                   22 |                   15 |
| +U |                   22 |                   14 |
| -U |                   23 |                    1 |
| +U |                   23 |                    2 |
| -U |                   22 |                   14 |
| +U |                   22 |                   13 |
| -U |                   23 |                    2 |
| +U |                   23 |                    3 |
| -U |                   22 |                   13 |
| +U |                   22 |                   12 |
| -U |                   23 |                    3 |
| +U |                   23 |                    4 |
| -U |                   22 |                   12 |
| +U |                   22 |                   11 |
| -U |                   23 |                    4 |
| +U |                   23 |                    5 |
| -U |                   22 |                   11 |
| +U |                   22 |                   10 |
| -U |                   23 |                    5 |
| +U |                   23 |                    6 |
| -U |                   22 |                   10 |
| +U |                   22 |                    9 |
| -U |                   23 |                    6 |
| +U |                   23 |                    7 |
| -U |                   22 |                    9 |
| +U |                   22 |                    8 |
| -U |                   23 |                    7 |
| +U |                   23 |                    8 |
| -U |                   22 |                    8 |
| +U |                   22 |                    7 |
| -U |                   23 |                    8 |
| +U |                   23 |                    9 |
| -U |                   22 |                    7 |
| +U |                   22 |                    6 |
| -U |                   23 |                    9 |
| +U |                   23 |                   10 |
| -U |                   22 |                    6 |
| +U |                   22 |                    5 |
| -U |                   23 |                   10 |
| +U |                   23 |                   11 |
| -U |                   22 |                    5 |
| +U |                   22 |                    4 |
| -U |                   23 |                   11 |
| +U |                   23 |                   12 |
| -U |                   22 |                    4 |
| +U |                   22 |                    3 |
| -U |                   23 |                   12 |
| +U |                   23 |                   13 |
| -U |                   22 |                    3 |
| +U |                   22 |                    2 |
| -U |                   23 |                   13 |
| +U |                   23 |                   14 |
| -U |                   22 |                    2 |
| +U |                   22 |                    1 |
| -U |                   23 |                   14 |
| +U |                   23 |                   15 |
| -D |                   22 |                    1 |
| -U |                   23 |                   15 |
| +U |                   23 |                   16 |
^CQuery terminated, received a total of 93 rows

Flink SQL> 

效果

在这里插入图片描述

结束

paimon与flink的整合环境ok

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

流月up

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值