flink
flink
青衣沽酒醉风尘
成为大V的道路任重而道远
展开
-
flink1.13.3集成iceberg0.12-(7)
iceberg0.12 master集成flink 1.13.3,自己下载master源码打包1. flink1.13.3下载地址1. flink1.13.3wget https://dlcdn.apache.org/flink/flink-1.13.3/flink-1.13.3-bin-scala_2.12.tgz这个包放到flink lib下wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-conn原创 2021-11-16 17:21:25 · 952 阅读 · 0 评论 -
flink1.12.3集成iceberg0.12-(6)
iceberg0.11 升级iceberg 0.12:release-notes 地址(采用对应的fink1.12.4版本)https://iceberg.apache.org/releases/#0120-release-notesjar包下载地址将下面两个包放到flink的lib目录下wget https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-flink-runtime/0.12原创 2021-11-16 17:20:09 · 1320 阅读 · 0 评论 -
flink1.11.1集成iceberg0.11(hadoop_catalog)-(5)
创建hive on iceberg 外部表:下载 jar包地址;https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-hive-runtime/0.11.0/iceberg-hive-runtime-0.11.0.jar将jar包放到hive的auxlib目录下此步骤略过进入 hivehive开启参数set iceberg.engine.hive.enabled=true...原创 2021-11-16 17:19:23 · 567 阅读 · 0 评论 -
flink1.11.1集成iceberg0.11(hadoop_catalog)-(4)
flink 读取 iceberg upsert数据流:读取iceberg数据// 创建执行环境envE StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().build();原创 2021-11-16 17:17:03 · 1078 阅读 · 1 评论 -
flink1.11.1集成iceberg0.11(hadoop_catalog)-(3)
iceberg表参数配置:iceberg表的参数配置可以配置在with模块例如:CREATE TABLE hadoop_catalog.iceberg.iceberg_t1 ( id BIGINT COMMENT 'unique id', data BIGINT) with ('write.distribution-mode'='hash'); 定义写入数据的分布:none:不洗牌行;hash : 散列按分区键分布;range : 如果表具有 SortOrder,则.原创 2021-11-16 17:15:44 · 1210 阅读 · 0 评论 -
flink1.11.1集成iceberg0.11(hadoop_catalog)-(2)
上一篇文章讲了flink集成iceberg需要的包,这篇讲下flink集成iceberg一些简单的demo测试1.进入sql-client.sh命令行# -j 指定 iceberg flink的集成包# -d 指定配置文件export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`./bin/sql-client.sh embedded -j /opt/iceberg0.11/iceberg-flink-runtime-0...原创 2021-11-16 17:10:56 · 1230 阅读 · 0 评论 -
flink1.11.1集成iceberg0.11(hadoop_catalog)-(1)
确定 cdh 环境,flink集群已经准备好cdh 6.3.2 flink1.11.1 scala2.11需要的包:iceberg集成flink包https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime/0.11.1/iceberg-flink-runtime-0.11.1.jarhttps://repo.maven.apache.org/maven2/org/apach.原创 2021-11-16 16:59:27 · 550 阅读 · 0 评论