flink1.12.7+hudi 问题总结

版本:CDH-6.3.2, flink-1.12.7 ,hudi -0.9.0/0.10.0

1.CDH安装flink,需要自己制作parcel,制作过程略;

2.hudi可以自己编译::https://github.com/apache/hudi,

也可以自己下载:https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink-bundle_2.12/0.9.0/hudi-flink-bundle_2.12-0.9.0.jar

3.将 hudi-flink-bundle_2.12-0.9.0.jar 复制到flink的lib下,我的路径是/opt/cloudera/parcels/FLINK/lib/flink/lib,可以根据自己的实际情况调整;

4. ../bin/start-cluster.sh              -- 启动cluster

   ../bin/sql-client.sh embedded  -- 启动flink sql client

问题1:创建表成功,但是执行查询报错:

Flink SQL> select * from t1;
[ERROR] Could not execute SQL statement. Reason:
java.lang.NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/ResolvedCatalogTable;

之前看网上说是jar的原因,参见https://github.com/ververica/flink-cdc-connectors/issues/197;

我测试了mysql-cdc,用flink-connector-mysql-cdc-1.4.0.jar 依然报这个错,换

flink-connector-mysql-cdc-1.2.0.jar  可以正常查询了

那么推测是jar版本的原因,所以我试了hudi-flink-bundle_2.12-0.9.0.jar和 hudi-flink-bundle_2.12-0.10.0.jar 依然不行,后来看官方文档 (Flink Guide | Apache Hudi)时候发现启动时候指定了jar,于是将启动命令替换为 ../bin/sql-client.sh embedded -j hudi-flink-bundle_2.12-0.9.0.jar shell,执行查询,报了另外一个错:

 
Flink SQL> select * from t1;
[ERROR] Could not execute SQL statement. Reason:
java.io.FileNotFoundException: File does not exist: hdfs:/flink/hudi_t1/.hoodie

检查jar包时发现,lib中原来有个hudi-flink_2.11-0.10.0.jar,我没注意到,可能默认用的是这个jar包,后面把文件名改为hudi-flink_2.11-0.10.0.jar1,不需要指定jar就能正常启动了;特别要说明一点,如果使用hudi-flink-bundle_2.12-0.10.0.jar ,依然会报org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/ResolvedCatalogTable,所以下文用的jar都是hudi-flink-bundle_2.12-0.9.0.jar,不在另外说明

问题2:直接执行查询,报  java.io.FileNotFoundException: File does not exist: hdfs:/flink/hudi_t1/.hoodie;插入数据报错

打开http://hadoop04:8082/#/overview(默认端口8081,我的端口改为了8082),能看到页面正常显示,进入sql client,执行插入语句,会显示 [INFO] Submitting SQL update statement to the cluster...,大约1min后报错:Exception in thread "main"  org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:215)
Caused by: java.lang.RuntimeException: Error running SQL job.

。。。

Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: 拒绝连接: localhost/127.0.0.1:8082

,sql client 会被关闭,然后刷新 http://hadoop04:8082/#/overview  页面,发现无法正常打开

原有的日志太少,无法定位错误,所以我想到将job提交到yarn去执行,依然报错,但是

提示执行 yarn logs -applicationId application_1648377899884_0004 去查看详细的日志,果然发现

Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: user=root, access=WRITE, inode="/user/flink":flink:flink:drwxr-x--x

,原来是我hdfs目录没有操作权限,赋权后

 任务成功执行,执行查询同样成功

 

问题3:Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration ‘classloader.check-leaked-classloader’

flink-conf.yaml 增加 :classloader.check-leaked-classloader: false 

  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值