flink1.12.7+hudi 问题总结

版本：CDH-6.3.2， flink-1.12.7 ，hudi -0.9.0/0.10.0

1.CDH安装flink,需要自己制作parcel，制作过程略;

2.hudi可以自己编译:：https://github.com/apache/hudi，

也可以自己下载:https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink-bundle_2.12/0.9.0/hudi-flink-bundle_2.12-0.9.0.jar

3.将 hudi-flink-bundle_2.12-0.9.0.jar 复制到flink的lib下，我的路径是/opt/cloudera/parcels/FLINK/lib/flink/lib，可以根据自己的实际情况调整；

4. ../bin/start-cluster.sh              -- 启动cluster

   ../bin/sql-client.sh embedded  -- 启动flink sql client

问题1:创建表成功，但是执行查询报错:

Flink SQL> select * from t1;
[ERROR] Could not execute SQL statement. Reason:
java.lang.NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/ResolvedCatalogTable;

之前看网上说是jar的原因，参见https://github.com/ververica/flink-cdc-connectors/issues/197;

我测试了mysql-cdc,用flink-connector-mysql-cdc-1.4.0.jar 依然报这个错，换

flink-connector-mysql-cdc-1.2.0.jar  可以正常查询了

那么推测是jar版本的原因，所以我试了hudi-flink-bundle_2.12-0.9.0.jar和 hudi-flink-bundle_2.12-0.10.0.jar 依然不行，后来看官方文档 (Flink Guide | Apache Hudi)时候发现启动时候指定了jar,于是将启动命令替换为 ../bin/sql-client.sh embedded -j hudi-flink-bundle_2.12-0.9.0.jar shell,执行查询，报了另外一个错:

 
Flink SQL> select * from t1;
[ERROR] Could not execute SQL statement. Reason:
java.io.FileNotFoundException: File does not exist: hdfs:/flink/hudi_t1/.hoodie

检查jar包时发现，lib中原来有个hudi-flink_2.11-0.10.0.jar,我没注意到,可能默认用的是这个jar包,后面把文件名改为hudi-flink_2.11-0.10.0.jar1，不需要指定jar就能正常启动了;特别要说明一点，如果使用hudi-flink-bundle_2.12-0.10.0.jar ,依然会报org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/ResolvedCatalogTable，所以下文用的jar都是hudi-flink-bundle_2.12-0.9.0.jar，不在另外说明

问题2:直接执行查询，报  java.io.FileNotFoundException: File does not exist: hdfs:/flink/hudi_t1/.hoodie;插入数据报错

打开http://hadoop04:8082/#/overview（默认端口8081,我的端口改为了8082）,能看到页面正常显示，进入sql client,执行插入语句，会显示 [INFO] Submitting SQL update statement to the cluster...，大约1min后报错:Exception in thread "main"  org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:215)
Caused by: java.lang.RuntimeException: Error running SQL job.

。。。

Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: 拒绝连接: localhost/127.0.0.1:8082

，sql client 会被关闭，然后刷新 http://hadoop04:8082/#/overview  页面，发现无法正常打开

原有的日志太少，无法定位错误，所以我想到将job提交到yarn去执行，依然报错，但是

提示执行 yarn logs -applicationId application_1648377899884_0004 去查看详细的日志，果然发现

Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: user=root, access=WRITE, inode="/user/flink":flink:flink:drwxr-x--x

，原来是我hdfs目录没有操作权限,赋权后

 任务成功执行，执行查询同样成功

 

问题3:Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration ‘classloader.check-leaked-classloader’

flink-conf.yaml 增加 ：classloader.check-leaked-classloader: false