问题1:Flink 1.12 Could not find any factory for identifier 'kafka' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath
解决方法:
pom.xml文件中加入依赖(也可去如下网站下载对应版本)
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.11/1.12.1
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-sql-connector-kafka_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<!--<scope>provided</scope>-->
</dependency>
问题2:ByteArraySerializer is not an instance of org.kafka.common.serialization.Deserializer
解决:看下pom里是不是引入了flink-connector-kafka_2.11-1.12.0.jar,这是pom的flink-connector-kafka与服务器的flink/lib/flink-sql-connector-kafka冲突,删掉flink-connector-kafka就行,引入flink-sql-connctor-kafka
问题3:java.lang.ClassCastException: org.codehaus.janino.CompilerFactory cannot be cast to org.codehaus.commons.compiler.ICompilerFactory
解决:本地环境jar包中flink-table-planner包与集群中flink/lib中的 flink-table_2.12-1.14.5.jar 冲突,在打包时在pom里写上<scope>provided</scope>不打flink-table-planner包即可
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_${scala.version}</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
问题4:Caused by: java.lang.AbstractMethodError: Method org/apache/hudi/sink/StreamWriteOperatorCoordinator.subtaskReady(ILorg/apache/flink/runtime/operators/coordination/OperatorCoordinator$SubtaskGateway;)V is abstract
解决:服务器上flink与hudi版本不兼容造成,在代码的pom里引入匹配的版本并且打包时把所有依赖打进去,这样在运行时用的是pom的依赖包,否则用的是服务器上flink与hudi的包
问题5:org.apache.flink.client.program.ProgramInvocationException:在jar文件中找不到
原因:打包时没有将pom依赖全打进去
解决:build换成这个
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<testSourceDirectory>src/test/java</testSourceDirectory>
<plugins>
<!-- 编译插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
<!-- 打jar包插件(会包含所有依赖) -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
问题6:Caused by: java.net.BindException: Could not start actor system on any port in port range 6123
原因:子用户提交flink时提示端口被占用
解决:切换root用户,kill掉任务,再用子用户提交
问题7: Could not deploy Yarn job cluster
解决:将日志拉取下来,查看具体原因,大部分是因为高可用出问题了,导致yarn没能提交
通过yarn logs -applicationId application_1668999502265_2814 > /user/joseph/application_1668999502265_2814.txt