Flink在standalone模式下的打包运行常见问题总结

Flink在standalone模式下的打包运行常见问题总结

flink应用在读取或者写入hdfs的时候容易出现以下问题：
错误1：

java.io.IOException: DataStreamer Exception: 
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:562)
Caused by: javax.xml.parsers.FactoryConfigurationError: Provider for class javax.xml.parsers.DocumentBuilderFactory cannot be created
	at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:311)
	at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)
	at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2516)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:981)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031)
	at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1251)
	at org.apache.hadoop.hdfs.protocol.HdfsConstants.<clinit>(HdfsConstants.java:76)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)

该问题一般是涉及hadoop中hdfs中配置相关的错误，需要在maven的pom.xml文件中清除涉及hadoop的hdfs相关的项目，同时在集群总配置好如下的配置
client端需要配置环境变量（vi /etc/profile）：

YARN_CONF_DIR 
HADOOP_HOME=/hadoop/hadoop-2.7.2/ 
HADOOP_CONF_DIR=/hadoop/hadoop-2.7.2/etc/hadoop/

(三者有一个即可，前提是集群中已经安装的hadoop、hdfs环境)

错误二

org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.at org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(PackagedProgram.java:592)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:188)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:126)
at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:866)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:252)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1101)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1098)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)

这种情况是：
1.打包的时候没有将主类mianClass文件打入jar包中，
2.或者程序运行时候没有指定正确的主类名称路径

最后建议使用flink on yarn模式提交flink应用 ,详请参考（https://blog.csdn.net/fct2001140269/article/details/85334323）；