Flink-1.8.1踩坑记录 - HDFS的scheme不支持问题

最新推荐文章于 2023-09-01 17:22:11 发布

chuba5570

最新推荐文章于 2023-09-01 17:22:11 发布

阅读量6.2k

点赞数 2

文章标签：大数据 runtime 网络

原文链接：https://my.oschina.net/u/2338224/blog/3101005

版权

前置环境说明：

centos 6.10

hadoop 2.7.3

oracle jdk1.8

flink1.8.1

scala 2.11

安装完集群之后提交一个jar包中的类运行报错 , （这个类里面写了操作HDFS的内容）

2019-09-02 18:42:01,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job DimensionState (1b47949b019929e25b8bee89539f9f1d) switched from state FAILING to FAILED.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
	... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
	at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
	... 11 more
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Could not restart the job DimensionState (1b47949b019929e25b8bee89539f9f1d) because the restart strategy prevented it.
java.lang.RuntimeException: Error while creating FileSystem when initializing the state of the BucketingSink.
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:379)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.createHadoopFileSystem(BucketingSink.java:1227)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initFileSystem(BucketingSink.java:432)
	at org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.initializeState(BucketingSink.java:376)
	... 8 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
	at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
	... 11 more
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Stopping checkpoint coordinator for job 1b47949b019929e25b8bee89539f9f1d.
2019-09-02 18:42:01,652 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore  - Shutting down
2019-09-02 18:42:01,662 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Job 1b47949b019929e25b8bee89539f9f1d reached globally terminal state FAILED.
2019-09-02 18:42:01,704 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Stopping the JobMaster for job DimensionState(1b47949b019929e25b8bee89539f9f1d).
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Suspending SlotPool.
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Close ResourceManager connection 0f8cea6c8e8a3904e338bfe4b4aca43b: JobManager is shutting down..
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl      - Stopping SlotPool.
2019-09-02 18:42:01,711 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@localhost:6123/user/jobmanager_0 for job 1b47949b019929e25b8bee89539f9f1d from the resource manager.
2019-09-02 18:42:01,712 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner           - JobManagerRunner already shutdown.

我提取一下上面关键字 ,方便搜索引擎爬取, 以试图帮助其他人:

1. while creating FileSystem when initializing the state of the BucketingSink.

2. Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

3.org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

1.8.1如果要操作hdfs的话，必须要在flink安装目录的 lib 下加上额外的jar包

官网有介绍:

https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html

所以这里原因就是在flink集群的CLASS_PATH下缺少了 HDFS相关的ja

解决办法: