flink UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘s3’
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugins: flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:513)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:214)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)
yarn-session模式提交task,出现UnsupportedFileSystemSchemeException,
查看 flink-conf.yaml
文件,参数fs.allowed-fallback-filesystems
已经配置。
回看错误,发现以下提示信息
The scheme is directly supported by Flink through the following plugins:
flink-s3-fs-hadoop, flink-s3-fs-presto.
Please ensure that each plugin resides within its own subfolder within the plugins directory.
See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html
应该是在升级flink版本的时候,忘记配置flink的plugins,查看plugins目录,果然没有flink-s3-fs-hadoop, flink-s3-fs-presto
如何配置这些插件,好在提示信息中给了参考网址,打开对应的网址,我得到了如何配置的信息。
按提示配置了s3-fs-hadoop和 flink-s3-fs-presto插件,命令如下:
# s3-fs-hadoop
mkdir ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.13.1.jar ./plugins/s3-fs-hadoop/
#flink-s3-fs-presto
mkdir ./plugins/s3-fs-presto
cp ./opt/flink-s3-fs-presto-1.13.1.jar ./plugins/s3-fs-presto/
重启一下yarn-session, 然后重新提交task,没有错误了。