UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘s3‘

flink UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme ‘s3’

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugins: flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:513)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
	at org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:214)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)

yarn-session模式提交task,出现UnsupportedFileSystemSchemeException,
查看 flink-conf.yaml文件,参数fs.allowed-fallback-filesystems已经配置。
在这里插入图片描述
回看错误,发现以下提示信息

The scheme is directly supported by Flink through the following plugins: 
flink-s3-fs-hadoop, flink-s3-fs-presto.
 Please ensure that each plugin resides within its own subfolder within the plugins directory. 
 See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html

应该是在升级flink版本的时候,忘记配置flink的plugins,查看plugins目录,果然没有flink-s3-fs-hadoop, flink-s3-fs-presto
在这里插入图片描述
如何配置这些插件,好在提示信息中给了参考网址,打开对应的网址,我得到了如何配置的信息。
在这里插入图片描述
按提示配置了s3-fs-hadoop和 flink-s3-fs-presto插件,命令如下:

# s3-fs-hadoop
mkdir ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.13.1.jar ./plugins/s3-fs-hadoop/

#flink-s3-fs-presto
mkdir ./plugins/s3-fs-presto
cp ./opt/flink-s3-fs-presto-1.13.1.jar ./plugins/s3-fs-presto/

重启一下yarn-session, 然后重新提交task,没有错误了。

在这里插入图片描述

  • 1
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值