背景说明
在搭建Flink的HA高可用集群后,打不开web UI界面
出现问题
打开log日志查看
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/
问题分析
1、最新版本的Flink没有包含Hadoop的相关依赖,无法直接操作HDFS
2、官网声明Flink没有对Hadoop3.x的支持,但若是没用到Hadoop3.x的新特性,只用Hadoop2.x的常规功能,现有jar包可行
问题解决
往Flink中的lib目录里添加两个jar包:
flink-shaded-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar
commons-cli-1.4.jar
添加完后,将lib目录分发给其他虚拟机。在这里,我的hadopp103配置的是备用master,若hadoop3:8081的界面要打开,那么hadoop103的虚拟机上也需要添加上面两个jar包
下载仓库分别是:
https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-hadoop-3-uber/3.1.1.7.2.1.0-327-9.0
https://mvnrepository.com/artifact/commons-cli/commons-cli/1.4