flink1.12整合hadoop3.0

通常Flink集群配置好后，还不能直接访问HDFS，因为缺少相关jar包，会弹出无法解析hdfs类型的schema的错误：
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

背景：在Flink分布式环境下，配置状态后端RocksDb采用Hdfs文件系统持久化检查点信息
代码：env.setStateBackend(new RocksDBStateBackend("hdfs://master:8020/flink/testjob/chk"));

下载地址：https://mvnrepository.com/

<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-hadoop-3-uber -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-shaded-hadoop-3-uber</artifactId>
    <version>3.1.1.7.2.1.0-327-9.0</version>
    <scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-cli/commons-cli -->
<dependency>
    <groupId>commons-cli</groupId>
    <artifactId>commons-cli</artifactId>
    <version>1.4</version>
</dependency>

将如上jar包，放入$FLINK_HOME/lib目录下，重启flink集群(standalone模式)
bin/stop-cluster.sh
bin/start-cluster.sh

启动flink作业
$FLINK_HOME/bin/flink run -p 3 -c my.etl.job.TestJob -d test-job.jar

检查作业正确性
1、Flink Dashboard查看job状态，excepion日志
2、Flink Dashboard查看task状态，subtask详情

Idea中Flink本地项目代码调试整合Hadoop
1、core-site.xml & hdfs-site.xml放入maven项目resources资源目录下
2、pom.xml中引入上述jar依赖（去除<scope>test</scope>）
3、模拟master主机root用户（避免win用户访问hdfs权限异常）
     System.setProperty("HADOOP_USER_NAME", "root");