Flink安装流程及问题

目录

1.Linux安装

2.Windows安装

---

1.Linux安装

    最近在虚拟机上部署很多的环境，但是由于很多都想用新版本一些的，所以遇到了很多问题，在这里记录一下。

    我下载的版本是flink-1.10.0-bin-scala_2.12.tgz，网址为Apache Flink: Downloads，流程如下

    当然可以根据需要更改版本，不过可能不太适合这篇文章的错误和排错方式，这点要注意。

    下载完成后进行安装，安装后可以选择修改/etc/profile文件和更改文件夹或ln操作等等，这里就不赘述了：

tar -zxvf flink-1.10.0-bin-scala_2.12.tgz 

    另外针对1.10版本的flink要注意的是其更改了类加载策略，关于flink1.10版本的新特性可以看这位大神的文章，很详细，也能帮你定位很多问题（Flink 1.10 新特性研究 | zhisheng的博客）。所以需要对/conf/flink-conf.yaml进行更改。

......
# The external address of the host on which the JobManager runs and can be
# reached by the TaskManagers and any clients which want to connect. This setting
# is only used in Standalone mode and may be overwritten on the JobManager side
# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.

jobmanager.rpc.address: 10.26.37.225

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123

......

# The classloading resolve order. Possible values are 'child-first' (Flink's default)
# and 'parent-first' (Java's default).
#
# Child first classloading allows users to use different dependency/library
# versions in their application than those in the classpath. Switching back
# to 'parent-first' may help with debugging dependency issues.
#
# 2020/04/29 更改类加载机制为 parent-first 默认为 child-first
 classloader.resolve-order: parent-first

......

    其他内容我进行了隐藏，为基本配置，我主要更改的点是：jobmanager.rpc.address: xx.xx.xx.xx，将其更改为了当前机器的IP地址，此外将类加载机制进行了更改，classloader.resolve-order: parent-first。这两句分别会解决对应的问题，可以选择不设置然后查看一下问题点。

    启动方式为在flink的解压目录内的bin文件夹下执行：

./start-cluster.sh 

    一定要在jps中看到如下才能判定为flink启动成功：

30737 StandaloneSessionClusterEntrypoint
31046 TaskManagerRunner

    另外如果要测试/flink-1.10.0/examples/streaming下的WordCount.jar要注意这个版本需要下一个对应的hadoop jar包，Central Repository: org/apache/flink/flink-shaded-hadoop-2-uber，具体内容可以参考（Apache Flink 1.9 Documentation: YARN Setup）

2.Windows安装

   在Windows的安装与Linux没有什么特别的差距，这里主要是针对文档中的测试用例进行一些问题总结和整理。

1.Maven工程

   在这里使用Idea进行Flink应用的开发前可以使用官方的文档进行一些测试，官方提供了flink-walkthrough-common，

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>1.10.0</version>
            <scope>provided</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-core -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>1.10.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.10.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>1.10.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-walkthrough-common -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-walkthrough-common_2.12</artifactId>
            <version>1.10.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-nop</artifactId>
            <version>1.7.2</version>
        </dependency>
    </dependencies>

2.问题解决

   在用Idea直接运行官方的用例时会发现该问题：

java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/functions/source/SourceFunction
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.api.functions.source.SourceFunction
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" 
Process finished with exit code 1

   解决方案：

      File——>Project Structure——>Modules——>Dependencies——>add Jars or Directories——>在这里将Flink包中的apache-flink-1.10.0\deps\lib\flink-dist_2.11-1.10.0.jar添加进环境中便可以解决该问题