Flink集群部署

一、架构说明(standalone模式)

standalone是Flink自带的一个分布式集群，它不依赖其他的资源调度框架、不依赖yarn等。Spark也有一个自带的分布式资源调度模型，也叫作standalone

standalone模式进程名介绍：

1. JobManager------->进程名字：StandaloneSessionClusterEntrypoint 我们又叫它：Master
2. TaskManager----->进程名字：TaskManagerRunner 我们又叫它：Worker

1.1 机器分布

机器名(hosts)	IP	角色
bigdata111	192.168.1.111	JobManager、TaskManager
bigdata112	192.168.1.112	TaskManager
bigdata113	192.168.1.113	TaskManager

1.3 搭建步骤

1. 上传 Flink 安装包至 Linux 服务器 master 节点。使用 tar 命令解压安装包

   tar -zxvf flink-1.7.2-bin-scala_2.12.tgz -C /home/xyp9x/
   

2. 修改 conf 目录下的 flink-conf.yaml 配置文件

   vi flink-conf.yaml
   
   # 指定jobmanager的地址(此处可使用ip/别名)
   jobmanager.rpc.address: bigdata111
   # 指定taskmanager的可用槽位的数量(槽位越多，运行的subTask就越多)
   taskmanager.numberOfTaskSlots: 2
   

3. 修改 conf 目录下的 masters 配置文件，指定 jobmanager 的所在节点

   vi masters
   
   bigdata111
   

4. 修改 conf 目录下的 slaves 配置文件，指定 taskmanager 的所在节点

   vi slaves
   
   bigdata111
   bigdata112
   bigdata113
   

5. 环境变量

   vi /etc/profile
   
   export FLINK_HOME=/home/xyp9x/flink-1.7.2
   export PATH=$PATH:$FLINK_HOME/bin
   
   source /etc/profile
   

6. 将配置好的 Flink ，拷贝到其他节点（112,113节点）

   scp -r flink-1.7.2/ xyp9x@bigdata112:`pwd`
   scp -r flink-1.7.2/ xyp9x@bigdata113:`pwd`
   

7. 配置完成，目前环境配置为：1个JobManager，3个 TaskManager

1.4 启动Flink集群

start-cluster.sh

1.5 访问JobManager 的 Web 管理界面

192.168.1.111:8081

二、搭建 maven 工程

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_2.12</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.12</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_2.12</artifactId>
            <version>1.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.bahir</groupId>
            <artifactId>flink-connector-redis_2.11</artifactId>
            <version>1.0</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.44</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!--  该插件用于将 Scala 代码编译成 class 文件 -->
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.4.6</version>
                <executions>
                    <execution>
                        <!--  声明绑定到 maven 的 compile 阶段 -->
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19</version>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

无法加载类时build可以换另外一个：

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19</version>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>