Apache Flink Standalone or HA搭建

Apache Flink（三）

Standalone模式

前提条件

JDK1.8+安装完成
HDFS正常启动（SSH免密认证）

• 设置CentOS进程数和文件数(重启生效) -可选

[root@HadoopNode00 ~]# vi /etc/security/limits.conf
* soft nofile 204800
* hard nofile 204800
* soft nproc 204800
* hard nproc 204800

• 配置主机名(重启生效)

[root@HadoopNode00 ~]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=HadoopNode00

• 配置主机名和IP的关系

[root@HadoopNode00 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.126.10 HadoopNode00

• 关闭防火墙

[root@HadoopNode00 ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@HadoopNode00 ~]# chkconfig iptables off
[root@HadoopNode00 ~]# chkconfig --list | grep iptables
iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

• 安装JD1.8,配置JAVA_HOME（~/.bashrc）-略
• 配置SSH面密码认证-略
• 安装配置Hadoop，配置HADOOP_HOME和HADOOP_CALSSPATH（~/.bashrc）- 略
• Flink 安装与配置

1,解压flink-1.8.1-bin-scala_2.11.tgz到指定目录下/home/flink

[root@HadoopNode00 ~]# mkdir /home/flink
[root@HadoopNode00 ~]# tar -zxf flink-1.8.1-bin-scala_2.11.tgz -C /home/flink/
[root@HadoopNode00 ~]# cd /home/flink/flink-1.8.1/
[root@HadoopNode00 flink-1.8.1]# ls -l
total 628
drwxr-xr-x. 2 502 games   4096 Dec 23 11:26 bin  # 执行脚本
drwxr-xr-x. 2 502 games   4096 Jun 25 16:10 conf # 配置目录
drwxr-xr-x. 6 502 games   4096 Dec 23 11:26 examples # 案例
drwxr-xr-x. 2 502 games   4096 Dec 23 11:26 lib # 系统依赖jars
-rw-r--r--. 1 502 games  11357 Jun 14  2019 LICENSE
drwxr-xr-x. 2 502 games   4096 Dec 23 11:26 licenses
drwxr-xr-x. 2 502 games   4096 Jun 24 23:02 log #系统启动日志，出错可以查看
-rw-r--r--. 1 502 games 596009 Jun 24 23:02 NOTICE
drwxr-xr-x. 2 502 games   4096 Dec 23 11:26 opt # Flink第三方可选jar，当需要的时候拷贝到lib下
-rw-r--r--. 1 502 games   1308 Jun 14  2019 README.txt
[root@HadoopNode00 flink-1.8.1]# tree conf/
conf/
├── flink-conf.yaml  # 主配置文件 √
├── log4j-cli.properties
├── log4j-console.properties
├── log4j.properties
├── log4j-yarn-session.properties
├── logback-console.xml
├── logback.xml
├── logback-yarn.xml
├── masters # 主节点信息，在单机环境下无需配置
├── slaves  # 计算节点信息 √
├── sql-client-defaults.yaml
└── zoo.cfg

2，配置slaves配置文件

[root@HadoopNode00 flink-1.8.1]# vi conf/slaves

HadoopNode00

3,配置flink-conf.yaml

#==============================================================================
# Common
#==============================================================================
jobmanager.rpc.address: HadoopNode00
# 表示从机的计算资源数
taskmanager.numberOfTaskSlots: 4 
# 配置任务的默认计算并行度 
parallelism.default: 3

4，启动Flink服务

[root@HadoopNode00 flink-1.8.1]# ./bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host HadoopNode00.
Starting taskexecutor daemon on host HadoopNode00.
[root@HadoopNode00 flink-1.8.1]# jps
10833 TaskManagerRunner
10340 StandaloneSessionClusterEntrypoint
10909 Jps

HA模式

参考：https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/jobmanager_high_availability.html

• 搭建HDFS HA集群，保证正常启动（以前课程）
• 配置HADOOP_CLASSPATH
• 配置FlinkHA（准备HadoopNode01~03）

    [root@HadoopNodeXX ~]# mkdir /home/flink
    [root@HadoopNodeXX ~]# tar -zxf flink-1.8.1-bin-scala_2.11.tgz -C /home/flink
    [root@HadoopNodeXX ~]# cd /home/flink/
    [root@HadoopNodeXX flink]# cd flink-1.8.1/
    [root@HadoopNodeXX flink-1.8.1]# vi conf/masters
    HadoopNode01:8081
    HadoopNode02:8081
    HadoopNode03:8081
    [root@HadoopNodeXX flink-1.8.1]# vi conf/slaves
    HadoopNode01
    HadoopNode02
    HadoopNode03
    [root@HadoopNodeXX flink-1.8.1]# vi conf/flink-conf.yaml
    taskmanager.numberOfTaskSlots: 4
    parallelism.default: 3
    high-availability: zookeeper
    high-availability.storageDir: hdfs:///flink/ha/
    high-availability.zookeeper.quorum: HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181
    high-availability.zookeeper.path.root: /flink
    high-availability.cluster-id: /default_ns
    
    state.backend: rocksdb
    state.checkpoints.dir: hdfs:///flink-checkpoints
    state.savepoints.dir: hdfs:///flink-savepoints
    state.backend.incremental: false
    state.backend.rocksdb.ttl.compaction.filter.enabled: true

启动Flink集群

    [root@HadoopNode01 flink-1.8.1]# ./bin/start-cluster.sh
    Starting HA cluster with 3 masters.
    Starting standalonesession daemon on host HadoopNode01.
    Starting standalonesession daemon on host HadoopNode02.
    Starting standalonesession daemon on host HadoopNode03.
    Starting taskexecutor daemon on host HadoopNode01.
    Starting taskexecutor daemon on host HadoopNode02.
    Starting taskexecutor daemon on host HadoopNode03.

用户可以通过 ./bin/jobmanager.sh start|stop来模拟主机故障

注意：

XX代表所有节点