Flink 集群

Flink 集群

前期准备资源

3台服务器，flink , JDK8

服务器IP	HOSTNAME	ROLE
192.168.79.8	node01	jobmanager taskmanager
192.168.79.9	node02	taskmanager
192.168.79.10	node03	taskmanager

各服务器配置环境变量

环境变量 /etc/profile

export JAVA_HOME=/usr/local/jdk-1.8.0_231
export FLINK_HOME=/home/tester/flink-1.15.2
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$FLINK_HOME/bin:$PATH

各服务器配置 hostname, hosts

jobmanager 配置 hostname, hosts

tester@node01:~/flink-1.15.2/conf$ cat /etc/hostname 
node01
tester@node01:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node01

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.79.9 node02
192.168.79.10 node03

配置 ~/flink-1.15.2/conf/flink-conf.yaml, 其他key: value保持不变

jobmanager.rpc.address: node01
taskmanager.numberOfTaskSlots: 2
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0
# 此处如果为localhost 会导致slot值不准确
jobmanager.bind-host: 0.0.0.0

配置 ~/flink-1.15.2/conf/masters

node01:8081

配置 ~/flink-1.15.2/conf/workers

node01
node02
node03

taskmanager 配置 hostname, hosts

tester@node02:~/flink-1.15.2/conf$ cat /etc/hostname 
node02
tester@node02:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node02

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.79.8 node01
192.168.79.10 node03

taskmanager 配置 hostname, hosts

tester@node03:~/flink-1.15.2/conf$ cat /etc/hostname 
node03
tester@node03:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node03

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.79.8 node01
192.168.79.9 node02

taskmanagers flink 文件配置

将 192.168.79.8 ~/flink-1.15.2/conf/ 下 flink-conf.yaml, master, workers 拷贝到 192.168.79.9 , 192.168.79.10 对应的flink conf 配置文件且覆盖

启动 flink 集群

只需启动jobmanager的服务器，另外服务器节点会被此节点启动

tester@node01:~/flink-1.15.2/bin$ start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host node01.
tester@node02's password: 
Starting taskexecutor daemon on host node02.
tester@node03's password: 
Starting taskexecutor daemon on host node03.

访问 http://192.168.79.8:8081 如果不成功，需注意防火墙端口8081是否已打开

此处仍需注意观察对应的日志文件。如果有类似 6123连接不上，也许将防火墙端口6123是否打开

停止 flink 集群

只需启动jobmanager的服务器，另外服务器节点会被此节点停止

tester@node01:~/flink-1.15.2/bin$ stop-cluster.sh 
tester@node02's password: 
Stopping taskexecutor daemon (pid: 10412) on host node02.
tester@node03's password: 
Permission denied, please try again.
tester@node03's password: 
Stopping taskexecutor daemon (pid: 9492) on host node03.
Stopping standalonesession daemon (pid: 10650) on host node01.