Flink 集群
前期准备资源
3台服务器,flink , JDK8
服务器IP | HOSTNAME | ROLE |
---|---|---|
192.168.79.8 | node01 | jobmanager taskmanager |
192.168.79.9 | node02 | taskmanager |
192.168.79.10 | node03 | taskmanager |
各服务器配置环境变量
环境变量 /etc/profile
export JAVA_HOME=/usr/local/jdk-1.8.0_231
export FLINK_HOME=/home/tester/flink-1.15.2
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$FLINK_HOME/bin:$PATH
各服务器配置 hostname, hosts
jobmanager 配置 hostname, hosts
tester@node01:~/flink-1.15.2/conf$ cat /etc/hostname
node01
tester@node01:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node01
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.79.9 node02
192.168.79.10 node03
配置 ~/flink-1.15.2/conf/flink-conf.yaml, 其他key: value保持不变
jobmanager.rpc.address: node01
taskmanager.numberOfTaskSlots: 2
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0
# 此处如果为localhost 会导致slot值不准确
jobmanager.bind-host: 0.0.0.0
配置 ~/flink-1.15.2/conf/masters
node01:8081
配置 ~/flink-1.15.2/conf/workers
node01
node02
node03
taskmanager 配置 hostname, hosts
tester@node02:~/flink-1.15.2/conf$ cat /etc/hostname
node02
tester@node02:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node02
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.79.8 node01
192.168.79.10 node03
taskmanager 配置 hostname, hosts
tester@node03:~/flink-1.15.2/conf$ cat /etc/hostname
node03
tester@node03:~/flink-1.15.2/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 node03
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.79.8 node01
192.168.79.9 node02
taskmanagers flink 文件配置
将 192.168.79.8 ~/flink-1.15.2/conf/ 下 flink-conf.yaml, master, workers 拷贝到 192.168.79.9 , 192.168.79.10 对应的flink conf 配置文件且覆盖
启动 flink 集群
只需启动jobmanager
的服务器,另外服务器节点会被此节点启动
tester@node01:~/flink-1.15.2/bin$ start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host node01.
tester@node02's password:
Starting taskexecutor daemon on host node02.
tester@node03's password:
Starting taskexecutor daemon on host node03.
访问 http://192.168.79.8:8081 如果不成功,需注意防火墙端口8081是否已打开
此处仍需注意观察对应的日志文件。如果有类似 6123连接不上,也许将防火墙端口6123是否打开
停止 flink 集群
只需启动jobmanager
的服务器,另外服务器节点会被此节点停止
tester@node01:~/flink-1.15.2/bin$ stop-cluster.sh
tester@node02's password:
Stopping taskexecutor daemon (pid: 10412) on host node02.
tester@node03's password:
Permission denied, please try again.
tester@node03's password:
Stopping taskexecutor daemon (pid: 9492) on host node03.
Stopping standalonesession daemon (pid: 10650) on host node01.