[大数据]flink1.15.4集群部署

环境准备

IP	Hostname	Role
10.0.0.1	flink-01	JobManager,TaskManager
10.0.0.2	flink-02	TaskManager
10.0.0.3	flink-03	TaskManager

安装JDK

yum install java-11-openjdk.x86_64  java-11-openjdk-devel.x86_64 -y

配置hosts

vim /etc/hosts
10.0.0.1 flink-01
10.0.0.2 flink-02
10.0.0.3 flink-03

配置免密

1. 在flink-01上面生成公钥和私钥对，按照提示一直按下回车键，直到生成密钥对为止。

ssh-keygen -t rsa

2. 将flink-01上生成的公钥复制到flink-01，flink-02，flink-03

ssh-copy-id root@flink-01
ssh-copy-id root@flink-02
ssh-copy-id root@flink-03

1. Flink集群部署

Master节点操作

1. 获取Flink安装包

cd /opt && wget https://archive.apache.org/dist/flink/flink-1.15.4/flink-1.15.4-bin-scala_2.12.tgz

2. 解压压缩包并创建软链接

tar xf flink-1.15.4-bin-scala_2.12.tgz
ln -s flink-1.15.4 flink

3. 修改配置文件flink-conf.yaml

vim /opt/flink/conf/flink-conf.yaml
jobmanager.rpc.address: 10.0.0.1
jobmanager.bind-host: 0.0.0.0
taskmanager.bind-host: 0.0.0.0
taskmanager.host: flink-01
taskmanager.memory.process.size: 14400m #taskmanager可用内存
taskmanager.numberOfTaskSlots: 3 #taskmanager任务数量 一般跟cpu数量相同
rest.address: flink-01
rest.bind-address: 0.0.0.0

4. 修改节点文件

vim /opt/flink/conf/masters
10.0.0.1:8081

vim/opt/flink/conf/workers
10.0.0.1
10.0.0.2
10.0.0.3

5. 将flink包发送到flink-02，flink-03

scp -r flink-1.15.4/ root@flink-02:/opt
scp -r flink-1.15.4/ root@flink-03:/opt

worker节点操作（flink-02 flink-03同步操作）

1. 创建软链接

cd /opt && ln -s flink-1.15.4 flink

2. 修改配置文件

taskmanager.host: 10.0.0.2/10.0.0.3 #根据服务器本身ip进行修改

启动flink集群（Master节点进行启动）

cd /opt/flink/bin && ./start-cluster.sh