Flink on yarn 环境从0-1

1.环境信息

[root@kafka2 ~]# cat /etc/hosts
10.0.83.71 kafka1 node-1
10.0.83.72 kafka2 node-2
10.0.83.73 kafka3 node-3

---

JobManager: 10.0.83.71
TaskManager: 10.0.83.72/ 10.0.83.73

java版本

我代码中后来用的是java11+flink1.14,如果想用java8那就代码中java的依赖版本切换成1.8即可
需要用java11可以自己安装下
ubuntu上: apt-get install openjdk-11-jdk -y
centos上: yum install java-11-openjdk-devel -y

文档中的代码地址:

https://gitee.com/mafei2018/flink2022.git

---

2.部署hadoop集群,设置免密登录

之前写的很详细了,可以参考:https://blog.51cto.com/mapengfei/2546950

3.flink集群部署

找到最新版本: https://flink.apache.org/downloads.html#apache-flink-1144
进入master节点:

[root@kafka1 ~]# cd /opt/
[root@kafka1 opt]# wget https://www.apache.org/dyn/closer.lua/flink/flink-1.14.4/flink-1.14.4-bin-scala_2.11.tgz
[root@kafka1 opt]# tar zxvf  flink-1.14.4-bin-scala_2.11.tgz 
[root@kafka1 ~]# cd /opt/flink-1.14.4/
[root@kafka1 opt]# cd flink-1.14.4/
[root@kafka1 flink-1.14.4]# vim conf/flink-conf.yaml

修改配置文件,进入conf目录

1.修改fink-conf.yaml文件,修改jobmanager.rpc.address 改为kafka1

[root@kafka1 flink-1.14.4]# sed -i 's/jobmanager.rpc.address: localhost/jobmanager.rpc.address: kafka1/g' conf/flink-conf.yaml

2.修改master文件,修改localhost:8081 改为 kafka1:8081
[root@kafka1 flink-1.14.4]# sed -i 's/localhost/kafka1/g' conf/masters

3.修改workers文件,修改localhost改为kafka2和kafka3
[root@kafka1 flink-1.14.4]# sed -i 's/localhost/kafka2\nkafka3/g' conf/workers

将master节点上修改好的flink分发到另外2台机器上

[root@kafka1 flink-1.14.4]# scp -r /opt/flink-1.14.4 kafka2:/opt
[root@kafka1 flink-1.14.4]# scp -r /opt/flink-1.14.4 kafka3:/opt

4.启动集群:

[root@kafka1 flink-1.14.4]# ./bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host kafka1.
Starting taskexecutor daemon on host kafka2.
Starting taskexecutor daemon on host kafka3.

5.开始运行

因为要测试的是nc,所以先到服务器上执行监听端口,新启动一个会话窗口执行

root@kafka1:~# nc -lk 666

## 把打包传到服务器上,flink run 执行

root@kafka1:~# /opt/flink-1.14.4/bin/flink run -c com.mafei.test.StreamWordCountByNc --detached /root/flink2022-1.0.jar

nc那个终端中造一些数据

root@kafka1:~# nc -lk 666
a b
a b1
aa
b2

在task manager上就可以看到了,因为代码中是print,所以在Stdout中可以展示