1. 规划目录:
/home/user/flink
2. 解压文件并重命名
tar zxvf flink-1.11.1-bin-scala_2.11.tgz
mv flink-1.11.1 flink
3. 配置conf
3.1 flink-conf.yaml
在Common部分增加或设置如下:
env.java.home: /usr/java/default
state.savepoints.dir: hdfs:/flink/savepoints
jobmanager.rpc.address: 10.88.0.86
jobmanager.rpc.port: 6123
blob.server.port: 35214
jobmanager.memory.process.size: 1600m
taskmanager.heap.mb: 20480m --TaskManager总共能使用的内存大小
taskmanager.numberOfTaskSlots: 16 --服务器CPU Core数一致,每一台机器上能使用的 CPU 个数
parallelism.default: 1 --集群中的总 CPU个数
3.2 masters
10.88.0.86:8081
3.3 works
10.88.0.87
10.88.0.88
3.4 zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
clientPort=2181
server.1=10.88.1.85:2888:3888
server.2=10.88.1.86:2888:3888
server.3=10.88.1.87:2888:3888
3.5 打包并发送到workers
tar zcvf flink flink.tgz
scp flink.tgz user@10.88.0.87:/home/user
scp flink.tgz user@10.88.0.88:/home/user
3.6 在workers上加压并略作配置的修改
vi flink-conf.yaml
Common部分
+ taskmanager.host: 10.88.0.87
4. 启动和使用
4.1 在所有服务器上配置环境变量
vi /etc/profile
FLINK_HOME=/home/user/flink
HADOOP_HOME=/usr/lib/hadoop
HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
HADOOP_CLASSPATH=`hadoop classpath`
PATH=$PATH:$JAVA_HOME/bin:$SPARK_HOME:$JRE_HOME/bin:$KAFKA_HOME/bin:$PHOENIX_HOME/bin:$FLINK_HOME/bin:$HADOOP_HOME/bin:/usr/local/redis/bin
export JAVA_HOME JRE_HOME PHOENIX_HOME FLINK_HOME KAFKA_HOME SPARK_HOME HADOOP_HOME HADOOP_MAPRED_HOME HADOOP_CLASSPATH CLASS_PATH PATH
4.2 在master上启动集群
start-cluster.sh
检查jps,master上:StandaloneSessionClusterEntrypoint
workers上:TaskManagerRunner
4.3 停止集群
stop-cluster.sh
4.4 启动任务
flink run workers2hbase-0.0.1-SNAPSHOT.jar
flink run workers2hbase-0.0.1-SNAPSHOT.jar -p 2 --并发度
检查有哪些任务
flink list -a
杀掉任务
flink stop 721edfb9b1fff06e8ee63ca9e7b9c183
5. 集群部署后可以做一下检查
start-scala-shell.sh remote 10.88.0.86 8081
Scala> val text = benv.fromElements(
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
"The slings and arrows of outrageous fortune",
"Or to take arms against a sea of troubles,")
Scala> val counts = text
.flatMap { _.toLowerCase.split("\\W+") }
.map { (_, 1) }.groupBy(0).sum(1)
Scala> counts.print()
6.添加jobmanager和taskmanager
添加jobmanager:
bin/jobmanager.sh ((start|start-foreground) cluster) | stop | stop-all
添加taskmanager:
bin/taskmanager.sh start | start-foreground | stop | stop-all
7. 利用yarn进行资源调度
7.1 启动一个YARN session用2个TaskManager(每个TaskManager分配1GB的堆空间)
yarn-session.sh -d -n 2 -jm 1024 -tm 1024 -s 2
参数:-d,后台运行session,flink yarn client将会只提交任务到集群然后关闭自己。注意:在这种情况下,无法使用flink停止yarn session,使用yarn工具来停止yarn session。
-n 2个TaskManager,每个TaskManager内存为1G(-tm)且占用了2个核(-s)
通过yarn application --list 查看状态
查看日志文件:yarn logs -applicationId <application ID>
nohup flink run -c com.du.mflink2hbase workers2hbase-0.0.1-SNAPSHOT.jar --port 8020 &
7.2 以集群模式提交任务,每次都会新建flink集群
flink run -m yarn-cluster -c com.du.mflink2hbase /home/user/lflink-demo-1.0-SNAPSHOT.jar
7.3 问题
Q1:Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=10, maxVirtualCores=4
修改yarn-site.xml
+
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>32</value>
</property>