flink的StandaloneHA部署flink1.8.0+hadoop2.6.5+zookeeper3.4.14

1.说明

目标

1）三台机器Centos7

机器	JobManager	TaskManager	JouralNode	DataNode	NameNode	Zkfc	Zookeeper
192.168.11.131	√	√	√	√	√	√	√
192.168.11.132	-	√	√	√	√	√	√
192.168.11.133	√	√	√	√	-	-	√

即JobManager部署131、132两台服务器，而NameNode部署131、133（Zkfc同步部署）

2）实现自启动上述服务

前提

1）jdk已安装
2）zookeeper的安装、自启动请参考：https://blog.csdn.net/sndayYU/article/details/100537922、https://blog.csdn.net/sndayYU/article/details/114047145

具体操作

1.免密登录

参考：https://blog.csdn.net/sndayYU/article/details/115036682
用户flink，别名分别时node1、node2、node3

注意
同理131的root免密到132、133

2.相关目录

/flink/为安装flink、hadoop、zookeeper安装目录，/data/为其数据及日志目录

chown flink:flink /data
chown flink:flink /flink
chmod 755 /data
chmod 755 /flink

3.zookeeper

基本配置

zookeeper的安装、自启动请参考：https://blog.csdn.net/sndayYU/article/details/100537922、https://blog.csdn.net/sndayYU/article/details/114047145

• 安装目录/flink/zookeeper-3.4.14/
• data目录/data/zookeeper/data/
• log目录/data/zookeeper/log/

dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs

server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

自启动

（参考：https://blog.csdn.net/sndayYU/article/details/114047145）：
systemctl start zookeeper

脚本/lib/systemd/system/zookeeper.service

[Unit]
Description=zookeeper
[Service]
Type=forking
Environment=ZOO_LOG_DIR=/data/zookeeper
Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
ExecStart=/flink/zookeeper-3.4.14/bin/zkServer.sh start
ExecStop=/flink/zookeeper-3.4.14/bin/zkServer.sh stop
PrivateTmp=true
User=flink
Group=flink
 
[Install]
WantedBy=multi-user.target

4.hadoop

基本配置

解压，配置core-site.xml、hdfs-site.xml、slaves、删除masters，注意检查目录权限是否为flink。
传到另外2台。

core-site.xml

<property>
        <name>fs.defaultFS</name>
		<!-- 要与hdfs-site.xml的dfs.nameservices 保持一直 -->
        <value>hdfs://hadoopha</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
		<!-- uat配置 file:/data/hadoop/tmp，不知道会不会有问题 -->
        <value>/data/hadoop/tmp</value>
    </property>
	<property>
        <name>ha.zookeeper.quorum</name>
		<value>node1:2181,node2:2181,node3:2181</value>
    </property>
    <property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>150000</value>
    </property>

hdfs-site.xml

<property>
	    <!-- 不需要file://会报错 -->
        <name>dfs.namenode.name.dir</name>
        <value>file:///data/hadoop/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///data/hadoop/data</value>
    </property>
	<!-- 采用默认值3 -->
	 <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
	<!-- core-site.xml有使用到 -->
    <property>
        <name>dfs.nameservices</name>
        <value>hadoopha</value>
    </property>
	<property>
        <!-- dfs.ha.namenodes.EXAMPLENAMESERVICE -->
        <name>dfs.ha.namenodes.hadoopha</name>
        <value>nn1,nn2</value>
    </property>
	<property>
        <!-- <name>dfs.namenode.rpc-address</name> -->
		<name>dfs.namenode.rpc-address.hadoopha.nn1</name>
        <value>node1:9000</value>
    </property>
	<property>
        <name>dfs.namenode.http-address.hadoopha.nn1</name>
        <value>node1:50070</value>
    </property>
	<property>
        <!-- <name>dfs.namenode.rpc-address</name> -->
		<name>dfs.namenode.rpc-address.hadoopha.nn2</name>
        <value>node2:9000</value>
    </property>
	<property>
        <name>dfs.namenode.http-address.hadoopha.nn2</name>
        <value>node2:50070</value>
    </property>
	<property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
	<property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://node1:8485;node2:8485;node3:8485/hadoopha</value>
    </property>
	<!-- 指定JournalNode在本地磁盘存放数据的位置 -->  
	<property>  
	    <name>dfs.journalnode.edits.dir</name>  
	    <value>/data/hadoop/journal</value>  
	</property>  
	<property>  
	    <name>dfs.client.failover.proxy.provider.hadoopha</name>  
	    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>  
	</property> 
	 <property>
        <name>dfs.ha.fencing.methods</name>
        <value>shell(/bin/true)</value>
    </property>
	<property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>
	
	<property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>30000</value>
    </property>

[root@node1 ~]# cat /etc/profile.d/my.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
export CALSSPATH=$JAVA_HOME/lib/*.*
export PATH=$PATH:$JAVA_HOME/bin

export HADOOP_HOME=/flink/hadoop-2.6.5
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

初始化

注意:若是不能一次性搞定，需要清空/data/hadoop目录，及zookeeper的相关数据

1）my.sh环境变量，hadoop version验证下，可以重启下服务器
2）3台启动zookeeper，及 切换flink 执行journalnode
/flink/hadoop-2.6.5/sbin/hadoop-daemon.sh start journalnode
3）node1:
hdfs namenode -format
/flink/hadoop-2.6.5/sbin/hadoop-daemon.sh start namenode
4) node2:
hdfs namenode -bootstrapStandby
5) node1:
hdfs zkfc -formatZK
/flink/hadoop-2.6.5/sbin/start-dfs.sh
6) 最后访问node1:50070、node2:50070

自启动

略

5.flink

基本配置

flink用户解压，配置flink-conf.yaml、slaves、masters，
hadoop的flink-shaded-hadoop-2-uber-2.6.5-7.0.jar放入flink的lib目录
传到另外2台服务器

./flink-1.8.0/log目录一定要有，否则启动会报错

flink-conf.yaml

high-availability: zookeeper
high-availability.storageDir: hdfs:///hadoopha/ha/
high-availability.zookeeper.quorum: node1:2181,node2:2181,node3:2181
state.backend: filesystem
state.checkpoints.dir: hdfs:///hadoopha/flink-checkpoints

slaves

node1
node2
node3

masters

node1:8081
node3:8081

启动

[flink@node1 bin]$ ./start-cluster.sh
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host node1.
Starting standalonesession daemon on host node3.
Starting taskexecutor daemon on host node1.
Starting taskexecutor daemon on host node2.
Starting taskexecutor daemon on host node3.
[flink@node1 bin]$ jps -----验证是否启动正确

访问：http://node1:8081/、http://node3:8081/

自启动

[root@node1 ~]# cat /lib/systemd/system/flink-jobmanager.service
[Unit]
Description=flink-jobmanager
[Service]
Type=forking
Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
Environment=HADOOP_CONF_DIR=/flink/hadoop-2.6.5/etc/hadoop
ExecStart=/flink/flink-1.8.0/bin/jobmanager.sh start
ExecStop=/flink/flink-1.8.0/bin/jobmanager.sh stop
PrivateTmp=true
User=flink
Group=flink
 
[Install]
WantedBy=multi-user.target
[root@node1 ~]# cat /lib/systemd/system/flink-taskmanager.service
[Unit]
Description=flink-taskmanager
[Service]
Type=forking
Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
Environment=HADOOP_CONF_DIR=/flink/hadoop-2.6.5/etc/hadoop
ExecStart=/flink/flink-1.8.0/bin/taskmanager.sh start
ExecStop=/flink/flink-1.8.0/bin/taskmanager.sh stop
PrivateTmp=true
User=flink
Group=flink
 
[Install]
WantedBy=multi-user.target

// 3.scp传到node2、node3
// 4.systemctl daemon-reload

node1、node3: systemctl start flink-jobmanager
node1、node2、node3: systemctl start flink-taskmanager
及systemctl enable xxxx

最终结果

自启动都配置好了后，重启3台电脑，可以发现都正确了，可以上传任务执行jar包

常见错误

• /etc/sudoers里面的权限：flink ALL=(ALL) NOPASSWD: ALL
• nn1和nn2需要配置正确
• zookeeper当测试时不用集群也是可以的，上面的集群配置相应需要做修改
• slaves里面配置的hosts就是datanode节点，start-dfs.sh启动时起作用