Flink集群搭建

docker版

https://hub.docker.com/_/flink

安装docker

yum install docker -y

安装docker-compose

yum install docker-compose -y

启动flink

echo '
version: "2.2"
services:
  jobmanager:
    image: flink:latest
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
  taskmanager:
    image: flink:latest
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2
  sql-client:
    image: flink:latest
    command: bin/sql-client.sh
    depends_on:
      - jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        rest.address: jobmanager'>docker-compose.yml
  docker-compose up -d

正常版

免密登录

虽然网上没说，但是看后面集群启动时的操作来看是做了免密登录的。

安装及配置

下载并配置集群

wget http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.8.1/flink-1.8.1-bin-scala_2.12.tgz && tar -zxvf flink-1.8.1-bin-scala_2.12.tgz && cd flink-1.8.1 && vi conf/masters

配置JobManager节点

192.168.1.146:8081
192.168.1.144:8081
192.168.1.141:8081

通过conf/masters配置HA（基于zookeeper）

#==============================================================================
# High Availability
#==============================================================================
# add by Qbit according to https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/jobmanager_high_availability.html
high-availability.cluster-id: /qbit
high-availability.zookeeper.path.root: /flink
# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability: zookeeper

# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
# 
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...) 
#
high-availability.storageDir: hdfs:///flink/ha/

# The list of ZooKeeper quorum peers that coordinate the high-availability
# setup. This must be a list of the form:
# "host1:clientPort,host2:clientPort,..." (default clientPort: 2181)
#
high-availability.zookeeper.quorum: 192.168.1.146:2181,192.168.1.144:2181,192.168.1.141:2181

启动

bin/start-cluster.sh

启动后8081端口可访问说明启动成功