SeaTunnel集群安装部署

1.集群准备工作

            

 

2.SeaTunnel安装

(1)下载seatunnel安装包 

Apache SeaTunnel

(2)解压下载好的tar.gz包

tar -zxvf /export/server/apache-seatunnel-2.3.3-bin.tar.gz -C ./

(3)查看Seatunnel使用的脚本

cd /export/server/apache-seatunnel-2.3.3目录下

install-plugin.sh                              --安装连接器脚本

seatunnel-cluster.sh                           -–集群模式启动脚本

seatunnel-cluster.sh                           --本地模式启动脚本

start-seatunnel-flink-13-connector-v2.sh       –-flink1.2-1.4版本引擎启动脚本

start-seatunnel-flink-15-connector-v2.sh       –-flink1.5-1.6版本引擎启动脚本

start-seatunnel-spark-2-connector-v2.sh        –-saprk2.x版本引擎启动脚本

start-seatunnel-spark-3-connector-v2.sh        –-saprk3.x版本引擎启动脚本

(4)下载连接器

cd /export/server/apache-seatunnel-2.3.3

./bin/install-plugin.sh

​​​​​​​3.配置环境变量

在/etc/profile.d/seatunnel.sh中配置环境变量

export SEATUNNEL_HOME=/export/server/apache-seatunnel-2.3.3

export PATH=$PATH:$SEATUNNEL_HOME/bin

source /etc/profile           #使环境变量生效
echo $SEATUNNEL_HOME  #查看变量是否生效

​​​​​​​4.配置 SeaTunnel Engine JVM

将 JVM 选项添加到$SEATUNNEL_HOME/bin/seatunnel-cluster.sh第一行

JAVA_OPTS="-Xms2G -Xmx2G"

​​​​​​​5.配置SeaTunnel

配置$SEATUNNEL_HOME/config/seatunnel.yaml文件

eg:

 seatunnel:

  engine:

    history-job-expire-minutes: 1440

    backup-count: 1

    queue-type: blockingqueue

    print-execution-info-interval: 60

    print-job-metrics-info-interval: 60

    slot-service:

      dynamic-slot: true

    checkpoint:

      interval: 10000

      timeout: 60000

      storage:

        type: hdfs

        max-retained: 3

        plugin-config:

          namespace: /tmp/seatunnel/checkpoint_snapshot

          storage.type: hdfs

          fs.defaultFS: hdfs://cdh01:8020 # Ensure that the directory has written permission

  

​​​​​​​6.配置SeaTunnel引擎

配置$SEATUNNEL_HOME/config/hazelcast.yaml文件

eg:

hazelcast:

  cluster-name: seatunnel

  network:

    rest-api:

      enabled: true

      endpoint-groups:

        CLUSTER_WRITE:

          enabled: true

        DATA:

          enabled: true

    join:

      tcp-ip:

        enabled: true

        member-list:

          - cdh01

          - cdh02

          - cdh03

    port:

      auto-increment: false

      port: 5801

  properties:

    hazelcast.invocation.max.retry.count: 20

    hazelcast.tcp.join.port.try.count: 30

    hazelcast.logging.type: log4j2

    hazelcast.operation.generic.thread.count: 50

​​​​​​​7.配置 SeaTunnel引擎服务器

配置$SEATUNNEL_HOME/config/hazelcast-client.yaml文件

cluster-name客户端必须与 SeaTunnel Engine相同。否则，SeaTunnel Engine 将拒绝客户端请求。

eg:

hazelcast-client:

  cluster-name: seatunnel

  properties:

    hazelcast.logging.type: log4j2

  network:

    cluster-members:

      - cdh01:5801

      - cdh02:5801

      - cdh03:5801

​​​​​​​8.部署SeaTunnel分布式集群

(1)拷贝安装包和配置文件

cd /export/server

   scp -r apache-seatunnel-2.3.3/ root@cdh02:$PWD

   scp -r apache-seatunnel-2.3.3/ root@cdh03:$PWD

cd /etc/profile.d/

   scp /etc/profile.d/seatunnel.sh root@cdh02:$PWD

   scp /etc/profile.d/seatunnel.sh root@cdh03:$PWD

使各个节点环境变量生效(ALL)

   source /etc/profile           #使环境变量生效
   echo $SEATUNNEL_HOME  #查看变量是否生效

(2)启动SeaTunnel集群

mkdir -p $SEATUNNEL_HOME/logs  -- 如果有请忽略

nohup $SEATUNNEL_HOME/bin/seatunnel-cluster.sh 2>&1 &   -- 每个节点启动集群

$SEATUNNEL_HOME/bin/stop-seatunnel-cluster.sh  -- 停止集群

日志将写入$SEATUNNEL_HOME/logs/seatunnel-engine-server.log

(3)任务提交命令

$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template

(4)任务停止命令

在$SEATUNNEL_HOME/logs/seatunnel-engine-server.log日志中查找运行的job_id

${SEATUNNEL_HOME}/bin/seatunnel.sh -can 749188983002497026   --job_id