数据湖仓一体(三)安装seatunnel

 上传安装包到/opt/software目录并解压

[bigdata@node106 software]$ tar -zxvf apache-seatunnel-2.3.4  -C /opt/services/ 

配置环境变量

[bigdata@node106 ~]$ sudo vim /etc/profile.d/bigdata_env.sh

export SEA_HOME=/opt/services/apache-seatunnel-2.3.4
export PATH=$PATH:$JAVA_HOME/bin:$ZK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$KAFKA_HOME/bin:$SEA_HOME/bin

分发环境变量

[bigdata@node106 ~]$ sudo ./bin/xsync /etc/profile.d/bigdata_env.sh

刷新环境变量，5台机器上执行

[bigdata@node106 ~]$ source /etc/profile

修改seatunnel-env.sh配置文件

[bigdata@node106 ~]$ vim /opt/services/apache-seatunnel-2.3.4/config/seatunnel-env.sh

# Home directory of spark distribution.
SPARK_HOME=${SPARK_HOME:-/opt/services/spark-3.3.1}
# Home directory of flink distribution.
FLINK_HOME=${FLINK_HOME:-/opt/services/flink-1.17.2}

修改seatunnel.yaml配置文件

plugin-config:
          namespace: /sea-checkpoints
          storage.type: hdfs
          fs.defaultFS: hdfs://mycluster:8020 # Ensure that the directory has written permission
          seatunnel.hadoop.dfs.nameservices: mycluster
          seatunnel.hadoop.dfs.ha.namenodes.mycluster: nn1,nn2
          seatunnel.hadoop.dfs.namenode.rpc-address.mycluster.nn1: node106:8020
          seatunnel.hadoop.dfs.namenode.rpc-address.mycluster.nn2: node107:8020
          seatunnel.hadoop.dfs.client.failover.proxy.provider.mycluster: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

修改hazelcast.yaml配置文件

[bigdata@node106 ~]$ vim /opt/services/apache-seatunnel-2.3.4/config/hazelcast.yaml

        member-list:
          - node106
          - node107
          - node108

修改plugin_config配置文件,选择需要的连接器下载

[bigdata@node106 ~]$ vim /opt/services/apache-seatunnel-2.3.4/config/plugin_config  

上传mysql-connector-java-8.0.30.jar包到lib、plugins目录下

[bigdata@node101 software]$ cp mysql-connector-java-8.0.30.jar /opt/services/apache-seatunnel-2.3.4/lib/ 
[bigdata@node101 software]$ cp mysql-connector-java-8.0.30.jar /opt/services/apache-seatunnel-2.3.4/plugins/ 

创建logs文件夹

[bigdata@node106 ~]$ mkdir -p $SEA_HOME/logs 

copy到其他节点

[bigdata@node107 ~]$ scp -r bigdata@node106: /opt/services/apache-seatunnel-2.3.4/ /opt/services/apache-seatunnel-2.3.4/                  
[bigdata@node108 ~]$ scp -r bigdata@node106: /opt/services/apache-seatunnel-2.3.4/ /opt/services/apache-seatunnel-2.3.4/  

编写sea.sh启动停止脚本

#!/bin/bash

if [ $# -lt 1 ]
then
    echo "No Args Input...[start,stop]"
    exit ;
fi
case $1 in
"start")
        echo ==================== 启动seatunnel服务 =========================
        echo ==================== 启动node106 =========================
        ssh node106 "$SEA_HOME/bin/seatunnel-cluster.sh -d"
        echo ==================== 启动node107 =========================
        ssh node107 "$SEA_HOME/bin/seatunnel-cluster.sh -d"
        echo ==================== 启动node108 =========================
        ssh node108 "$SEA_HOME/bin/seatunnel-cluster.sh -d"
;;
"stop")
        echo ==================== 关闭seatunnel服务 =========================
       echo ==================== 关闭node106 ============================
        ssh node106 "$SEA_HOME/bin/stop-seatunnel-cluster.sh"
       echo ==================== 关闭node107 ============================
        ssh node107 "$SEA_HOME/bin/stop-seatunnel-cluster.sh"
       echo ==================== 关闭node108 ============================
        ssh node108 "$SEA_HOME/bin/stop-seatunnel-cluster.sh"


;;
*)
    echo "Input Args Error...[start,stop]"
;;
esac

授权

[bigdata@node106 bin]$ chmod +x sea.sh

分发到其他机器

[bigdata@node106 bin]$ xsync  sea.sh 

启动seatunnel

[bigdata@node106 bin]$ sea.sh start