Seatunnel部署

基础环境

Seatunnel version 2.3.3
Seatunnel Web 1.0.1

JDK

• version >= 1.8.0 下载地址
• 环境变量

➜  vim .zshrc
JAVA_HOME=/usr/local/jdk-17.0.12
export PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH:.
export CLASSPATH

➜  ~ java -version
java version "17.0.12" 2024-07-16 LTS
Java(TM) SE Runtime Environment (build 17.0.12+8-LTS-286)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.12+8-LTS-286, mixed mode, sharing)

Maven

• 下载地址

maven 版本： 3.6.3 ⚠️⚠️⚠️太高版本好像不支持 dependency:get 在安装插件时会下载不到指定目录⚠️⚠️⚠️

• 环境变量

➜  ~ vim ~/.zshrc
M2_HOME=/usr/local/apache-maven-3.6.3
export PATH=$M2_HOME/bin:$PATH

➜  ~ mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/apache-maven-3.6.3
Java version: 17.0.12, vendor: Oracle Corporation, runtime: /usr/local/jdk-17.0.12
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "5.14.0-480.el9.aarch64", arch: "aarch64", family: "unix"

• 配置代理

 <proxies>
    <!-- proxy
     | Specification for one proxy, to be used in connecting to the network.
     | -->
    <proxy>
      <id>http</id>
      <active>true</active>
      <protocol>http</protocol>
      <username></username>
      <password></password>
      <host>192.168.10.2</host>
      <port>8889</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
    <proxy>
      <id>https</id>
      <active>true</active>
      <protocol>https</protocol>
      <username></username>
      <password></password>
      <host>192.168.10.2</host>
      <port>8889</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>
  </proxies>

Seatunnel 安装

安装插件

下载地址

• 下载

export version="2.3.3"
wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

• 配置环境变量

如果 SeaTunnel Engine 和 Seatunnel Web 必须配置

export SEATUNNEL_HOME=/root/apache-seatunnel-2.3.3
export PATH=$PATH:$SEATUNNEL_HOME/bin

安装插件

# 修改插件脚本,替换使用本地mvn
➜  apache-seatunnel-2.3.6 vim bin/install-plugin.sh
:%s/${SEATUNNEL_HOME}\/mvnw/mvn/g
# 编辑 插件文件
➜  apache-seatunnel-2.3.6 vim config/plugin_config
# 安装插件
➜  apache-seatunnel-2.3.6 sh bin/install-plugin.sh 2.3.6
# 创建插件文件夹
➜  apache-seatunnel-2.3.6 mkdir connectors/{flink,flink-sql,seatunnel,spark}
# 测试安装状态
➜  apache-seatunnel-2.3.6 ./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

# 打印如下信息表示运行成功
***********************************************
           Job Statistic Information
***********************************************
Start Time                : 2024-08-19 20:47:46
End Time                  : 2024-08-19 20:47:47
Total Time(s)             :                   1
Total Read Count          :                  32
Total Write Count         :                  32
Total Failed Count        :                   0
***********************************************

部署 SeaTunnel Engine

官方文档

• 配置环境变量

export SEATUNNEL_HOME=/root/apache-seatunnel-2.3.3
export PATH=$PATH:$SEATUNNEL_HOME/bin

• 修改配置文件
• 创建日子文件夹 mkdir -p $SEATUNNEL_HOME/logs
• 启动 ./bin/seatunnel-cluster.sh -d
• 查看日志 tail -f $SEATUNNEL_HOME/logs/seatunnel-engine-server.log

部署 Seatunnel Web

seatunnel 版本 要求 2.3.3

安装教程

• 下载地址
• 下载 1.0.1 并解压
• 初始化数据库
• 修改配置文件
• 下载 mysql-connector-java-${version}.jar 到 lib 下
• 运行bin/download_datasource.sh 下载数据源jar包，并复制到 lib 下
• 拷贝seatunnel 配置文件

cp $SEATUNNEL_HOME/config/hazelcast-client.yaml conf
cp $SEATUNNEL_HOME/connectors/plugin-mapping.properties conf

• 启动 sh bin/seatunnel-backend-daemon.sh start
• 访问 http://127.0.0.1:8801/ui/

常见问题

单机版seatunnel启动报错
无数据源问题

V2.3.4

• Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType=‘seatunnel’, pluginType=‘source’, pluginName=‘FakeSource’} not found.

cp connectors/seatunnel/* lib/ 将connecter 放到 lib 下