seatunnel入门案例，集群模式

qzWsong

已于 2023-06-19 16:07:28 修改

阅读量2.6k

点赞数

分类专栏： seatunnel 文章标签： java 开发语言

于 2023-06-13 17:32:40 首次发布

本文链接：https://blog.csdn.net/java_creatMylief/article/details/131192388

版权

seatunnel 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

安装部署

解压

tar -zxvf apache-SeaTunnel-incubating-2.3.1-bin.tar.gz

环境变量

vi /etc/profile

export SEATUNNEL_HOME=/data/soft/seatunnel-2.3.1
export PATH=$PATH:${SEATUNNEL_HOME}/bin

安装plugin

# 备份原有的组件配置
cp  ${SEATUNNEL_HOME}/config/plugin_config ${SEATUNNEL_HOME}/config/plugin_config_bak
# 创建一个新的
vi  ${SEATUNNEL_HOME}/config/plugin_config
# 只安装这三种插件，因为他的插件太多了，下载很慢，用哪个下哪个
--connectors-v2--
connector-jdbc
connector-kafka
connector-console
--end--

# 运行安装命令

sh ${SEATUNNEL_HOME}/bin/install-plugin.sh 2.3.1

添加资源jar包

cd ${SEATUNNEL_HOME}/lib
# 添加以下依赖jar，连接mysql需要使用
mysql-connector-j-8.0.31.jar

SEATUNNEL 配置文件

env：环境设置

source：数据源设置

sink：数据去向设置

transform: 数据转换设置

env {
    execution.parallelism = 1
}
source {
        Kafka {
                topic = "seatunnel_topic"
				format = text
				field_delimiter = "#"
                schema = {
				  fields {
					name = "string"
					age = "int"
				  }
				}
                consumer.group = "yuanqu-group11"
                bootstrap.servers = "hdp01:6667"
                start_mode = "latest"
                kafka.config = {
                  security.protocol = "SASL_PLAINTEXT"
                  sasl.mechanism = "PLAIN"
                  sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required  username=\"XXX\" password=\"XX@XXXX\";"
                }
        }
}
transform {

}
sink {
    jdbc {
		url = "jdbc:mysql://172.16.10.143:3306/test_db"
		driver = "com.mysql.cj.jdbc.Driver"
		user = "root"
		password = "XXXX"
		query = "insert into seatunnel_test(name,age) values(?,?)"
	}
}

运行方式

seatunnel 包括三种执行引擎

seatunnel 引擎(zeta)

本地模式

# -e local 指定本地执行
${SEATUNNEL_HOME}/bin/seatunnel.sh --config test.conf -e local

集群模式

启动集群

# 启动集群,我这是一台机器伪集群不需要任何配置，如果配置看https://seatunnel.apache.org/docs/2.3.1/seatunnel-engine/deployment
nohup ${SEATUNNEL_HOME}/bin/seatunnel-cluster.sh & 1>/dev/null 2>&1

启动集群任务

# -e local 指定本地执行
nohup ${SEATUNNEL_HOME}/bin/seatunnel.sh --config test.conf & 1>/dev/null 2>&1

停止任务

启动任务之后会有一个jobid，如果直接kill 进程是不管用的，seatunnel任务依然会执行，需要使用 -can jobid 结束任务

#我们需要使用  -can, --cancel-job              Cancel job by JobId
${SEATUNNEL_HOME}/bin/seatunnel.sh -can 720924469203304449

 Authorized only. All activity will be monitored and reported 
Last login: Mon Jun 19 14:27:58 2023 from 10.10.10.14
[manager@hdpcli01 ~]$ ${SEATUNNEL_HOME}/bin/seatunnel.sh --list
JAVA_OPTS: -Dhazelcast.client.config=/data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml -Dseatunnel.config=/data/soft/seatunnel-2.3.1/config/seatunnel.yaml -Dhazelcast.config=/data/soft/seatunnel-2.3.1/config/hazelcast.yaml -Dlog4j2.configurationFile=/data/soft/seatunnel-2.3.1/config/log4j2_client.properties -Dseatunnel.logs.path=/data/soft/seatunnel-2.3.1/logs -Dseatunnel.logs.file_name=seatunnel-starter-client -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-1757821675590923600 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/seatunnel.yaml
Jun 19, 2023 4:07:03 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
INFO: seatunnel.home is /home/manager
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/hazelcast.yaml
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
Jun 19, 2023 4:07:03 PM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /data/soft/seatunnel-2.3.1/config/hazelcast-client.yaml
2023-06-19 16:07:04,474 INFO  com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
2023-06-19 16:07:04,562 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-06-19 16:07:04,563 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-06-19 16:07:04,597 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
2023-06-19 16:07:04,601 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:5801
2023-06-19 16:07:04,640 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-06-19 16:07:04,640 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:da118827-3f65-4fe6-b769-1ca26c71d693, server version: 5.1, local address: /127.0.0.1:57914
2023-06-19 16:07:04,642 INFO  com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-06-19 16:07:04,662 INFO  com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] 

Members [1] {
        Member [localhost]:5801 - da118827-3f65-4fe6-b769-1ca26c71d693
}

2023-06-19 16:07:04,694 INFO  com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
Job ID              Job Name   Job Status  Submit Time              Finished Time            
------------------  ---------  ----------  -----------------------  -----------------------  
723074781175349249  SeaTunnel  CANCELED    2023-06-19 15:21:02.191  2023-06-19 15:21:22.951
723074554087342081  SeaTunnel  CANCELED    2023-06-19 15:20:08.102  2023-06-19 15:20:28.864
723073982751834113  SeaTunnel  CANCELED    2023-06-19 15:17:51.834  2023-06-19 15:18:03.142
723072313091686401  SeaTunnel  CANCELED    2023-06-19 15:11:13.734  2023-06-19 15:11:24.541
723071390659379201  SeaTunnel  CANCELED    2023-06-19 15:07:33.821  2023-06-19 15:09:55.188
723068648918351873  SeaTunnel  CANCELED    2023-06-19 14:56:40.13   2023-06-19 14:57:11.011
723067564967591937  SeaTunnel  CANCELED    2023-06-19 14:52:21.739  2023-06-19 14:52:34.226

2023-06-19 16:07:04,817 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-06-19 16:07:04,822 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:da118827-3f65-4fe6-b769-1ca26c71d693, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:57914->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-06-19 16:07:04.741, lastWriteTime=2023-06-19 16:07:04.740, closedTime=2023-06-19 16:07:04.818, connected server version=5.1}
2023-06-19 16:07:04,822 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-06-19 16:07:04,825 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-06-19 16:07:04,826 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
[manager@hdpcli01 ~]$