seatuunel学习

1. 本地打包执行 bin/seatunnel.sh --config ./config/v2.batch.config.template -e local报错：
   
   解决方案：
   seatunnel启动脚本报错是因为jvm_options文件在windows系统下的换行符导致，在linux下执行的config底下

sed -i 's/\r//' * 

即可，或者修改源码的换行符

2.执行 bin/seatunnel.sh --config ./config/v2.batch.config.template -e loca
Caused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType=‘seatunnel’, pluginType=‘source’, pluginName=‘FakeSource’} not found.

要将connectors的包缺少依赖包找不到执行 bin/seatunnel.sh --config ./config/v2.batch.config.template -e local
3.Caused by: java.lang.NullPointerException
at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.getPrimaryKey(MySqlDialect.java:130)
这个是把包复制到了lib包下面导致的
3.seatuunel web启动和打包报tsx报错

"strict": false, 这个tsconfig.json改成false   
  package.json  这个改成  "build:prod": "vite build --mode production", ```

4.本地启动
后端：
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelServer
-DSEATUNNEL_HOME=E:\jituab1.0\apache seatunnel\seatunnel\seatunnel-dist\target\apache-seatunnel-2.3.6-bin.tar\apache-seatunnel-2.3.6-bin\apache-seatunnel-2.3.6
![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/bed404de05c349099550135f28d8624a.png)
web:
org.apache.seatunnel.app.SeatunnelApplication
-DSEATUNNEL_HOME=E:\jituab1.0\apache seatunnel\seatunnel\seatunnel-dist\target\apache-seatunnel-2.3.6-bin.tar\apache-seatunnel-2.3.6-bin\apache-seatunnel-2.3.6
E:\jituab1.0\apache seatunnel\seatunnel-web
ST_WEB_BASEDIR_PATH=E:\jituab1.0\apache seatunnel\seatunnel-web\seatunnel-web-dist\target\apache-seatunnel-web-1.0.1-SNAPSHOT\apache-seatunnel-web-1.0.1-SNAPSHOT


![在这里插入图片描述](https://i-blog.csdnimg.cn/direct/b323d930584245b997360d0c0048551a.png)

5服务器启动web服务
sudo sh /opt/seatunnel/apache-seatunnel-web-1.0.0-bin/bin/seatunnel-backend-daemon.sh start



6`在这里插入代码片``

```java
- Unable to create a source for identifier 'Jdbc'.

   private static <T, SplitT extends SourceSplit, StateT extends Serializable>
            SeaTunnelSource<T, SplitT, StateT> createAndPrepareSource(
                    TableSourceFactory factory, ReadonlyConfig options, ClassLoader classLoader) {
       // options.getSourceMap().put("query", "SELECT *  FROM `seatunnel`.`user`");
        TableSourceFactoryContext context = new TableSourceFactoryContext(options, classLoader);
        ConfigValidator.of(context.getOptions()).validate(factory.optionRule());
        TableSource<T, SplitT, StateT> tableSource = factory.createSource(context);
        return tableSource.createSource();
    }

前端代码生成文件

JobExecutorServiceImpl

    @Override
    public Result<Long> jobExecute(Integer userId, Long jobDefineId) {

        JobExecutorRes executeResource =
                jobInstanceService.createExecuteResource(userId, jobDefineId);
        String jobConfig = executeResource.getJobConfig();

        String configFile = writeJobConfigIntoConfFile(jobConfig, jobDefineId);

        Long jobInstanceId =
                executeJobBySeaTunnel(userId, configFile, executeResource.getJobInstanceId());
        return Result.success(jobInstanceId);
    }

web页面操作

web请用最新版本我这里用的是最新版本

pg报错Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: Can not find catalog table with factoryId [Postgres]

跟踪代码发现表写错了表明

public boolean tableExists(TablePath tablePath) throws CatalogException {
        try {
            if (StringUtils.isNotBlank(tablePath.getDatabaseName())) {
                return databaseExists(tablePath.getDatabaseName())
                        && listTables(tablePath.getDatabaseName())
                                .contains(tablePath.getSchemaAndTableName());
            }

            return listTables(defaultDatabase).contains(tablePath.getSchemaAndTableName());
        } catch (DatabaseNotExistException e) {
            return false;
        }
    }

web加载插件代码

  protected List<Factory> getPluginFactories() {
        List factories;
        if (this.pluginDir.toFile().exists()) {
            log.debug("load plugin from plugin dir: {}", this.pluginDir);

            List files;
            try {
                files = FileUtils.searchJarFiles(this.pluginDir);
            } catch (IOException var4) {
                throw new RuntimeException(String.format("Can not find any plugin(source/sink/transform) in the dir: %s", this.pluginDir));
            }

            factories = FactoryUtil.discoverFactories(new URLClassLoader((URL[])files.toArray(new URL[0])));
        } else {
            log.warn("plugin dir: {} not exists, load plugin from classpath", this.pluginDir);
            factories = FactoryUtil.discoverFactories(Thread.currentThread().getContextClassLoader());
        }

        return factories;
    }

本地web启动配置seatunnel 地址引用插件

System.getProperty(“SEATUNNEL_HOME”)引用这个地址不生效
可以在这个上面配置或环境-DSEATUNNEL_HOME=E:\jituab1.0\databridge\seatunnel\seatunnel-dist\target\apache-seatunnel-2.3.7-bin.tar\apache-seatunnel-2.3.7-bin\apache-seatunnel-2.3.7

引用本地包

 <dependency>
            <groupId>org.apache.seatunnel</groupId>
            <artifactId>seatunnel-common</artifactId>
            <version>${seatunnel-framework.version}</version>
            <scope>system</scope>
            <systemPath>D:/other-workspace/seatunnel-web/lib/seatunnel-common-2.3.5-SNAPSHOT-2.12.15.jar</systemPath>
        </dependency>

查看日志

点击web执行

会在页面工程底下profile底下生成一个模板文件是生成到引擎执行脚本
其文件是jobDefineId=15677394387200 以这个开头的文件名称
15677394387200.conf
seatunnel引擎执行bin/seatunnel.sh -l

从上面图可以看出job_name的名称前缀是web页面生成的执行实例id
InstanceId=15686439704192 其jobid是911069562043105281

cd logs

可以看的是对应的日志是job-911069562043105281.log