java(spring boot) 调用kettle脚本

java 调用kettle脚本

可以提前熟悉一下构建过程　一个简单的java项目

之前的kettle docker化的过程中，我是直接将Spoon整个项目导入的，所以造成文件臃肿．生产环境使用crontab方式部署，但是因为服务器遭到挖矿脚本的攻击．所以普通用户crontab服务会被攻击而停止运行．而且使用crontab的方式确实不便于管理（这里推荐XXL-JOB分布式调度平台)．当然在实际开发的过程中我们也不能灵活的部署,所以我们如果能借助熟悉的开发语言来调用脚本的话能够让开发工作变得更加简单．kettle的循环脚本确实在次数过多的时候会出现问题.

我们直接进入主题．

新建KettleDome.java文件(可以在编译器上开发)

import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.exception.KettleException;
import org.pentaho.di.trans.Trans;
import org.pentaho.di.trans.TransMeta;

public class KettleDome{
    public static void main(String[] args) throws KettleException{
        String fileName = "/~/test.ktr";
        System.out.println("-----------");
        KettleEnvironment.init();
        TransMeta transMeta = new TransMeta(fileName);
        Trans trans = new Trans(transMeta);
        trans.execute(null);
        trans.waitUntilFinished();
    }
}

将spoon中的lib目录拷贝到/~/lib目录下

构建class文件

javac -extdirs /~/lib  KettleDome.java
-extdirs:加入第三方jar包目录

构建jar包

jar cvf KettleDome.jar KettleDome.class 
touch MANIFEST.MF
vim MANIFEST.MF
~~
		nifest-Version: 1.0
		Created-By: 1.4.2 (Sun Microsystems Inc.)	
		Main-Class: KettleDome
~~
jar umf MANIFEST.MF KettleDome.jar 

运行脚本

java -jar -Djava.ext.dirs=/~/lib/ KettleDome.jar
-Djava.ext.dirs:加入第三方jar包目录

运行结果

2019/12/13 16:20:43 - user_tmp - 为了转换解除补丁开始  [user_tmp]
...
2019/12/13 16:20:46 - kettle-docker - 为'kettle-docker'创建数据库连接池...
2019/12/13 16:20:46 - kettle-docker - 为'kettle-docker'创建数据库连接池成功
...execute
2019/12/13 16:21:15 - 去除重复记录.0 - 完成处理 (I=0, O=0, R=1758, W=1741, U=0, E=0)
2019/12/13 16:21:19 - 表输出.0 - 完成处理 (I=0, O=1741, R=1741, W=1741, U=0, E=0)

如果使用maven

将核心包下载到本地仓库(data/tool/data-integration/lib 是我安装spoon的目录)

相关包查询
 ls -al | egrep "kettle-(core|engine)|metastore"
kettle相关jar包现在到本地maven仓库

mvn install:install-file -Dfile=/data/tool/data-integration/lib/kettle-core-8.2.0.0-342.jar -DgroupId=org.pentaho -DartifactId=kettle-core -Dversion=8.2.0.0-342 -Dpackaging=jar 

mvn install:install-file -Dfile=/data/tool/data-integration/lib/kettle-engine-8.2.0.0-342.jar -DgroupId=org.pentaho -DartifactId=kettle-engine -Dversion=8.2.0.0-342 -Dpackaging=jar 

mvn install:install-file -Dfile=/data/tool/data-integration/lib/metastore-8.2.0.0-342.jar -DgroupId=org.pentaho -DartifactId=metastore -Dversion=8.2.0.0-342 -Dpackaging=jar 

如果你需要启动一个里面含有 表输入(出)&excel&java脚本的.ktr文件那么你需要引入如下的文件

 <properties>
        <java.version>1.8</java.version>
        <kettle.version>8.2.0.0-342</kettle.version>
    </properties>


    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>


        <!--ETL-->
        <dependency>
            <groupId>org.pentaho</groupId>
            <artifactId>kettle-core</artifactId>
            <version>${kettle.version}</version>
        </dependency>
        <dependency>
            <groupId>org.pentaho</groupId>
            <artifactId>kettle-engine</artifactId>
            <version>${kettle.version}</version>
        </dependency>
        <dependency>
            <groupId>org.pentaho</groupId>
            <artifactId>metastore</artifactId>
            <version>${kettle.version}</version>
        </dependency>
<!--        <dependency>-->
<!--            <groupId>org.pentaho</groupId>-->
<!--            <artifactId>kettle-dbdialog</artifactId>-->
<!--            <version>${kettle.version}</version>-->
<!--        </dependency>-->

        <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-vfs2 -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-vfs2</artifactId>
            <version>2.4.1</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>28.1-jre</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/commons-lang/commons-lang -->
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/commons-codec/commons-codec -->
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.13</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-io -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.2</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.codehaus.janino/commons-compiler -->
        <dependency>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>commons-compiler</artifactId>
            <version>3.1.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.codehaus.janino/janino -->
        <dependency>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>janino</artifactId>
            <version>3.1.0</version>
        </dependency>
<!--        &lt;!&ndash; https://mvnrepository.com/artifact/org.apache.poi/poi &ndash;&gt;-->
<!--        <dependency>-->
<!--            <groupId>org.apache.poi</groupId>-->
<!--            <artifactId>poi</artifactId>-->
<!--            <version>4.1.1</version>-->
<!--        </dependency>-->
        <!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
        <dependency>
            <groupId>org.apache.poi</groupId>
            <artifactId>poi-ooxml</artifactId>
            <version>4.1.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/net.sourceforge.jexcelapi/jxl -->
        <dependency>
            <groupId>net.sourceforge.jexcelapi</groupId>
            <artifactId>jxl</artifactId>
            <version>2.6.12</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/commons-dbcp/commons-dbcp -->
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>





        <!--ETL-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.43</version>
        </dependency>

    </dependencies>

java文件

package com.etl.test;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.pentaho.di.core.KettleEnvironment;
import org.pentaho.di.core.exception.KettleException;
import org.pentaho.di.trans.Trans;
import org.pentaho.di.trans.TransMeta;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

/**
 * @Author: xuzz
 * @Description:
 * @Date: Created in 上午11:00 19-12-13
 * @Modified By:
 */
@RunWith(SpringRunner.class)
@SpringBootTest
public class TestDome {
    @Test
    public void test() throws KettleException {

        String fileName = "/~/test.ktr";
        System.out.println("-----------");
        KettleEnvironment.init();
        TransMeta transMeta = new TransMeta(fileName);
        Trans trans = new Trans(transMeta);
        trans.execute(null);
        trans.waitUntilFinished();

    }
}

当然这些是你没有经过首次试错的得到的结果.

以上:当前用户的.kettle目录下的配置文件依然有效.

.kjb的调用方式

		JobMeta jm = new JobMeta("/~/~.kjb",null);
        Job job = new Job(null, jm);
        job.start();
        job.waitUntilFinished();

.kjb .ktr传参

作业:
job.setParameterValue("key","value");
job.setVariable(key,v);
转换:
trans.setParameterValue("key","value");
trans.setVariable(key,v);