datax 自定义transformer

datax 自带的transformer ：

自定义transformer源码

 

TransformerRegister的loadTransformerFromLocalStorage方法，该方法首先会查找datax目录下的local_storage/transformer的目录并便利transformer目录，加载transformer目录下的transformer.json文件，并加载目录下的jar包比如一个叫udf_aes的transformer结构如下所示

//获取目录并遍历$datax_home/local_storage/transformer
        String[] paths = new File(CoreConstant.DATAX_STORAGE_TRANSFORMER_HOME).list();
        if (null == paths) {
            return;
        }

        for (final String each : paths) {
            try {
                if (transformers == null || transformers.contains(each)) {
                    loadTransformer(each); //便利并注册transformer
                }
            } catch (Exception e) {
                LOG.error(String.format("skip transformer(%s) loadTransformer has Exception(%s)", each, e.getMessage()), e);
            }

        }

自定义transformer的代码

public class YoreDecryptTransformer extends Transformer {
    private static final Logger LOG = LoggerFactory.getLogger(YoreDecryptTransformer.class);
    public YoreDecryptTransformer() {
        this.setTransformerName("udf_aes");
    }



    public com.alibaba.datax.common.element.Record evaluate(com.alibaba.datax.common.element.Record record, Object... paras) {

        int columnIndex;
        String aesKey = null;
        try {
            if(paras.length == 1){
                columnIndex = (Integer) paras[0];
            }else if(paras.length ==2){
                columnIndex = (Integer) paras[0];
                aesKey = (String) paras[1];
            }else{
                throw new RuntimeException(getTransformerName() + " paras at moust 2");
            }
        }catch (Exception e){
            throw DataXException.asDataXException(TransformerErrorCode.TRANSFORMER_ILLEGAL_PARAMETER, "paras:" + Arrays.asList(paras).toString() + " => " + e.getMessage());
        }
        Column column = record.getColumn(columnIndex);
        try {
            String oriValue = column.asString();
            if(oriValue == null){
                return record;
            }

            String newValue;
            if(aesKey == null){
                newValue = AesUtil.decrypt(oriValue);
            }else if(aesKey.trim().length()<1){
                newValue = AesUtil.decrypt(oriValue);
                LOG.warn("指定的解密密钥 key={} 无效，将采用默认密钥解密", aesKey);
            }else {
                AesUtil.init(aesKey);
                newValue = AesUtil.decrypt(oriValue);
            }


            record.setColumn(columnIndex, new StringColumn(newValue));

        }catch (Exception e){
            throw DataXException.asDataXException(TransformerErrorCode.TRANSFORMER_RUN_EXCEPTION, e.getMessage(),e);
        }
        return record;
    }

继承transformer类，并实现evaluate方法，evaluate的方法即为方法的实现类。关于依赖问题，因为datax的依赖并没有上传到maven，所以依赖可能成为问题,可以用下面的方式解决。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.wsp.cn</groupId>
    <artifactId>datax-transformer</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <datax.version>0.0.1-SNAPSHOT</datax.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.3.2</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba.datax</groupId>
            <artifactId>datax-common</artifactId>
            <version>${datax.version}</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/lib/datax-common-0.0.1-SNAPSHOT.jar</systemPath>
        </dependency>

        <!-- DataX Transformer -->
        <dependency>
            <groupId>com.alibaba.datax</groupId>
            <artifactId>datax-transformer</artifactId>
            <version>${datax.version}</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/lib/datax-transformer-0.0.1-SNAPSHOT.jar</systemPath>
        </dependency>

        <!-- DataX Core -->
        <dependency>
            <groupId>com.alibaba.datax</groupId>
            <artifactId>datax-core</artifactId>
            <version>${datax.version}</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/lib/datax-core-0.0.1-SNAPSHOT.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.12</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.3.2</version>
        </dependency>


    </dependencies>



</project>

上传jar包，并编写transformer.json文件到local_storage/transformer/udf_aes目录即可。

{
   "class":"YoreDecryptTransformer",
   "name":"udf_aes"
}