Dolphinscheduler使用Datax同步mongodb到doris

最新推荐文章于 2024-05-22 16:55:50 发布

Jmayday

最新推荐文章于 2024-05-22 16:55:50 发布

阅读量1.9k

点赞数 2

分类专栏： ETL 文章标签： mongodb etl

本文链接：https://blog.csdn.net/Jmayday/article/details/129734099

版权

ETL 专栏收录该内容

29 篇文章 50 订阅

订阅专栏

1、dolphinscheduler与datax集成环境变量配置

export JAVA_HOME=/opt/soft/java
export  PYTHON_HOME=/usr/bin/python
export  DATAX_HOME=/opt/soft/datax
export  PATH=$PYTHON_HOEM:$DATAX_HOME/bin:$JAVA_HOME/bin:$PATH

截图如下:

注意服务器上安装python2.7，以及datax安装在对应目录下。

2、同步mongodb 到doris

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "mongodbreader", 
                    "parameter": {
                        "address": ["xxx:27017","xxxx:27017","xxxx:27017"], 
                        "collectionName": "leases", 
                        "column": [
                        {
                        "name":"_id",
                        "type":"string"
                        },
                        {
                        "name":"id",
                        "type":"string"
                        },
                        {
                        "name":"storeid",
                        "type":"string"
                        },
                        {
                        "name":"storename",
                        "type":"string"
                        },
                        {
                        "name":"title",
                        "type":"string"
                        },
                        {
                        "name":"source",
                        "type":"string"
                        },
                        {
                        "name":"code",
                        "type":"string"
                        },
                        {
                        "name":"codeyear",
                        "type":"string"
                        },
                        {
                        "name":"codemonth",
                        "type":"string"
                        },
                        {
                        "name":"codesort",
                        "type":"string"
                        },
                        {
                        "name":"classify",
                        "type":"string"
                        },
                        {
                        "name":"property",
                        "type":"string"
                        },
                        {
                        "name":"propertydir",
                        "type":"string"
                        },
                        {
                        "name":"leaseduration",
                        "type":"string"
                        },
                        {
                        "name":"leasestartdate",
                        "type":"string"
                        },
                        {
                        "name":"leaseenddate",
                        "type":"string"
                        },
                        {
                        "name":"calcacreage",
                        "type":"string"
                        },
                        {
                        "name":"acreage",
                        "type":"string"
                        },
                        {
                        "name":"insidespace",
                        "type":"string"
                        },
                        {
                        "name":"residentialpoolarea",
                        "type":"string"
                        },
                        {
                        "name":"handoverdate",
                        "type":"string"
                        },
                        {
                        "name":"billingdate",
                        "type":"string"
                        },
                        {
                        "name":"deposit",
                        "type":"string"
                        },
                        {
                        "name":"rental",
                        "type":"string"
                        },
                        {
                        "name":"otherdeposit",
                        "type":"string"
                        },
                        {
                        "name":"quitedat",
                        "type":"string"
                        },
                        {
                        "name":"quitamount",
                        "type":"string"
                        }
                        ], 
                        "dbName": "xxxx", 
                        "authDb":"admin",
                        "userName": "xxxx", 
                        "userPassword": "xxxx"
                    }
                }, 
                "writer": {
                    "name": "doriswriter", 
                    "parameter": {
                        "column": ["_id","id","storeid","storename","title","source","code","codeyear","codemonth","codesort","classify","property","propertydir","leaseduration","leasestartdate","leaseenddate","calcacreage","acreage","insidespace","residentialpoolarea","handoverdate","billingdate","deposit","rental","otherdeposit","quitedat","quitamount"], 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://xxxx:9030/ODS", 
                                "selectedDatabase":"ODS",
                                "table": ["test"]
                            }
                        ], 
                        "loadProps":{},
                        "loadUrl":["xxxx"],
                        "password": "****",
                        "postSql":[],  
                        "username": "root", 
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}

3、mogodb中json格式的集合，datax可用如下方式获取属性

                       {
                        "name":"tenant.company_phone",
                        "type":"document.string"
                        },