datax同步数据：数据源：mongo，目标源：hdfs

最新推荐文章于 2024-09-07 12:16:05 发布

远方有海，小样不乖

最新推荐文章于 2024-09-07 12:16:05 发布

阅读量706

点赞数 1

分类专栏： datax 文章标签：大数据 Powered by 金山文档

本文链接：https://blog.csdn.net/anguoan/article/details/129128016

版权

datax 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

该配置文件描述了一个使用DataX从MongoDB读取特定时间范围内的carVoltage集合数据，并将这些数据写入HDFS的过程。数据包括_id,plateNum,terminalId,time（日期）和voltage（电压）字段，写入模式为追加。

摘要由CSDN通过智能技术生成

一.配置文件mongo-hdfs.json

{
    "job": {
        
        "setting": {
        
            "speed": {
            "channel": 2
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mongodbreader",
                    "parameter": {
                    "address": ["cdh3:27017"],
                    "userName": "",
                    "userPassword": "",
                    "dbName": "db",
                    "collectionName": "carVoltage",
                            "query":"{time:{ $gte: ISODate('2016-11-28T11:16:33.303Z'), $lte: ISODate('2017-12-28T11:16:33.303Z') }}",
                    "column": [
                        {
                            "index":0,
                            "name": "_id",
                            "type": "string"
                            }, {
                            "index":1,
                            "name": "plateNum",
                            "type": "string"
                            }, {
                            "index":2,
                            "name": "terminalId",
                            "type": "string"
                            }, {
                            "index":3,
                            "name": "time",
                            "type": "date"
                            }, {
                            "index":4,
                            "name": "voltage",
                            "type": "double"
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "hdfswriter",
                    "parameter": {
                    "defaultFS": "hdfs://10.x.x.x:8020",
                    "fileType": "text",
                    "path": "/user/anguoan",
                    "fileName": "zhenghaoReader",
                    "column": [
                        {
                        "index":0,
                        "name": "id",
                        "type": "string"
                        }, 
                        {
                        "index":1,
                        "name": "plateNum",
                        "type": "string"
                        }, 
                        {
                        "index":2,
                        "name": "terminalId",
                        "type": "string"
                        }, 
                        {
                        "index":3,
                        "name": "time",
                        "type": "date"
                        }, 
                        {
                        "index":4,
                        "name": "voltage",
                        "type": "double"
                        }    
                    ],
                    "writeMode": "append",
                    "fieldDelimiter": ","
                    }
                }
            }
        ]
    }

}