Datax同步GP到MongoDB

1.整体流程

读取GP数据-同步到MongoDB。

整体流程用datax完成，使用postgresqlreader和mongodbwriter，（需要符合MongoDB版本），

2.版本问题

202309的dataxmongodbwriter不支持本次用到的mongo6.0.10版本，

mongo-java-driver是3.2.2版本，对应mongo版本可查看，图截不，需要

Compatibility — Java Sync (mongodb.com)

 

 修改兼容的版本，重新编译，上传mongobwriter插件在/datax/plugin/writer

3.配置文件

{
  "job": {
    "setting": {
      "speed": {
        "channel": "3"
      },
	  "priority": 1
    },
    "content": [
      {
		 "reader": {
          "name": "postgresqlreader",
          "parameter": {
            "username": "",
            "password": "",
            "column": [
                "id",
				"province_code",
				"province_name",
				"org_code",
				"org_name",
				"business_type",
				"account_period",
				"risk_type",
				"risk_three_category_code",
				"risk_three_category_name",
				"risk_category_amount_day",
				"risk_category_amount_month",
				"risk_category_amount_year"
            ],
            "connection": [
              {
                "table": [
                  ""
                ],
                "jdbcUrl": [
				"jdbc:postgresql://ip:端口/postgres?characterEncoding=UTF-8&currentSchema=？？？"
                ]
              }
            ]
          }
        },
		"writer": {
          "name": "mongodbwriter",
          "parameter": {
            "address": [
              ""
            ],
            "userName": "",
            "userPassword": "",
            "dbName": "",
            "collectionName": "",
            "column": [
			{"name": "id", "type": "string"},
			{"name": "province_code", "type": "string"},
			{"name": "province_name", "type": "string"},
			{"name": "org_code", "type": "string"},
			{"name": "org_name", "type": "string"},
			{"name": "business_type", "type": "string"},
			{"name": "account_period", "type": "string"},
			{"name": "risk_type", "type": "string"},
			{"name": "risk_three_category_code", "type": "string"},
			{"name": "risk_three_category_name", "type": "string"},
			{"name": "risk_category_amount_day", "type": "int"},
			{"name": "risk_category_amount_month", "type": "int"},
			{"name": "risk_category_amount_year", "type": "int"}],
			"upsertInfo": {
              "isUpsert": "true",
              "upsertKey": "id"
            }
          }
        }
      }
    ]
  }
}

4.问题注意

4.1字段类型datax会报脏数据的问题

4.2Authentication failed问题

一直认证失败，Mongodb中的每个库都是独立，需要自己创建账号，在需要同步的数据库中建立自己的账号。

> use data
> db.createUser(
    {
      user: "user01",
      pwd: "123456",
      roles: ["readWrite"]
    }
 ) 

4.3 MongoServerError: not primary

本公司mongo服务器有三台，1主2从，原本一直操作的主节点，系统故障后重启主节点换了，需要把访问地址换成主节点的地址就好了

4.4 DS调度器节点安装mongosh命令

下载mongodb-mongosh-1.10.6.x86_64.rpm

rpm -ivh mongodb-mongosh-1.10.6.x86_64.rpm