c使用tcp传输的二进制数据_数据传输 | dtle 使用初探

于 2020-11-26 14:39:50 发布
阅读量168 收藏
点赞数
文章标签： c使用tcp传输的二进制数据
作者：马莹乐
爱可生研发团队成员，负责 mysql 中间件和数据库管理平台的测试。擅长找茬（测试技术爱好者），欢迎大家试用 dtle~
本文来源：原创投稿
*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。

dtle 是一款爱可生 MySQL 开源数据传输中间件，此文简要介绍初步使用方法。项目地址：https://github.com/actiontech/dtle

一、下载安装

环境说明：

  • 准备三台主机，主机名分别为：node4、node5、node6
  • 三台均为 agent，另将 node4 和 node5 作为 manager

1.1 下载

下载 dtle 最新的 release rpm 包

地址：https://github.com/actiontech/dtle/releases

423e9d9f4df190c9f43f7859d7b82598.png

dtle 安装包下载：

wget https://github.com/actiontech/dtle/releases/download/v2.19.11.0/dtle-2.19.11.0.x86_64.rpm

1.2 安装

安装步骤：

https://actiontech.github.io/dtle-docs-cn/4/4.0_installation.html

b3d63c83ed4718c8b2df1153c75f751f.png

分别将 node4，node5 和 node6 的 dtle 安装完成。

1.3 启动 dtle

dtle 安装完成后，并未启动，需要先配置 dtle 的 conf 文件，再启动。

1.4 配置 dtle.conf

配置相关说明：

https://actiontech.github.io/dtle-docs-cn/4/4.1nodeconfiguration.html

dtle.conf 准备：以下配置文件替分别替换原有 dtle.conf（/opt/dtle/etc/dtle/dtle.conf）

node4 的 dtle.conf

# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true

bind_addr = "172.100.9.4"

# Modify our port to avoid a collision with server
ports {
    http = 8190
}

# Enable the manager
manager {
    enabled = true

    # Self-elect, should be 3 or 5 for production,
    # Addresses to attempt to join when the server starts.
    join = [ "172.100.9.4","172.100.9.5" ]
}

# Enable the agent
agent {
    enabled = true
    managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}

metric{
   collection_interval = "15s"
   publish_allocation_metrics = "true"
   publish_node_metrics = "true"
}

addresses {
    http = "172.100.9.4"
    rpc = "172.100.9.4"
    serf = "172.100.9.4"
}
advertise {
    http = "172.100.9.4"
    rpc = "172.100.9.4"
    serf = "172.100.9.4"
}

node5 的 dtle.conf

# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true

bind_addr = "172.100.9.5"

# Modify our port to avoid a collision with server
ports {
    http = 8190
}

# Enable the manager
manager {
    enabled = true

    # Self-elect, should be 3 or 5 for production,
    # Addresses to attempt to join when the server starts.
    join = [ "172.100.9.4","172.100.9.5" ]
}

# Enable the agent
agent {
    enabled = true
    managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}

metric{
   collection_interval = "15s"
   publish_allocation_metrics = "true"
   publish_node_metrics = "true"
}

addresses {
    http = "172.100.9.5"
    rpc = "172.100.9.5"
    serf = "172.100.9.5"
}
advertise {
    http = "172.100.9.5"
    rpc = "172.100.9.5"
    serf = "172.100.9.5"
}

node6 的 dtle.conf

# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true

bind_addr = "172.100.9.6"

# Modify our port to avoid a collision with server
ports {
    http = 8190
}

# Enable the manager
manager {
    enabled = false

    # Self-elect, should be 3 or 5 for production,
    # Addresses to attempt to join when the server starts.
    join = [ "172.100.9.4","172.100.9.5" ]
}

# Enable the agent
agent {
    enabled = true
    managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}

metric{
   collection_interval = "15s"
   publish_allocation_metrics = "true"
   publish_node_metrics = "true"
}

addresses {
    http = "172.100.9.6"
    rpc = "172.100.9.6"
    serf = "172.100.9.6"
}
advertise {
    http = "172.100.9.6"
    rpc = "172.100.9.6"
    serf = "172.100.9.6"
}

1.5 启动 dtle

启动命令说明：

https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html

93dcc6fb390fd94745b5ae35f643a1f7.png

40f1af20834da1be60a886424c97950c.png

1.6 查看 3 个节点 dtle 进程

node 4

25c924c9f95fb504dc262ee25c6d4a89.png

node 5

4d391465621ffaa9c7a8c15cffae5cca.png

node 6

f93bdfc92274192c631a7bc24f75b79a.png

1.7 尝试查看 dtle 架构是否符合预期

命令说明：

https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html

f5574a19d2b8155603f572011fe39eae.png

6c91d3455c95250056eb3b605f541501.png

其中，members 展示的是集群中 manager 节点的信息。

HTTP API：

https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html

c27e778392e393948f929dce32793ee9.png

输出内容为 json 格式，可以安装 jq，使之格式化展示。

二、源端/目标端准备

MySQL 迁移用户准备。

2.1 创建用户

源端用户创建，

b96ab3030a28ce5461081ec533630fbe.png

目标端用户创建，

0b0f8854366eb401c61552e5e92623c9.png

有关源端/目标端用户的最小权限说明：

https://actiontech.github.io/dtle-docs-cn/4/4.5mysqluser_privileges.html

2.2 数据准备

源端数据准备，

5c907e9788c456f0307166ceaad6db49.png

目标端测试前库表情况，

38b236a7c331850948aed64a1f38cd40.png

2.3 创建 job

创建全量 + 增量，表级别迁移 job，分别使用 node5 和 node6 节点的 dtle 作为源端和目标端迁移的 agent。

job 配置相关参数：

https://actiontech.github.io/dtle-docs-cn/4/4.3jobconfiguration.html

2.3.1 准备 job.json

job.json

{
    "Name":"have_a_try",
    "Failover":false,
    "Orders":[],
    "Type":"synchronous",
    "Tasks":[
        {
            "Type":"Src",
            "NodeId":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
            "Config":{
                "Gtid":"",
                "DropTableIfExists":false,
                "SkipCreateDbTable":false,
                "ApproveHeterogeneous":true,
                "ReplChanBufferSize":"600",
                "ChunkSize":"2000",
                "ExpandSyntaxSupport":false,
                "MsgBytesLimit":"20480",
                "MsgsLimit":"65536",
                "BytesLimit":"67108864",
                "GroupMaxSize":"1",
                "GroupTimeout":"100",
                "SqlFilter":[],
                "ReplicateDoDb":[
                    {
                        "TableSchema":"test",
                        "Tables":[
                            {
                                "TableName":"test1"
                            }
                        ]
                    }
                ],
                "ConnectionConfig":{
                    "Host":"172.100.9.1",
                    "Port":"3306",
                    "User":"src_test",
                    "Password":"test"
                }
            }
        },
        {
            "Type":"Dest",
            "NodeId":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
            "Config":{
                "ParallelWorkers":"1",
                "ConnectionConfig":{
                    "Host":"172.100.9.2",
                    "Port":"3306",
                    "User":"dest_test",
                    "Password":"test"
                }
            }
        }
    ]
}

其中 NodeId 可通过命令或者 HTTP API 查询得知。

2.3.2 创建 job

创建 job 试用 HTTP API：

https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html

创建完成后查询 job 状态为 running。

629eba5f03b3275467e4664f076b1d28.png

job 详细信息可使用 HTTP API 查询，示例：

[
    {
        "CreateIndex":109,
        "ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
        "JobModifyIndex":2372,
        "JobSummary":{
            "Constraints":null,
            "CreateIndex":109,
            "Datacenters":[
                "dc1"
            ],
            "EnforceIndex":false,
            "Failover":false,
            "ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
            "JobModifyIndex":2372,
            "ModifyIndex":2372,
            "Name":"have_a_try",
            "Orders":[

            ],
            "Region":"global",
            "Status":"running",
            "StatusDescription":"",
            "Tasks":[
                {
                    "Config":{
                        "GroupMaxSize":"1",
                        "ChunkSize":"2000",
                        "BinlogFile":"1.000002",
                        "MsgsLimit":"65536",
                        "GroupTimeout":"100",
                        "DropTableIfExists":false,
                        "ExpandSyntaxSupport":false,
                        "BytesLimit":"67108864",
                        "MsgBytesLimit":"20480",
                        "ReplChanBufferSize":"600",
                        "NatsAddr":"172.100.9.6:8193",
                        "ApproveHeterogeneous":true,
                        "SqlFilter":[

                        ],
                        "BinlogPos":4261,
                        "ConnectionConfig":{
                            "Host":"172.100.9.1",
                            "Port":"3306",
                            "User":"src_test",
                            "Password":"*"
                        },
                        "TrafficAgainstLimits":0,
                        "Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
                        "ReplicateDoDb":[
                            {
                                "TableSchema":"test",
                                "Tables":[
                                    {
                                        "TableName":"test1"
                                    }
                                ]
                            }
                        ],
                        "SkipCreateDbTable":false
                    },
                    "ConfigLock":{

                    },
                    "Constraints":null,
                    "Driver":"MySQL",
                    "Leader":false,
                    "NodeID":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
                    "NodeName":"",
                    "Type":"Src"
                },
                {
                    "Config":{
                        "BinlogFile":"1.000002",
                        "BinlogPos":4261,
                        "ConnectionConfig":{
                            "Host":"172.100.9.2",
                            "Port":"3306",
                            "User":"dest_test",
                            "Password":"*"
                        },
                        "Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
                        "ParallelWorkers":"1",
                        "NatsAddr":"172.100.9.6:8193"
                    },
                    "ConfigLock":{

                    },
                    "Constraints":null,
                    "Driver":"MySQL",
                    "Leader":true,
                    "NodeID":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
                    "NodeName":"",
                    "Type":"Dest"
                }
            ],
            "Type":"synchronous"
        },
        "ModifyIndex":2372,
        "Name":"have_a_try",
        "Status":"running",
        "StatusDescription":"",
        "Type":"synchronous"
    }
]

2.4 查询目标端迁移情况

目标端迁移前后对比：

252a7e83dec747cdc9a37db3699a1b53.png

新增两个库其中 test 库是我们 job 迁移的结果。

dtle 库相关说明：

https://actiontech.github.io/dtle-docs-cn/3/3.3impacton_dest.html

2e02ff01d1643019565d67afd92f67e1.png

2.5 在源端执行普通 DML 操作

183ed4a16e044da9dfdc151a6cea84ff.png

2.6 在源端执行普通 DDL 操作

5bf31546add7e23a86184c87b3f431b4.png

小结

更多用法，详见项目文档，欢迎试用~

https://actiontech.github.io/dtle-docs-cn/

weixin_39760433
关注
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值