greenplum 同步 mysql_mysql增量同步数据到Greenplum

有没有从mysql或者mariadb 中迁移增量数据到 PostgreSQL 或 Greenplum的相关方案呢?

目前有使用过阿里的mysql2gp这个工具,数据全量迁移没有问题,但通过解析binlog迁移增量数据的时候遇到了一些问题,当需要同步的表格数量非常多的时候,进程进自动退出GP,下面是堆栈信息

[root@Local-GP1 corefile]# gdb core-addr2line-11-530-530-7733-1493717638

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

...

Reading symbols from /usr/bin/addr2line...Reading symbols from /usr/lib/debug/usr/bin/addr2line.debug...done.

done.

[New Thread 7733]

Reading symbols from /usr/lib64/libbfd-2.20.51.0.2-5.44.el6.so...Reading symbols from /usr/lib/debug/usr/lib64/libbfd-2.20.51.0.2-5.44.el6.so.debug...done.

done.

Loaded symbols for /usr/lib64/libbfd-2.20.51.0.2-5.44.el6.so

Reading symbols from /opt/greenplum/greenplum-db/./lib/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /opt/greenplum/greenplum-db/./lib/libz.so.1

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib/modules/2.6.32-642.15.1.el6.x86_64/vdso/vdso.so...Reading symbols from /usr/lib/debug/lib/modules/2.6.32-642.15.1.el6.x86_64/vdso/vdso.so.debug...done.

done.

Loaded symbols for /lib/modules/2.6.32-642.15.1.el6.x86_64/vdso/vdso.so

Core was generated by `addr2line -s -e /opt/greenplum/greenplum-db-4.3.12.0/bin/postgres 0xb04062 0xb0'.

Program terminated with signal 11, Segmentation fault.

#0 read_abbrevs (abfd=0x1d84060, section=, offset=, symbol=0x0, symbols=0x4, filename_ptr=0x6059b8, functionname_ptr=0x6059c0, linenumber_ptr=0x6059c8,

addr_size=, pinfo=0x1d845b0) at ./dwarf2.c:663

663 cur_abbrev->number = abbrev_number;

Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64

(gdb) bt

#0 read_abbrevs (abfd=0x1d84060, section=, offset=, symbol=0x0, symbols=0x4, filename_ptr=0x6059b8, functionname_ptr=0x6059c0, linenumber_ptr=0x6059c8,

addr_size=, pinfo=0x1d845b0) at ./dwarf2.c:663

#1 parse_comp_unit (abfd=0x1d84060, section=, offset=, symbol=0x0, symbols=0x4, filename_ptr=0x6059b8, functionname_ptr=0x6059c0, linenumber_ptr=0x6059c8,

addr_size=, pinfo=0x1d845b0) at ./dwarf2.c:2131

#2 find_line (abfd=0x1d84060, section=, offset=, symbol=0x0, symbols=0x4, filename_ptr=0x6059b8, functionname_ptr=0x6059c0, linenumber_ptr=0x6059c8,

addr_size=, pinfo=0x1d845b0) at ./dwarf2.c:3187

#3 0x00007fdf263b0347 in _bfd_dwarf2_find_nearest_line (abfd=, section=, symbols=, offset=,

filename_ptr=, functionname_ptr=, linenumber_ptr=0x6059c8, addr_size=0, pinfo=0x1d845b0) at ./dwarf2.c:3258

#4 0x00007fdf263bc9da in _bfd_elf_find_nearest_line (abfd=0x1d84060, section=0x1d87210, symbols=0x7fdf26837010, offset=6579362, filename_ptr=0x6059b8, functionname_ptr=0x6059c0, line_ptr=0x6059c8)

at elf.c:7330

#5 0x0000000000401cd0 in find_address_in_section (abfd=, section=, data=) at addr2line.c:161

#6 0x00007fdf263a00ac in bfd_map_over_sections (abfd=0x1d84060, operation=0x401c70 , user_storage=0x0) at section.c:1299

#7 0x0000000000402126 in translate_addresses (argc=18, argv=0x7fff654ac1b8) at addr2line.c:216

#8 process_file (argc=18, argv=0x7fff654ac1b8) at addr2line.c:316

Greenplum 集群之间同步数据方法及性能.....................................................................................1 1 概述.................................................................................................................................................2 2 相同集群相同数据库不同 SCHEMA 之间同步数据....................................................................2 2.1 查看原始表的大小行数与结构......................................................................................... 2 2.2 同步语句..............................................................................................................................2 2.3 查看 cpu 与内存的使用情况..............................................................................................3 2.3.1 查看 Master CPU 与内存使用情况.........................................................................3 2.3.2 查看数据节点的 CPU 使用情况..............................................................................3 2.3.3 查看数据节点的内存与磁盘使用情况..................................................................6 2.4 查看耗时与表的大小..........................................................................................................7 3 相同集群不同数据库之间同步数据.............................................................................................8 3.1 使用 pg_dum 方式同步数据..............................................................................................8 3.1.1 查看原始表的信息...................................................................................................8 3.1.2 把数据下载到磁盘...................................................................................................9 3.1.2.1 下载数据语句............................................................................................... 9 3.1.2.2 查看 Master 节点详细信息.........................................................................9 3.1.3 把数据导入到数据库中........................................................................................ 10 3.1.3.1 导入数据库语句......................................................................................... 10 3.1.3.2 查看 Master 节点的详细信息...................................................................11 3.1.3.3 查看数据节点的详细信息.........................................................................11 3.1.4 验证数据的准确性................................................................................................ 13 3.1.5 pg_dump 同步数据总结........................................................................................13 3.2 使用 dblink 同步数据........................................................................................................13 3.2.1 dblink 同步相同集群不同数据库的数据.............................................................13 3.2.1.1 同步语句......................................................................................................13 3.2.1.2 查看 Master 节点的详细信息...................................................................15 3.2.1.3 查看数据节点的详细信息.........................................................................15 3.2.2 验证数据的准确性................................................................................................ 17 3.2.3 dblink 使用总结......................................................................................................17 3.3 使用 gptransfer 同步数据.................................................................................................17 3.3.1 gptransfer 介绍.......................................................................................................17 3.3.2 gptransfer 命令参数介绍.......................................................................................18 3.3.3 查看集群及硬件信息............................................................................................ 19 3.3.4 同集群之间同步数据............................................................................................ 19 3.3.4.1 查看表的详细信息..................................................................................... 19 3.3.4.2 进行表数据同步......................................................................................... 19 3.3.4.3 查看硬件详细信息..................................................................................... 21 3.3.4.4 查看数据的准确性..................................................................................... 23 3.3.4.5 使用 gptransfer 总结...................................................................................23 3.3.5 不同集群之间同步数据........................................................................................ 24
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值