datax的断点续传实战

最新推荐文章于 2024-04-02 09:26:04 发布
最新推荐文章于 2024-04-02 09:26:04 发布
阅读量2.2k 收藏
点赞数
分类专栏： datax 文章标签： oracle 数据库 database
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/viviliving/article/details/121408581
版权

思路：

查看日志，分成了多少个tasks

2021-11-17 05:32:36.654 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [51] tasks.
2021-11-17 05:32:36.654 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [51] tasks.

然后看看分段

select "PSN_CLCT_DETL_ID","EMP_CLCT_DETL_ID","CLCT_BILL_ID","EMP_NO","PSN_NO","CASHYM","ACCRYM_BEGN","ACCRYM_END","INSUTYPE","PSN_TYPE","PSN_IDET_TYPE","INSU_IDET","EMP_CLCTSTD","PSN_CLCTSTD","WAG","QUOT_CLCT_FLAG","EMP_CLCT_PARAVAL","EMP_INTO_PARAVAL","EMP_CLCT_AMT","EMP_CLCT_INTO_ACCT_AMT","PSN_CLCT_PARAVAL","PSN_INTO_PARAVAL","PSN_CLCT_AMT","PSN_CLCT_INTO_ACCT_AMT","OTH_CLCT_TRAF_AMT","OTH_CLCT_AMT","FINSUBS_AMT","FINSUBS_TRAF_AMT","TRAF_SUMAMT","CLCT_SUMAMT","INTE","LATEFEE","CLCT_FLAG","CLCT_TYPE","CLCT_TIME","ARVLER","ARVL_BCHNO","REVS_FLAG","TRAFER","TRAF_TIME","INTSURY_TIME","URSN_TIME","DCLA_PRD","ELEC_TAXRPT_NO","PEAWKR_FLAG","POOLAREA_NO","INSUTYPE_RETR_FLAG","TAXDEPT_CODE","PLAN_BCHNO","PSN_INSU_RLTS_ID","CLCTSTD_CRTF_RULE_CODG","CLCT_RULE_TYPE_CODG","INSU_CLCT_MONS","INIT_PSN_NO","INIT_PSN_CLCT_DETL_ID","ACCTER","CRTE_OPTINS_NO","CRTER_ID","CRTER_NAME","CRTE_TIME","OPTINS_NO","OPTER_ID","OPTER_NAME","OPT_TIME","RID","UPDT_TIME","BILL_FLAG","TRAF_FLAG" from STAF_PSN_CLCT_DETL_D where (INSUTYPE in ('310')) and  ('150000202111141816040068648282' <= RID AND RID < '150000202111141816480072686711') 

。。。。

整理出来所有

b  (INSUTYPE in ('310')) and  ('150000202111141816040068648282' <= RID AND RID < '150000202111141816480072686711') 
b  (INSUTYPE in ('310')) and  ('150000202111141816480072686711' <= RID AND RID < '150000202111141818410079517748') 
b  (INSUTYPE in ('310')) and  ('150000202111141818410079517748' <= RID AND RID < '150000202111141820570086267923') 
b  (INSUTYPE in ('310')) and  ('150000202111141820570086267923' <= RID AND RID < '150000202111141823020092502583') 
b  (INSUTYPE in ('310')) and  ('150000202111141823020092502583' <= RID AND RID < '150000202111141823140093105051') 
x  (INSUTYPE in ('310')) and  ('150000202111141823140093105051' <= RID AND RID < '150000202111141836290032288915') 
b  (INSUTYPE in ('310')) and  ('150000202111141836290032288915' <= RID AND RID < '150000202111141837380035554718') 
b  (INSUTYPE in ('310')) and  ('150000202111141837380035554718' <= RID AND RID < '150000202111141838590039379617') 
b  (INSUTYPE in ('310')) and  ('150000202111141838590039379617' <= RID AND RID < '150000202111141842040048301370') 
a  (INSUTYPE in ('310')) and  ('150000202111141842040048301370' <= RID AND RID < '150000202111141844320055457938') 
x  (INSUTYPE in ('310')) and  ('150000202111141844320055457938' <= RID AND RID < '150000202111141849170068916149') 
b  (INSUTYPE in ('310')) and  ('150000202111141849170068916149' <= RID AND RID < '150000202111141849380069887006') 
b  (INSUTYPE in ('310')) and  ('150000202111141849380069887006' <= RID AND RID < '150000202111141849400069987234') 
x  (INSUTYPE in ('310')) and  ('150000202111141849400069987234' <= RID AND RID < '150000202111141850060071112917') 
b  (INSUTYPE in ('310')) and  ('150000202111141850060071112917' <= RID AND RID < '150000202111141858090092201128') 
b  (INSUTYPE in ('310')) and  ('150000202111141858090092201128' <= RID AND RID < '150000202111141858090092213329') 
b  (INSUTYPE in ('310')) and  ('150000202111141858090092213329' <= RID AND RID < '150000202111141859380096099132') 
b  (INSUTYPE in ('310')) and  ('150000202111141859380096099132' <= RID AND RID < '150000202111141901250000652263') 
x  (INSUTYPE in ('310')) and  ('150000202111141901250000652263' <= RID AND RID < '150000202111141911280025548754') 
b  (INSUTYPE in ('310')) and  ('150000202111141911280025548754' <= RID AND RID < '150000202111141917350040445532') 
b  (INSUTYPE in ('310')) and  ('150000202111141917350040445532' <= RID AND RID < '150000202111141918590043454385') 
b  (INSUTYPE in ('310')) and  ('150000202111141918590043454385' <= RID AND RID < '150000202111141919270044198126') 
b  (INSUTYPE in ('310')) and  ('150000202111141919270044198126' <= RID AND RID < '150000202111141921020046212640') 
x  (INSUTYPE in ('310')) and  ('150000202111141921020046212640' <= RID AND RID < '150000202111141932010063110436') 
x  (INSUTYPE in ('310')) and  ('150000202111141932010063110436' <= RID AND RID < '150000202111141936300083264192') 
b  (INSUTYPE in ('310')) and  ('150000202111141936300083264192' <= RID AND RID < '150000202111141939380097209616') 
b  (INSUTYPE in ('310')) and  ('150000202111141939380097209616' <= RID AND RID < '150000202111141942380010692106') 
a  (INSUTYPE in ('310')) and  ('150000202111141942380010692106' <= RID AND RID < '150000202111141954400065964500') 
b  (INSUTYPE in ('310')) and  ('150000202111141954400065964500' <= RID AND RID < '150000202111141956480075731953') 
b  (INSUTYPE in ('310')) and  ('150000202111141956480075731953' <= RID AND RID < '150000202111141957050077017772') 
b  (INSUTYPE in ('310')) and  ('150000202111141957050077017772' <= RID AND RID < '150000202111142000200091227087') 
b  (INSUTYPE in ('310')) and  ('150000202111142000200091227087' <= RID AND RID < '150000202111142004200007864399') 
x  (INSUTYPE in ('310')) and  ('150000202111142004200007864399' <= RID AND RID < '150000202111142010210032105719') 
x  (INSUTYPE in ('310')) and  ('150000202111142010210032105719' <= RID AND RID < '150000202111142020210069962674') 
b  (INSUTYPE in ('310')) and  ('150000202111142020210069962674' <= RID AND RID < '150000202111142024000084103761') 
b  (INSUTYPE in ('310')) and  ('150000202111142024000084103761' <= RID AND RID < '150000202111142027180095847236') 
b  (INSUTYPE in ('310')) and  ('150000202111142027180095847236' <= RID AND RID < '150000202111142030110006573311') 
b  (INSUTYPE in ('310')) and  ('150000202111142030110006573311' <= RID AND RID < '150000202111142044140057633722') 
x  (INSUTYPE in ('310')) and  ('150000202111142044140057633722' <= RID AND RID < '150000202111142057180006022373') 
b  (INSUTYPE in ('310')) and  ('150000202111142057180006022373' <= RID AND RID < '150000202111142059190014102556') 
b  (INSUTYPE in ('310')) and  ('150000202111142059190014102556' <= RID AND RID < '150000202111142101590022496265') 
x  (INSUTYPE in ('310')) and  ('150000202111142101590022496265' <= RID AND RID < '150000202111142107570046710236') 
b  (INSUTYPE in ('310')) and  ('150000202111142107570046710236' <= RID AND RID < '150000202111142111380061698072') 
b  (INSUTYPE in ('310')) and  ('150000202111142111380061698072' <= RID AND RID < '150000202111142113100068120046') 
b  (INSUTYPE in ('310')) and  ('150000202111142113100068120046' <= RID AND RID < '150000202111142122140004349472') 
b  (INSUTYPE in ('310')) and  ('150000202111142122140004349472' <= RID AND RID < '150000202111142125060015622749') 
a  (INSUTYPE in ('310')) and  ('150000202111142125060015622749' <= RID AND RID < '150000202111142137530061457315') 
b  (INSUTYPE in ('310')) and  ('150000202111142137530061457315' <= RID AND RID < '150000202111142141390074617059') 
b  (INSUTYPE in ('310')) and  ('150000202111142141390074617059' <= RID AND RID <= '150000202111142147380095865377') 
b  (INSUTYPE in ('310')) and  ((RID < '150000202111141816040068648282') OR ('150000202111142147380095865377' < RID)) 
b  (INSUTYPE in ('310')) and  RID IS NULL

在日志内找那些完成了

2021-11-17 05:44:35.623 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[45] is successed, used[718797]ms

其他未完成的都delete半截记录

delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141823140093105051' <= RID AND RID < '150000202111141836290032288915') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141844320055457938' <= RID AND RID < '150000202111141849170068916149') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141849400069987234' <= RID AND RID < '150000202111141850060071112917') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141901250000652263' <= RID AND RID < '150000202111141911280025548754') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141921020046212640' <= RID AND RID < '150000202111141932010063110436') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111141932010063110436' <= RID AND RID < '150000202111141936300083264192') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111142004200007864399' <= RID AND RID < '150000202111142010210032105719') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111142010210032105719' <= RID AND RID < '150000202111142020210069962674') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111142044140057633722' <= RID AND RID < '150000202111142057180006022373') ;
delete FROM staf_psn_clct_detl_d where CRTER_ID='NEUSOFT_HET'  and  (INSUTYPE in ('310')) and  ('150000202111142101590022496265' <= RID AND RID < '150000202111142107570046710236') ;
 然后开始重新传输，按照分片键

python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141823140093105051.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141842040048301370.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141844320055457938.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141849400069987234.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141901250000652263.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141921020046212640.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141932010063110436.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111141942380010692106.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111142004200007864399.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111142010210032105719.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111142044140057633722.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111142101590022496265.json
python ..\..\bin\datax.py  _PSN_CLCT_DETL_D150000202111142125060015622749.json

转换完毕核对总数即可 

附job.json

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "oraclereader",
          "parameter": {
            "column": [
              "\"PSN_CLCT_DETL_ID\"",
              "\"EMP_CLCT_DETL_ID\"",
              "\"CLCT_BILL_ID\"",
              "\"EMP_NO\"",
              "\"PSN_NO\"",
              "\"CASHYM\"",
              "\"ACCRYM_BEGN\"",
              "\"ACCRYM_END\"",
              "\"INSUTYPE\"",
              "\"PSN_TYPE\"",
              "\"PSN_IDET_TYPE\"",
              "\"INSU_IDET\"",
              "\"EMP_CLCTSTD\"",
              "\"PSN_CLCTSTD\"",
              "\"WAG\"",
              "\"QUOT_CLCT_FLAG\"",
              "\"EMP_CLCT_PARAVAL\"",
              "\"EMP_INTO_PARAVAL\"",
              "\"EMP_CLCT_AMT\"",
              "\"EMP_CLCT_INTO_ACCT_AMT\"",
              "\"PSN_CLCT_PARAVAL\"",
              "\"PSN_INTO_PARAVAL\"",
              "\"PSN_CLCT_AMT\"",
              "\"PSN_CLCT_INTO_ACCT_AMT\"",
              "\"OTH_CLCT_TRAF_AMT\"",
              "\"OTH_CLCT_AMT\"",
              "\"FINSUBS_AMT\"",
              "\"FINSUBS_TRAF_AMT\"",
              "\"TRAF_SUMAMT\"",
              "\"CLCT_SUMAMT\"",
              "\"INTE\"",
              "\"LATEFEE\"",
              "\"CLCT_FLAG\"",
              "\"CLCT_TYPE\"",
              "\"CLCT_TIME\"",
              "\"ARVLER\"",
              "\"ARVL_BCHNO\"",
              "\"REVS_FLAG\"",
              "\"TRAFER\"",
              "\"TRAF_TIME\"",
              "\"INTSURY_TIME\"",
              "\"URSN_TIME\"",
              "\"DCLA_PRD\"",
              "\"ELEC_TAXRPT_NO\"",
              "\"PEAWKR_FLAG\"",
              "\"POOLAREA_NO\"",
              "\"INSUTYPE_RETR_FLAG\"",
              "\"TAXDEPT_CODE\"",
              "\"PLAN_BCHNO\"",
              "\"PSN_INSU_RLTS_ID\"",
              "\"CLCTSTD_CRTF_RULE_CODG\"",
              "\"CLCT_RULE_TYPE_CODG\"",
              "\"INSU_CLCT_MONS\"",
              "\"INIT_PSN_NO\"",
              "\"INIT_PSN_CLCT_DETL_ID\"",
              "\"ACCTER\"",
              "\"CRTE_OPTINS_NO\"",
              "\"CRTER_ID\"",
              "\"CRTER_NAME\"",
              "\"CRTE_TIME\"",
              "\"OPTINS_NO\"",
              "\"OPTER_ID\"",
              "\"OPTER_NAME\"",
              "\"OPT_TIME\"",
              "\"RID\"",
              "\"UPDT_TIME\"",
              "\"BILL_FLAG\"",
              "\"TRAF_FLAG\""
            ],
            "where": "(INSUTYPE in ('310')) and  ('150000202111141844320055457938' <= RID AND RID < '150000202111141849170068916149') ",
            "splitPk": "",
            "password": "17648110166",
            "username": "whm_sjzh",
            "connection": [
              {
                "table": [
                  "STAF_PSN_CLCT_DETL_D"
                ],
                "jdbcUrl": [
                  "jdbc:oracle:thin:@//10.55.141.10/sjzlhetdb"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "column": [
              "`PSN_CLCT_DETL_ID`",
              "`EMP_CLCT_DETL_ID`",
              "`CLCT_BILL_ID`",
              "`EMP_NO`",
              "`PSN_NO`",
              "`CASHYM`",
              "`ACCRYM_BEGN`",
              "`ACCRYM_END`",
              "`INSUTYPE`",
              "`PSN_TYPE`",
              "`PSN_IDET_TYPE`",
              "`INSU_IDET`",
              "`EMP_CLCTSTD`",
              "`PSN_CLCTSTD`",
              "`WAG`",
              "`QUOT_CLCT_FLAG`",
              "`EMP_CLCT_PARAVAL`",
              "`EMP_INTO_PARAVAL`",
              "`EMP_CLCT_AMT`",
              "`EMP_CLCT_INTO_ACCT_AMT`",
              "`PSN_CLCT_PARAVAL`",
              "`PSN_INTO_PARAVAL`",
              "`PSN_CLCT_AMT`",
              "`PSN_CLCT_INTO_ACCT_AMT`",
              "`OTH_CLCT_TRAF_AMT`",
              "`OTH_CLCT_AMT`",
              "`FINSUBS_AMT`",
              "`FINSUBS_TRAF_AMT`",
              "`TRAF_SUMAMT`",
              "`CLCT_SUMAMT`",
              "`INTE`",
              "`LATEFEE`",
              "`CLCT_FLAG`",
              "`CLCT_TYPE`",
              "`CLCT_TIME`",
              "`ARVLER`",
              "`ARVL_BCHNO`",
              "`REVS_FLAG`",
              "`TRAFER`",
              "`TRAF_TIME`",
              "`INTSURY_TIME`",
              "`URSN_TIME`",
              "`DCLA_PRD`",
              "`ELEC_TAXRPT_NO`",
              "`PEAWKR_FLAG`",
              "`POOLAREA_NO`",
              "`INSUTYPE_RETR_FLAG`",
              "`TAXDEPT_CODE`",
              "`PLAN_BCHNO`",
              "`PSN_INSU_RLTS_ID`",
              "`CLCTSTD_CRTF_RULE_CODG`",
              "`CLCT_RULE_TYPE_CODG`",
              "`INSU_CLCT_MONS`",
              "`INIT_PSN_NO`",
              "`INIT_PSN_CLCT_DETL_ID`",
              "`ACCTER`",
              "`CRTE_OPTINS_NO`",
              "`CRTER_ID`",
              "`CRTER_NAME`",
              "`CRTE_TIME`",
              "`OPTINS_NO`",
              "`OPTER_ID`",
              "`OPTER_NAME`",
              "`OPT_TIME`",
              "`RID`",
              "`UPDT_TIME`",
              "`BILL_FLAG`",
              "`TRAF_FLAG`"
            ],
            "password": "4VbEw8Zo!sHJ5gn",
            "username": "clctcent",
            "connection": [
              {
                "table": [
                  "staf_psn_clct_detl_d"
                ],
                "jdbcUrl": "jdbc:mysql://192.168.7.137:5066/clctcent_db?serverTimezone=GMT%2B8"
              }
            ]
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "byte": -1,
        "channel": 10
      },
      "errorLimit": {
"record": 10000,
"percentage": 1
      }
    }
  }
}

大数据领域中Hadoop的数据迁移与整合方案
AI天才研究院
04-29 664
在企业数字化转型进程中，Hadoop作为主流大数据平台，常面临集群扩容、版本升级、多源数据融合等需求。集群演进：从Hadoop 2.x向3.x升级，或从自建集群迁移至云Hadoop（如AWS EMR、阿里云E-MapReduce）。多源融合：整合关系型数据库（MySQL/Oracle）、日志系统（ELK）、IoT设备等异构数据源到Hadoop数据湖。业务连续性：保障迁移过程中业务系统零中断，数据完整性与一致性不受破坏。
手把手教程本地调试Datax
最新发布
fox_233的博客
06-14 970
使用Datax做数仓同步数据得工具，有时需要自己开发或者修改某个reader或writer插件，那么本地调试就很重要。
断点续传功能原理
sxty0220 博客之家
11-13 2625
断点续传可以分为两部分：一部分是断点，一部分是续传。断点的由来是在下载过程中，将一个下载文件分成了多个部分，同时进行多个部分一起下载，当某个时间点，任务被暂停了，此时下载暂停的位置就是断点了。续传就是当一个未完成的下载任务再次开始时，会从上次的断点继续传送。 使用多线程断点续传下载的时候，将下载或上传任务（一个文件或一个压缩包）人为的划分为几个部分，每一个部分采用一个线程进行上...
实现HTTP断点续传
codestart的专栏
11-10 1381
在HTTP协议中，获取资源可以使用GET方法，但在使用GET方法指定一个资源后，会获取到该资源的所有字节。在HTTP协议中，提供Range属性，可以指定获取资源某一段数据。如使用telnet连接www.cz88.net 的80端口:GET /2004/7-8/0153-2.htm HTTP/1.1Host: www.cz88.netAccept: */*Referer: http://www.
文件下载（断点续传）——dataTask
lg767201403的博客
07-07 689
// 文件的存放路径（caches） #define FilePath [[NSSearchPathForDirectoriesInDomains(NSCachesDirectory, NSUserDomainMask, YES) lastObject] stringByAppendingPathComponent:@"test.mp4"] // 文件的已下载长度 #define Downloa
离线数据同步神器：DataX，支持几乎所有异构数据源的离线同步到MaxCompute
weixin_33856370的博客
11-13 434
概述 DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。 DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件...
【瀚高数据库同步秘籍】：DataX插件搭建至性能优化全攻略（含实战案例分析）
[【瀚高数据库同步秘籍】：DataX插件搭建至性能优化全攻略（含实战案例分析）](https://opengraph.githubassets.com/b048bb3a2ad86cc490e4965487b9343699ab7f5b403dca793eff02c5168644b2/niteshdm/datax) ...
【实战-01】flink cdc 实时数据同步利器
热门推荐
笔生花的博客
12-21 1万+
对很多初入门的人来说是无法理解cdc到底是什么个东西。有这样一个需求，比如在mysql数据库中存在很多数据，但是公司要把mysql中的数据同步到数据仓库(starrocks), 数据仓库你可以理解为存储了各种各样来自不同数据库中表。数据的同步目前对mysql来说比较常见是方式是使用：datax 和 canal配合， 为什么需要这两个框架配合呢？因为datax不支持实时的同步， datax只能定义一个范围去同步，而且同步结束后程序就结束了。
【自定义DataX瀚高reader插件】：从零开始到高级用法
本文深入探讨了DataX瀚高reader插件的开发、实战演练以及优化与扩展。首先介绍了DataX架构和瀚高数据库的特点，阐述了插件开发的基础环境搭建与核心代码解析。接着，通过实战演练章节，详细讲解了reader插件的编写、...
【深入DataX架构】：瀚高同步插件工作机制与性能优化技巧揭秘
DataX是一个开源的数据同步工具，本论文对DataX架构进行概述，并深入探讨瀚高同步插件的工作机制及其性能优化。瀚高同步插件通过特定的数据传输流程和核心组件实现高效的数据同步，同时具备异常处理策略以保障数据...
DataLink是一个满足各种异构数据源之间的实时增量同步分布式可扩展的数据交换平台
08-08
DataLink是一个满足各种异构数据源之间的实时增量同步，分布式、可扩展的数据交换平台
datax-web进行本地代码调试
王傲旗的大数据之路
08-10 1737
如何进行本地对datax-web项目进行Debug 先来思考一个问题：为什么要在本地进行Debug? 答： 但凡是结果几天代码的同学都应该明白，我们写完代码之后打成jar包扔到linux服务器中去跑，最多也就看看错误日志出现在了什么地方，但是如果在本地的IDE中进行debug就可以快速的定位到问题！ 实操一下 1.将最新的dev分支代码拉到本地 2.修改配置文件 admin分支下的bootstrap.properties文件 配置本地datax的地址 3.启动项目 首先启动admin模块下的启动类 然
idea本地debug调试DATAX插件运行
b452608的专栏
01-30 2850
接触datax是看重他的数据采集功能和可以扩展插件的功能，根据官方说明，DATAX是以python来调用插件的运行json配置，但对于二次开发插件的同学来说，需要本地调试。VM options处填入 -Ddatax.home=C:\Users\clj\Documents\DataXPlus\target\datax\datax，具体路径根据你自己项目而定。需要注意的时，这种方式调试如果修改了代码，需要重新在DATAX目录执行打包后，新的代码才能生效，才能进入断点。1.源码部署方式请查阅。
ETL工具之datax
runqu的博客
04-02 2041
DataX是由阿里巴巴集团开发的开源数据同步工具，用于实现大数据集群内数据的离线同步。它主要用于数据仓库的数据导入和导出，如将关系型数据库中的数据导入到Hadoop集群中，或者将Hadoop集群中的数据导出到关系型数据库中。可扩展性：DataX支持多种数据源和目标的数据同步，如关系型数据库、Hadoop集群、Hive、HBase等；高性能：DataX利用多线程和流式计算技术，实现高速数据迁移；
数据同步工具调研选型：SeaTunnel 与 DataX 、Sqoop、Flume、Flink CDC 对比
weixin_54625990的博客
11-14 5930
Apache SeaTunnel 是一个非常易用的超高性能分布式数据集成产品，支持海量数据的离线及实时同步。每天可稳定高效同步万亿级数据，已应用于数百家企业生产，也是首个由国人主导贡献到 Apache 基金会的数据集成顶级项目。SeaTunnel 主要解决数据集成领域的常见问题：数据源多样：常用的数据源有数百种，版本不兼容。随着新技术的出现，出现了更多的数据源。用户很难找到能够全面快速支持这些数据源的工具。复杂同步场景。
开源共建 | 中国移动冯江涛：ChunJun（原FlinkX）在数据入湖中的应用
a958014226的博客
11-23 534
ChunJun（原FlinkX）是一个基于 Flink 提供易用、稳定、高效的批流统一的数据集成工具。2018年4月，秉承着开源共享的理念，数栈技术团队在github上开源了FlinkX，承蒙各位开发者的合作共建，FlinkX得到了快速发展。 两年后的2022年4月，技术团队决定对FlinkX进行整体升级，并更名为ChunJun，希望继续和各位优秀开发者合作，进一步推动数据集成/同步的技术发展。 因该文创作于于FlinkX更名为ChunJun之前，因此文中仍用FlinkX来进行分享，重要的事情说三遍： Fl
CDC实时数据同步
顺其自然~专栏
01-24 408
CDC实时数据同步指的是Change Data Capture（数据变更捕获）技术在数据同步过程中的应用。CDC技术允许在数据源发生变化时，实时地捕获这些变化，并将其应用到目标系统中，从而保持数据的同步性。CDC实时数据同步具有以下优点：1）实时性：能够几乎实时地将数据变更同步到目标系统中，保持数据的实时性。2）高效性：只传输变更数据，减少了数据传输量，降低了网络带宽和存储需求。3）灵活性：能够支持不同类型的数据源和目标系统，并适应复杂的数据同步需求。
etl调度工具 ctm_实用数据库同步工具汇总，帮你找到迁移转换的“最优”选择！...
weixin_39531037的博客
10-21 1849
引言作为资深的DBA同胞你是否在工作中也存在这样的情况呢？公司要搭建数据平台，首要的工作就是把旧库的数据导入到新库中，虽然各种数据库都提供了导入导出的工具，但是数据需要存储到各个地方，MongoDB、HBase、MySQL、Oracle等各种各样的不同数据库，同步起来就有一些困难了。那么如何做好两个数据库之间、不同类型的数据库之间的相互迁移转换呢？今天小编就常用的数据库同步、迁移转换工具进行一个汇...
阿里云DataWorks数据集成（DataX）架构&实践分享
敲代码的乔帮主
09-29 7060
阿里云DataWorks数据集成（DataX）架构&实践分享 分享嘉宾：罗海伟阿里云 编辑整理：约理中国科学院大学 目录 阿里云DataWorks数据集成（DataX）架构&实践分享 ▌为什么需要数据集成 数据集成的应用场景 数据集成的角色和地位 ▌阿里云数据集成大事记 ▌阿里云数据集成产品定位和业务支撑情况 阿里云数据集成产品定位 阿里云数据集成业务支撑情况 ▌阿里云数据集成DataX设计核心思路 离线数据同步原理 实时数据同步原理 ▌阿里云数据集成Dat..
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值