DataX系列9-DataX同步案例

最新推荐文章于 2024-02-03 08:00:44 发布
最新推荐文章于 2024-02-03 08:00:44 发布
阅读量251 收藏
点赞数
分类专栏： 大数据和数据仓库 # ETL 文章标签： hive mysql DataX MySQL DataX Hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010520724/article/details/122042232
版权

文章目录

一. MySQL同步到Hive

1.1 MySQL数据准备

MySQL测试数据准备

1.2 Hive端提前建表

CREATE TABLE `ods_fact_sale_orc_datax`(
  `id` bigint, 
  `sale_date` string, 
  `prod_name` string, 
  `sale_nums` int)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS ORC;

1.3 json配置文件准备

{
    "setting": {},
    "job": {
        "setting": {
            "speed": {
                "channel": 2
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "abc123",
                        "column": [
                            "id",
                            "sale_date",
                            "prod_name",
                            "sale_nums"
                        ],
                        "splitPk": "id",
                        "connection": [
                            {
                                "table": [
                                    "fact_sale"
                                ],
                                "jdbcUrl": [
     "jdbc:mysql://10.31.1.122:3306/test"
                                ]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "hdfswriter",
                    "parameter": {
                        "defaultFS": "hdfs://10.31.1.123:8020",
                        "fileType": "orc",
                        "path": "/user/hive/warehouse/test.db/ods_fact_sale_orc_datax",
                        "fileName": "ods_fact_sale_datax",
                        "column": [
                            {
                                "name": "id",
                                "type": "bigint"
                            },
                            {
                                "name": "sale_date",
                                "type": "string"
                            },
                            {
                                "name": "prod_name",
                                "type": "string"
                            },
                            {
                                "name": "sale_nums",
                                "type": "int"
                            }
                        ],
                        "writeMode": "append",
                        "fieldDelimiter": "\t",
                        "compress":"NONE"
                    }
                }
            }
        ]
    }
}

1.4 测试记录

代码:

 python datax.py  --jvm="-Xms3G -Xmx3G" hdfsreader.json

测试记录:

[15:13:58] [root@10-31-1-119 bin]# python datax.py --jvm="-Xms3G -Xmx3G" mysqltohdfs.json
[15:13:58] 
[15:13:58] DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
[15:13:58] Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
[15:13:58] 
[15:13:58] 
[15:13:59] 2021-11-24 15:14:06.967 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
[15:13:59] 2021-11-24 15:14:06.974 [main] INFO  Engine - the machine info  => 
[15:13:59] 
[15:13:59]         osInfo: Oracle Corporation 1.8 25.242-b08
[15:13:59]         jvmInfo:        Linux amd64 3.10.0-1127.el7.x86_64
[15:13:59]         cpu num:        8
[15:13:59] 
[15:13:59]         totalPhysicalMemory:    -0.00G
[15:13:59]         freePhysicalMemory:     -0.00G
[15:13:59]         maxFileDescriptorCount: -1
[15:13:59]         currentOpenFileDescriptorCount: -1
[15:13:59] 
[15:13:59]         GC Names        [PS MarkSweep, PS Scavenge]
[15:13:59] 
[15:13:59]         MEMORY_NAME                    | allocation_size                | init_size                      
[15:13:59]         PS Eden Space                  | 768.00MB                       | 768.00MB                       
[15:13:59]         Code Cache                     | 240.00MB                       | 2.44MB                         
[15:13:59]         Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
[15:13:59]         PS Survivor Space              | 128.00MB                       | 128.00MB                       
[15:13:59]         PS Old Gen                     | 2,048.00MB                     | 2,048.00MB                     
[15:13:59]         Metaspace                      | -0.00MB                        | 0.00MB                         
[15:13:59] 
[15:13:59] 
[15:13:59] 2021-11-24 15:14:06.989 [main] INFO  Engine - 
[15:13:59] {
[15:13:59]         "content":[
[15:13:59]                 {
[15:13:59]                         "reader":{
[15:13:59]                                 "name":"mysqlreader",
[15:13:59]                                 "parameter":{
[15:13:59]                                         "column":[
[15:13:59]                                                 "id",
[15:13:59]                                                 "sale_date",
[15:13:59]                                                 "prod_name",
[15:13:59]                                                 "sale_nums"
[15:13:59]                                         ],
[15:13:59]                                         "connection":[
[15:13:59]                                                 {
[15:13:59]                                                         "jdbcUrl":[
[15:13:59]                                                                 "jdbc:mysql://10.31.1.122:3306/test"
[15:13:59]                                                         ],
[15:13:59]                                                         "table":[
[15:13:59]                                                                 "fact_sale"
[15:13:59]                                                         ]
[15:13:59]                                                 }
[15:13:59]                                         ],
[15:13:59]                                         "password":"******",
[15:13:59]                                         "splitPk":"id",
[15:13:59]                                         "username":"root"
[15:13:59]                                 }
[15:13:59]                         },
[15:13:59]                         "writer":{
[15:13:59]                                 "name":"hdfswriter",
[15:13:59]                                 "parameter":{
[15:13:59]                                         "column":[
[15:13:59]                                                 {
[15:13:59]                                                         "name":"id",
[15:13:59]                                                         "type":"bigint"
[15:13:59]                                                 },
[15:13:59]                                                 {
[15:13:59]                                                         "name":"sale_date",
[15:13:59]                                                         "type":"string"
[15:13:59]                                                 },
[15:13:59]                                                 {
[15:13:59]                                                         "name":"prod_name",
[15:13:59]                                                         "type":"string"
[15:13:59]                                                 },
[15:13:59]                                                 {
[15:13:59]                                                         "name":"sale_nums",
[15:13:59]                                                         "type":"int"
[15:13:59]                                                 }
[15:13:59]                                         ],
[15:13:59]                                         "compress":"NONE",
[15:13:59]                                         "defaultFS":"hdfs://10.31.1.123:8020",
[15:13:59]                                         "fieldDelimiter":"\t",
[15:13:59]                                         "fileName":"ods_fact_sale_datax",
[15:13:59]                                         "fileType":"orc",
[15:13:59]                                         "path":"/user/hive/warehouse/test.db/ods_fact_sale_orc_datax",
[15:13:59]                                         "writeMode":"append"
[15:13:59]                                 }
[15:13:59]                         }
[15:13:59]                 }
[15:13:59]         ],
[15:13:59]         "setting":{
[15:13:59]                 "speed":{
[15:13:59]                         "channel":2
[15:13:59]                 }
[15:13:59]         }
[15:13:59] }
[15:13:59] 
[15:13:59] 2021-11-24 15:14:07.002 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
[15:13:59] 2021-11-24 15:14:07.005 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
[15:13:59] 2021-11-24 15:14:07.006 [main] INFO  JobContainer - DataX jobContainer starts job.
[15:13:59] 2021-11-24 15:14:07.010 [main] INFO  JobContainer - Set jobId = 0
[15:13:59] 2021-11-24 15:14:07.291 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://10.31.1.122:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
[15:13:59] 2021-11-24 15:14:07.308 [job-0] INFO  OriginalConfPretreatmentUtil - table:[fact_sale] has columns:[id,sale_date,prod_name,sale_nums,reserverd1].
[15:13:59] 十一月 24, 2021 3:14:07 下午 org.apache.hadoop.util.NativeCodeLoader <clinit>
[15:13:59] 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[15:14:00] 2021-11-24 15:14:08.104 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
[15:14:00] 2021-11-24 15:14:08.105 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
[15:14:00] 2021-11-24 15:14:08.105 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] do prepare work .
[15:14:00] 2021-11-24 15:14:08.178 [job-0] INFO  HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/test.db/ods_fact_sale_orc_datax] 目录下写入相应文件名前缀  [ods_fact_sale_datax] 的文件
[15:14:00] 2021-11-24 15:14:08.179 [job-0] INFO  JobContainer - jobContainer starts to do split ...
[15:14:00] 2021-11-24 15:14:08.179 [job-0] INFO  JobContainer - Job set Channel-Number to 2 channels.
[15:14:00] 2021-11-24 15:14:08.194 [job-0] INFO  SingleTableSplitUtil - split pk [sql=SELECT MIN(id),MAX(id) FROM fact_sale] is running... 
[15:14:00] 2021-11-24 15:14:08.219 [job-0] INFO  SingleTableSplitUtil - After split(), allQuerySql=[
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (1 <= id AND id < 78762161) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (78762161 <= id AND id < 157524321) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (157524321 <= id AND id < 236286481) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (236286481 <= id AND id < 315048641) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (315048641 <= id AND id < 393810801) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (393810801 <= id AND id < 472572961) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (472572961 <= id AND id < 551335120) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (551335120 <= id AND id < 630097279) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (630097279 <= id AND id < 708859438) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  (708859438 <= id AND id <= 787621597) 
[15:14:00] select id,sale_date,prod_name,sale_nums from fact_sale  where  id IS NULL
[15:14:00] ].
[15:14:00] 2021-11-24 15:14:08.219 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] splits to [11] tasks.
[15:14:00] 2021-11-24 15:14:08.219 [job-0] INFO  HdfsWriter$Job - begin do split...
[15:14:00] 2021-11-24 15:14:08.223 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__bd91f932_1225_4aa3_90cf_7e5c15c5537f]
[15:14:00] 2021-11-24 15:14:08.224 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__328e36ec_e8e4_4147_95f4_83748bc18a73]
[15:14:00] 2021-11-24 15:14:08.224 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__871e2e30_3818_4c36_87d8_3142bb1b2703]
[15:14:00] 2021-11-24 15:14:08.224 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__13e6edef_d40a_4500_8b22_71588d17e4e9]
[15:14:00] 2021-11-24 15:14:08.225 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__df33a43d_e258_4fb1_ae2c_d5259fd1a554]
[15:14:00] 2021-11-24 15:14:08.225 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__3b0f910c_67fb_421b_a4f4_bed340f035db]
[15:14:00] 2021-11-24 15:14:08.225 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__e904b458_efc3_4191_a1d5_6594c86f3f75]
[15:14:00] 2021-11-24 15:14:08.226 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__fc598d87_4dc4_4c31_9289_59b6a143e471]
[15:14:00] 2021-11-24 15:14:08.226 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__ac9c6e09_99d9_40a7_acd0_18032961cd15]
[15:14:00] 2021-11-24 15:14:08.226 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__94aeb798_65df_4b7a_8ef4_97adaf99b5ab]
[15:14:00] 2021-11-24 15:14:08.227 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__4b4a7925_88db_442f_80cb_c270dc034b6b]
[15:14:00] 2021-11-24 15:14:08.227 [job-0] INFO  HdfsWriter$Job - end do split.
[15:14:00] 2021-11-24 15:14:08.227 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] splits to [11] tasks.
[15:14:00] 2021-11-24 15:14:08.245 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
[15:14:00] 2021-11-24 15:14:08.254 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
[15:14:00] 2021-11-24 15:14:08.256 [job-0] INFO  JobContainer - Running by standalone Mode.
[15:14:00] 2021-11-24 15:14:08.267 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [2] channels for [11] tasks.
[15:14:00] 2021-11-24 15:14:08.273 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
[15:14:00] 2021-11-24 15:14:08.273 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
[15:14:00] 2021-11-24 15:14:08.286 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
[15:14:00] 2021-11-24 15:14:08.289 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select id,sale_date,prod_name,sale_nums from fact_sale  where  (1 <= id AND id < 78762161) 
[15:14:00] ] jdbcUrl:[jdbc:mysql://10.31.1.122:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
[15:14:00] 2021-11-24 15:14:08.289 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
[15:14:00] 2021-11-24 15:14:08.291 [0-0-1-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select id,sale_date,prod_name,sale_nums from fact_sale  where  (78762161 <= id AND id < 157524321) 
[15:14:00] ] jdbcUrl:[jdbc:mysql://10.31.1.122:3306/test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
[15:14:00] 2021-11-24 15:14:08.312 [0-0-1-writer] INFO  HdfsWriter$Task - begin do write...
[15:14:00] 2021-11-24 15:14:08.312 [0-0-1-writer] INFO  HdfsWriter$Task - write to file : [hdfs://10.31.1.123:8020/user/hive/warehouse/test.db/ods_fact_sale_orc_datax__dcc27788_0305_445b_acaa_54c8e61a99c4/ods_fact_sale_datax__328e36ec_e8e4_4147_95f4_83748bc18a73]
[15:14:00] 2021-11-24 15:14:08.312 [0-0-0-writer] INFO  HdfsWriter$Task - begin do write...
[15:14:00] 2021-11-24 15:14:08.312 [0-0-0-writer] INFO  HdfsWriter$Task - write to file : 
..................................................................
..................................................................
..................................................................
..................................................................
.................................................................
...................................................................
[15:42:10] 2021-11-24 15:42:18.638 [job-0] INFO  JobContainer - 
[15:42:10]          [total cpu info] => 
[15:42:10]                 averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
[15:42:10]                 -1.00%                         | -1.00%                         | -1.00%
[15:42:10]                         
[15:42:10] 
[15:42:10]          [total gc info] => 
[15:42:10]                  NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
[15:42:10]                  PS MarkSweep         | 4                  | 1                  | 0                  | 0.285s             | 0.096s             | 0.000s             
[15:42:10]                  PS Scavenge          | 3131               | 590                | 256                | 24.641s            | 5.802s             | 2.277s             
[15:42:10] 
[15:42:10] 2021-11-24 15:42:18.638 [job-0] INFO  JobContainer - PerfTrace not enable!
[15:42:10] 2021-11-24 15:42:18.639 [job-0] INFO  StandAloneJobContainerCommunicator - Total 767830000 records, 18367983662 bytes | Speed 10.37MB/s, 454337 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 2,272.990s |  All Task WaitReaderTime 16.823s | Percentage 100.00%
[15:42:10] 2021-11-24 15:42:18.640 [job-0] INFO  JobContainer - 
[15:42:10] 任务启动时刻                    : 2021-11-24 15:14:07
[15:42:10] 任务结束时刻                    : 2021-11-24 15:42:18
[15:42:10] 任务总计耗时                    :               1691s
[15:42:10] 任务平均流量                    :           10.37MB/s
[15:42:10] 记录写入速度                    :         454337rec/s
[15:42:10] 读出记录总数                    :           767830000
[15:42:10] 读写失败总数                    :                   0
[15:42:10]
只是甲
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
datax的json范例
08-26
datax的简单范例. 有一些解释,适用于初学者
Datax入门案例
最新发布
m0_63069778的博客
04-04 708
首先，确保已经安装好mysql和datax。本案例实现从mysql同步数据到hive，从hive同步数据到datax。
DataX 概述、部署、数据同步运用示例
JIE的博客 --- moon_coder
09-15 3245
DataX是阿里巴巴集团开源的、通用的数据抽取工具，广泛使用的离线数据同步工具/平台。它设计用于支持多种数据源之间的高效数据传输，可以实现不同数据源之间的数据同步、迁移、ETL（抽取、转换、加载）等数据操作。多数据源支持：DataX支持多种数据源，包括关系型数据库（如 MySQL、Oracle、SQL Server）、NoSQL 数据库（如 MongoDB、HBase、Redis）、HDFS、FTP、Hive 等。扩展性。
DataX案例实操
prefect_start的博客
02-04 1855
DataX案例实操，DataX案例实操，DataX案例实操
阿里出品dataX入门案例
东城庞太师
05-16 222
前提：dataX安装部署完毕，环境验证OK。 案例： 1.在dataX的job目录下创建一个stream2stream.json 文件 内容如下： { "job": { "content": [ { "reader": { "name": "streamreader", "parameter": { .
dataX+xxl-job+openjdk11+默认执行器+python
07-07
【标题】"dataX+xxl-job+openjdk11+默认执行器+python" 提供了一种基于Java和Python的分布式任务调度解决方案，结合了dataX数据同步工具、XXL-JOB任务调度框架以及OpenJDK 11运行环境。这个组合用于实现高效的数据...
DataX Web分布式数据同步工具 v2.1.2.zip
03-18
DataX Web是一款强大的分布式数据同步工具，主要用于大数据领域中的数据迁移和同步任务。在这个v2.1.2版本的压缩包中，用户可以获取到源码以及相关的系统软件工具，适用于毕业设计论文、计算机案例研究或者模板建站...
datax-clickhousereader
02-21
DataX是阿里巴巴开源的一款高效、稳定、强大的数据同步工具，它支持多种数据源之间的数据迁移。在本案例中，我们关注的是DataX中的ClickHouse Reader插件，用于从ClickHouse数据库中读取数据并进行进一步的操作。 ...
datax的json范例--datax的简单范例. 有一些解释,适用于初学者
07-03
DataX是阿里巴巴开源的一款强大的数据同步工具，它能够高效、稳定地进行大数据迁移工作，支持多种数据源之间的数据同步。这个压缩包中的内容是关于DataX的JSON范例和一个相关的图片，对于初学者来说是很好的学习资源...
datax-python3.rar
05-30
DataX 是阿里巴巴开源的一款高效、稳定、强大的数据同步工具，它能够支持各种数据源之间的数据迁移。在Python 3环境下使用 DataX，我们需要对原有的Python 2版本进行适配，因为Python 2与Python 3在语法上存在一些...
dataX数据同步
yabin1016的博客
05-24 495
dataX linux环境部署： 1 参考文章： https://blog.csdn.net/Titanium_LL/article/details/100859499 2 环境参数：centos7 3 crontab定时任务参考https://www.pianshen.com/article/8309173488/ crontab不执行报错解决： 解决 sh: java: command not found 问题 在执行脚本上加入如下配置即可 #!/bin/bash JAV...
阿里又开源一款数据同步工具 DataX，稳定又高效，好用到爆！
zhangzhensea的专栏
03-09 1337
阿里数据库备份工具
【Spark数仓项目】需求八：MySQL的DataX全量导入和增量导入Hive
qq_31412425的博客
07-14 1902
本需求将模拟从MySQL中向Hive数仓中导入数据，数据以时间分区。测试两种导入场景，一种是将数据全量导入，即包含所有时间分区；另一种是每天运行调度，仅导入当天时间分区中的用户数据。mysql表建表语句：hive注意字段时间戳，我们将从以上MySQL向Hive导入数据。编写datax的json脚本"job" : {执行导入操作在mysql中添加测试数据 导入mysql中7-11的数据到hive下7-11分区在hive下创建分区运行dataX脚本。
DataX基本使用教程
bababuzaijia的博客
02-25 1045
DataX基本使用教程
阿里又开源一款数据同步工具DataX，稳定又高效，好用到爆！
emprere的博客
02-03 263
因公众号更改推送规则，请点“在看”并加“星标”第一时间获取精彩技术分享点击关注#互联网架构师公众号，领取架构师全套资料 都在这里0、2T架构师学习资料干货分上一篇：2T架构师学习资料干货分享大家好，我是互联网架构师！前言有个项目的数据量高达五千万，但是因为报表那块数据不太准确，业务库和报表库又是跨库操作，所以并不能使用 SQL 来进行同步。当时的打算是通过mysqldump或者存储的方式来进行...
使用DataX和sqoop将数据从MySQL导入Hive
Davidchou3165的博客
10-23 3230
使用DataX和Sqoop两种方式进行数据同步，对比两种技术的优劣
【datax】mysql同步数据到hive
weixin_46680124的博客
09-22 378
datax同步mysql数据到hive
datax 模板_Datax入门 - 阿伦哥的博客 - OSCHINA - 中文开源技术交流社区
weixin_32823993的博客
12-30 126
第一步、创建创业的配置文件(json格式)可以通过命令查看配置模板： python datax.py -r {YOUR_READER} -w {YOUR_WRITER}如：python datax.py -r streamreader -w streamwriterzlh@zlhdeMacBook-Pro:~/software/datax/bin$ python datax.py -r strea...
批量数据之DataX数据同步
上善若泪
01-26 1360
DataX是阿里云DataWorks数据集成 的开源版本，主要就是用于实现数据间的离线同步。DataX致力于实现包括关系型数据库（MySQL、Oracle 等）、HDFS、Hive、ODPS、HBase、FTP 等 各种异构数据源（即不同的数据库） 间稳定高效的数据同步功能。为了解决异构数据源同步问题，DataX将复杂的网状同步链路变成了星型数据链路 ，DataX作为中间传输载体负责连接各种数据源；当需要接入一个新的数据源时，只需要将此数据源对接到DataX，便能跟已有的数据源作为无缝数据同步。
datax-web-2.1.2
07-27
datax-web-2.1.2是一个版本为2.1.2的DataX Web工具。要安装datax-web-2.1.2，您可以按照以下步骤进行操作： 1. 下载datax-web-2.1.2.tar.gz压缩包，并将其放置在服务器上。 2. 解压缩datax-web-2.1.2.tar.gz文件，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值