hadoop distcp 同步集群数据

命令1:hadoop distcp -update -delete -p hdfs://Smaster:9000//newexchange  hdfs://Sslave0:9000/newexchange

命令2:hadoop distcp -update -delete -p  webhdfs://192.168.88.22:50070/mysql_datas      webhdfs://Sslave0:50070/mysql_datas

hadoop distcp  源数据    复制到备份服务器

-update -delete -p  删除本地的可以不加

命令1 别介绍是同版本复制同步，命令2 不同hadoop版本复制同步 但命令1 我在hadoop3.0.3 获取 hadoop2.6.0数据可以用

hadoop@Sslave0 mapreduce]$ hdfs dfs -ls  /    #hadoop-3.0.3
Found 3 items
drwxr-xr-x   - hadoop supergroup          0 2018-08-03 15:42 /hive2.3
drwxrwxrwx   - hadoop supergroup          0 2018-08-03 16:09 /tmp
drwxr-xr-x   - hadoop supergroup          0 2018-08-03 13:54 /user

[hadoop@Sslave0 mapreduce]$ hdfs dfs -ls hdfs://Smaster:9000/     #hadoop-2.6.0             
Found 10 items
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 16:22 hdfs://Smaster:9000/check
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 17:40 hdfs://Smaster:9000/dsqdata
drwxrwxrwx   - hadoop supergroup          0 2018-08-02 16:05 hdfs://Smaster:9000/hive
drwxr-xr-x   - hadoop supergroup          0 2018-08-03 15:55 hdfs://Smaster:9000/mysql_datas
drwxr-xr-x   - hadoop supergroup          0 2018-08-03 11:08 hdfs://Smaster:9000/newexchange
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 16:25 hdfs://Smaster:9000/spark_jars
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 16:25 hdfs://Smaster:9000/test
drwxrwxrwx   - hadoop supergroup          0 2018-08-03 10:43 hdfs://Smaster:9000/tmp
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 16:11 hdfs://Smaster:9000/user
drwxr-xr-x   - hadoop supergroup          0 2018-08-02 16:25 hdfs://Smaster:9000/ziang_test

[hadoop@Sslave0 ~]$ hadoop distcp webhdfs://Smaster:50070/mysql_datas      webhdfs://Sslave0:50070/mysql_datas             
2018-08-03 16:23:36,464 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[webhdfs://192.168.88.22:50070/mysql_datas], targetPath=webhdfs://Sslave0:50070/mysql_datas, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false}, sourcePaths=[webhdfs://192.168.88.22:50070/mysql_datas], targetPathExists=false, preserveRawXattrsfalse
2018-08-03 16:23:36,627 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-08-03 16:23:38,310 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 26; dirCnt = 1
2018-08-03 16:23:38,310 INFO tools.SimpleCopyListing: Build file listing completed.
2018-08-03 16:23:38,312 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
2018-08-03 16:23:38,312 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
2018-08-03 16:23:38,468 INFO tools.DistCp: Number of paths in the copy list: 26
2018-08-03 16:23:38,546 INFO tools.DistCp: Number of paths in the copy list: 26
2018-08-03 16:23:38,567 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-08-03 16:23:38,693 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1533283659094_0006
2018-08-03 16:23:38,999 INFO mapreduce.JobSubmitter: number of splits:5
2018-08-03 16:23:39,064 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-08-03 16:23:39,265 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533283659094_0006
2018-08-03 16:23:39,267 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-08-03 16:23:39,534 INFO conf.Configuration: resource-types.xml not found
2018-08-03 16:23:39,534 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-08-03 16:23:39,642 INFO impl.YarnClientImpl: Submitted application application_1533283659094_0006
2018-08-03 16:23:39,721 INFO mapreduce.Job: The url to track the job: http://Sslave0:8088/proxy/application_1533283659094_0006/
2018-08-03 16:23:39,721 INFO tools.DistCp: DistCp job-id: job_1533283659094_0006
2018-08-03 16:23:39,723 INFO mapreduce.Job: Running job: job_1533283659094_0006
2018-08-03 16:23:48,934 INFO mapreduce.Job: Job job_1533283659094_0006 running in uber mode : false
2018-08-03 16:23:48,936 INFO mapreduce.Job:  map 0% reduce 0%
2018-08-03 16:24:07,254 INFO mapreduce.Job:  map 20% reduce 0%
2018-08-03 16:24:10,281 INFO mapreduce.Job:  map 40% reduce 0%
2018-08-03 16:24:21,376 INFO mapreduce.Job:  map 60% reduce 0%
2018-08-03 16:24:22,383 INFO mapreduce.Job:  map 80% reduce 0%
2018-08-03 16:24:29,452 INFO mapreduce.Job:  map 100% reduce 0%
2018-08-03 16:24:30,476 INFO mapreduce.Job: Job job_1533283659094_0006 completed successfully
2018-08-03 16:24:30,994 INFO mapreduce.Job: Counters: 42
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=1036715
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=6315
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=40
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=10
                WEBHDFS: Number of bytes read=9365494
                WEBHDFS: Number of bytes written=9365494
                WEBHDFS: Number of read operations=232
                WEBHDFS: Number of large read operations=0
                WEBHDFS: Number of write operations=76
        Job Counters
                Launched map tasks=5
                Other local map tasks=5
                Total time spent by all maps in occupied slots (ms)=127246
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=63623
                Total vcore-milliseconds taken by all map tasks=63623
                Total megabyte-milliseconds taken by all map tasks=130299904
        Map-Reduce Framework
                Map input records=26
                Map output records=0
                Input split bytes=685
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=907
                CPU time spent (ms)=12660
                Physical memory (bytes) snapshot=1198813184
                Virtual memory (bytes) snapshot=16667459584
                Total committed heap usage (bytes)=822607872
                Peak Map Physical memory (bytes)=252973056
                Peak Map Virtual memory (bytes)=3337940992
        File Input Format Counters
                Bytes Read=5630
        File Output Format Counters
                Bytes Written=0
        DistCp Counters
                Bandwidth in Btyes=5871336
                Bytes Copied=9365494
                Bytes Expected=9365494
                Files Copied=25
                DIR_COPY=1

[hadoop@Sslave0 ~]$ hadoop distcp -update -delete -p  webhdfs://Smaster:50070/mysql_datas      webhdfs://Sslave0:50070/mysql_datas
2018-08-03 16:27:25,141 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=true, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[REPLICATION, BLOCKSIZE, USER, GROUP, PERMISSION, CHECKSUMTYPE, TIMES], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[webhdfs://192.168.88.22:50070/mysql_datas], targetPath=webhdfs://Sslave0:50070/mysql_datas, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false}, sourcePaths=[webhdfs://192.168.88.22:50070/mysql_datas], targetPathExists=true, preserveRawXattrsfalse
2018-08-03 16:27:25,326 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-08-03 16:27:27,555 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 25; dirCnt = 0
2018-08-03 16:27:27,555 INFO tools.SimpleCopyListing: Build file listing completed.
2018-08-03 16:27:27,557 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
2018-08-03 16:27:27,558 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
2018-08-03 16:27:27,739 INFO tools.DistCp: Number of paths in the copy list: 25
2018-08-03 16:27:27,833 INFO tools.DistCp: Number of paths in the copy list: 25
2018-08-03 16:27:27,858 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2018-08-03 16:27:27,991 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1533283659094_0008
2018-08-03 16:27:28,238 INFO mapreduce.JobSubmitter: number of splits:3
2018-08-03 16:27:28,298 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-08-03 16:27:28,471 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1533283659094_0008
2018-08-03 16:27:28,473 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-08-03 16:27:28,822 INFO conf.Configuration: resource-types.xml not found
2018-08-03 16:27:28,823 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-08-03 16:27:28,964 INFO impl.YarnClientImpl: Submitted application application_1533283659094_0008
2018-08-03 16:27:29,031 INFO mapreduce.Job: The url to track the job: http://Sslave0:8088/proxy/application_1533283659094_0008/
2018-08-03 16:27:29,032 INFO tools.DistCp: DistCp job-id: job_1533283659094_0008
2018-08-03 16:27:29,033 INFO mapreduce.Job: Running job: job_1533283659094_0008
2018-08-03 16:27:44,220 INFO mapreduce.Job: Job job_1533283659094_0008 running in uber mode : false
2018-08-03 16:27:44,221 INFO mapreduce.Job:  map 0% reduce 0%
2018-08-03 16:27:59,632 INFO mapreduce.Job:  map 33% reduce 0%
2018-08-03 16:28:00,703 INFO mapreduce.Job:  map 67% reduce 0%
2018-08-03 16:28:04,737 INFO mapreduce.Job:  map 100% reduce 0%
2018-08-03 16:28:06,772 INFO mapreduce.Job: Job job_1533283659094_0008 completed successfully
2018-08-03 16:28:07,072 INFO mapreduce.Job: Counters: 40
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=622029
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=5309
                HDFS: Number of bytes written=1488
                HDFS: Number of read operations=24
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=6
                WEBHDFS: Number of bytes read=0
                WEBHDFS: Number of bytes written=0
                WEBHDFS: Number of read operations=128
                WEBHDFS: Number of large read operations=0
                WEBHDFS: Number of write operations=25
        Job Counters
                Launched map tasks=3
                Other local map tasks=3
                Total time spent by all maps in occupied slots (ms)=62674
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=31337
                Total vcore-milliseconds taken by all map tasks=31337
                Total megabyte-milliseconds taken by all map tasks=64178176
        Map-Reduce Framework
                Map input records=25
                Map output records=25
                Input split bytes=408
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=455
                CPU time spent (ms)=6740
                Physical memory (bytes) snapshot=701685760
                Virtual memory (bytes) snapshot=10003836928
                Total committed heap usage (bytes)=465567744
                Peak Map Physical memory (bytes)=238911488
                Peak Map Virtual memory (bytes)=3337875456
        File Input Format Counters
                Bytes Read=4901
        File Output Format Counters
                Bytes Written=1488
        DistCp Counters
                Bandwidth in Btyes=0
                Bytes Skipped=9365494
                Files Skipped=25

yarn application -list   可以看到任务进度

数据同步结果，

失误 有同步一遍，开发在跑数据