hadoop hdfs 数据迁移到其他集群

# hadoop fs -cat /srclist
Warning: $HADOOP_HOME is deprecated.

hdfs://sht-sgmhadoopcm-01:9011/jdk-6u45-linux-x64.bin
hdfs://sht-sgmhadoopcm-01:9011/upload
hdfs://sht-sgmhadoopcm-01:9011/oracle

# hadoop distcp -Ddfs.replication=2 -Ddistcp.bytes.per.map=1073741824 -Ddfs.client.socket-timeout=240000000 -Dipc.client.connect.timeout=40000000 -i -f hdfs://sht-sgmhadoopcm-01:9011/srclist hdfs://sht-sgmhadoopdn-01:9011/
Warning: $HADOOP_HOME is deprecated.

19/04/09 23:20:00 INFO tools.DistCp: srcPaths=[hdfs://sht-sgmhadoopcm-01:9011/jdk-6u45-linux-x64.bin, hdfs://sht-sgmhadoopcm-01:9011/upload, hdfs://sht-sgmhadoopcm-01:9011/oracle]
19/04/09 23:20:00 INFO tools.DistCp: destPath=hdfs://sht-sgmhadoopdn-01:9011/
19/04/09 23:20:00 INFO tools.DistCp: sourcePathsCount=9
19/04/09 23:20:00 INFO tools.DistCp: filesToCopyCount=6
19/04/09 23:20:00 INFO tools.DistCp: bytesToCopyCount=1.5g
19/04/09 23:20:06 INFO mapred.JobClient: Running job: job_201904091624_0021
19/04/09 23:20:07 INFO mapred.JobClient:  map 0% reduce 0%
19/04/09 23:20:23 INFO mapred.JobClient:  map 38% reduce 0%
19/04/09 23:20:41 INFO mapred.JobClient:  map 56% reduce 0%
19/04/09 23:20:44 INFO mapred.JobClient:  map 83% reduce 0%
19/04/09 23:21:02 INFO mapred.JobClient:  map 100% reduce 0%
19/04/09 23:21:07 INFO mapred.JobClient: Job complete: job_201904091624_0021
19/04/09 23:21:07 INFO mapred.JobClient: Counters: 22
19/04/09 23:21:07 INFO mapred.JobClient:   Job Counters 
19/04/09 23:21:07 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=56645
19/04/09 23:21:07 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
19/04/09 23:21:07 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
19/04/09 23:21:07 INFO mapred.JobClient:     Launched map tasks=1
19/04/09 23:21:07 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
19/04/09 23:21:07 INFO mapred.JobClient:   File Input Format Counters 
19/04/09 23:21:07 INFO mapred.JobClient:     Bytes Read=1224
19/04/09 23:21:07 INFO mapred.JobClient:   File Output Format Counters 
19/04/09 23:21:07 INFO mapred.JobClient:     Bytes Written=0
19/04/09 23:21:07 INFO mapred.JobClient:   FileSystemCounters
19/04/09 23:21:07 INFO mapred.JobClient:     HDFS_BYTES_READ=1613761362
19/04/09 23:21:07 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=22815
19/04/09 23:21:07 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1613759979
19/04/09 23:21:07 INFO mapred.JobClient:   distcp
19/04/09 23:21:07 INFO mapred.JobClient:     Files copied=6
19/04/09 23:21:07 INFO mapred.JobClient:     Bytes copied=1613759979
19/04/09 23:21:07 INFO mapred.JobClient:     Bytes expected=1613759979
19/04/09 23:21:07 INFO mapred.JobClient:   Map-Reduce Framework
19/04/09 23:21:07 INFO mapred.JobClient:     Map input records=7
19/04/09 23:21:07 INFO mapred.JobClient:     Physical memory (bytes) snapshot=65015808
19/04/09 23:21:07 INFO mapred.JobClient:     Spilled Records=0
19/04/09 23:21:07 INFO mapred.JobClient:     CPU time spent (ms)=33820
19/04/09 23:21:07 INFO mapred.JobClient:     Total committed heap usage (bytes)=92995584
19/04/09 23:21:07 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1536688128
19/04/09 23:21:07 INFO mapred.JobClient:     Map input bytes=1124
19/04/09 23:21:07 INFO mapred.JobClient:     Map output records=0
19/04/09 23:21:07 INFO mapred.JobClient:     SPLIT_RAW_BYTES=159

 

转载于:https://www.cnblogs.com/ilifeilong/p/10680580.html