黑猴子的家：Hadoop 集群间数据拷贝

集群间数据拷贝,可以使用scp rsync distcp等等 方法，再次我只介绍一下distcp，scp和rsync在linux章节已有介绍，就不多说了

1、网址

http://hadoop.apache.org/docs/r2.8.2/hadoop-distcp/DistCp.html

2、概述

DistCp版本2（分布式副本）是用于大型集群间/集群内复制的工具。它使用MapReduce来实现分布，错误处理和恢复以及报告。它将文件和目录列表扩展为映射任务的输入，每个任务都将复制源列表中指定文件的一个分区。

以前的DistCp的实现在它的使用以及它的可扩展性和性能方面都有它的一些怪癖和缺点。DistCp重构的目的是解决这些缺点，使其能够以编程方式使用和扩展。引入了新的范例来提高运行时间和设置性能，同时保留默认的传统行为。

本文档旨在描述新的DistCp的设计，其崭新功能，最佳使用以及与传统实施的偏差。

3、帮助命令

[victor@node1 hadoop]$ hadoop distcp
usage: distcp OPTIONS [source_path...] <target_path>
 OPTIONS
 -append Reuse existing data in target files and
 append new data to them if possible
 -async Should distcp execution be blocking
 -atomic Commit all changes or none
 -bandwidth <arg> Specify bandwidth per map in MB
 -delete Delete from target, files missing in source
 -diff <arg> Use snapshot diff report to identify the
 difference between source and target
 -f <arg> List of files that need to be copied
 -filelimit <arg> (Deprecated!) Limit number of files copied
 to <= n
 -filters <arg> The path to a file containing a list of
 strings for paths to be excluded from the
 copy.
 -i Ignore failures during copy
 -log <arg> Folder on DFS where distcp execution logs
 are saved
 -m <arg>      Max number of concurrent maps to use for
 copy
 -mapredSslConf <arg> Configuration for ssl config file, to use
 with hftps://. Must be in the classpath.
 -numListstatusThreads <arg>   Number of threads to use for building file
 listing (max 40).
 -overwrite Choose to overwrite target files
 unconditionally, even if they exist.
 -p <arg>  preserve status (rbugpcaxt)(replication,
 block-size, user, group, permission,
 checksum-type, ACL, XATTR, timestamps). If
 -p is specified with no <arg>, then
 preserves replication, block size, user,
 group, permission, checksum type and
 timestamps. raw.* xattrs are preserved when
 both the source and destination paths are
 in the /.reserved/raw hierarchy (HDFS
 only). raw.* xattrpreservation is
 independent of the -p flag. Refer to the
 DistCp documentation for more details.
 -sizelimit <arg> (Deprecated!) Limit number of files copied
 to <= n bytes
 -skipcrccheck Whether to skip CRC checks between source
 and target paths.
 -strategy <arg> Copy strategy to use. Default is dividing
 work based on file sizes
 -tmp <arg> Intermediate work path to be used for
 atomic commit
 -update Update target, copying only missingfiles or
 directories

4、使用方法

[victor@node1 hadoop]$ bin/hadoop distcp hdfs://nn1:9000/foo/bar hdfs://nn6:9000/bar/foo

把nn1集群的/foo/bar目录下的所有文件或目录名展开并存储到nn6集群中，这些文件内容的拷贝工作被分配给多个map任务， 然后每个TaskTracker分别执行从nn1到nn2的拷贝操作，注意DistCp使用绝对路径进行操作。

5、命令行中可以指定多个源目录

[victor@node1 hadoop]$ bin/hadoop distcp \
hdfs://nn1:9000/foo/a \
hdfs://nn1:9000/foo/b \
hdfs://nn6:9000/bar/foo

6、使用-f选项，从文件里获得多个源

[victor@node1 hadoop]$ hadoop distcp -f \
hdfs://nn1:9000/srclist \
hdfs://nn6:9000/bar/foo 

//尖叫提示：srclist的内容是 hdfs://nn1:9000/foo/a 和hdfs://nn1:9000/foo/b