不同hadoop版本之间迁移数据存在问题， Check-sum

####场景

从hadoop2.6传数据到hadoop2.7，遇到了下面问题。

####报错信息

Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwel
l/part-00000.bz2 to hdfs://c9/tmp/hive_exportdir/data/201407/20140710/signal-dwell/part-00000.bz2
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwell/part-00000.
bz2 and hdfs://c9/tmp/hive_exportdir/.distcp.tmp.attempt_1492021243934_11314_m_000019_0.
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more

####两种方法

1. 修改hdfs-site.xml

dfs.checksum.type = CRC32

2.加 -skipcrccheck参数

hadoop distcp -update -skipcrccheck hftp://X.X.X.X:50070/user/dd_edw/warehouse/X/dt=2014-04-10 /user/impala/warehouse/1

3.加上-Ddfs.checksum.type=CRC32参数，保持两边一致

 hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true -Ddfs.checksum.type=CRC32 webhdfs://132.121.97.48:50070/glabel/data /tmp/hive_exportdir/ >glabel_data_distcp.log 2>&1 &