####场景
从hadoop2.6传数据到hadoop2.7,遇到了下面问题。
####报错信息
Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwel
l/part-00000.bz2 to hdfs://c9/tmp/hive_exportdir/data/201407/20140710/signal-dwell/part-00000.bz2
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwell/part-00000.
bz2 and hdfs://c9/tmp/hive_exportdir/.distcp.tmp.attempt_1492021243934_11314_m_000019_0.
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
####两种方法
1. 修改hdfs-site.xml
dfs.checksum.type = CRC32
2.加 -skipcrccheck参数
hadoop distcp -update -skipcrccheck hftp://X.X.X.X:50070/user/dd_edw/warehouse/X/dt=2014-04-10 /user/impala/warehouse/1
3.加上-Ddfs.checksum.type=CRC32参数,保持两边一致