MapReduce的疑难杂症

1.Java.net.SocketTimeoutException: 60000millis timeout while waiting for channel to be ready for read. ch

若果排除代码纰漏，这就是电脑配置引起的问题，因为切片分的太多所以导致运行超时。应该用ConbineTextInputFormat规定每个任务读取文件的最低大小，从而减少切分数

在驱动类中添加

job.setInputFormatClass(CombineTextInputFormat.class);
CombineTextInputFormat.setMaxInputSplitSize(job, 4194304);
CombineTextInputFormat.setMinInputSplitSize(job, 2000000);

2.

Error: java.lang.RuntimeException: java.io.EOFException
        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:165)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1268)
        at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:35)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:87)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1600)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at cn.bigdata.mapreduce.climate.ClimateSecondarySort$DateTemperaturePair.readFields(ClimateSecondarySort.java:54)
        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)
        ... 14 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

此错误是我在写二次排序程序的时候出现的，问题是在于我的WritableComparable实现类中，write方法中的wirte与readFields方法中的read不一致造成的，如：

@Override
public void readFields(DataInput in) throws IOException {
yearMonth = in.readLine();
temperature = in.readInt();
}


@Override
public void write(DataOutput out) throws IOException {
out.writeChars(yearMonth);
out.writeInt(temperature);
}

yearMonth类型是字符串，这样写的话就会报上面的错误，必须将yearMonth类型改为其它并且保证两个方法中读写类型一致才可以

3.在eclipse远程调试mr程序时出的错：

Java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:476)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:530)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at com.cloud.bigdata.ad.background.TaskExecutor.run(TaskExecutor.java:135)
at com.cloud.bigdata.ad.test.background.executors.isp.HostPVByISPHostTaskExecutorTest.test(HostPVByISPHostTaskExecutorTest.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at org.springframework.test.ConditionalTestCase.runBare(ConditionalTestCase.java:79)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

其最主要的问题四hadoop对应版本的Windows编译jar包没有到位，去下载后还要将hadoop.dll文件放入C:/windows/System32下

hadoop-2.7.3 windows版编译jar包下载地址http://download.csdn.net/download/dongtest/9701830