1.....今天写 Hadoop程序的时候 出现了这么一个问题
java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:165)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1268)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:35)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:88)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1600)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2019)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(Unknown Source)
at com.mjm.tq.Weather.readFields(Weather.java:55)
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)
... 16 more
检查 错误原因:
@Override
public void readFields(DataInput in) throws IOException {
this.year = in.readInt();
this.month = in.readInt();
this.day = in.readInt();
this.temperature = in.readInt();
}
@Override
public void write(DataOutput out) throws IOException {
//这边应该是writeInt()
out.write(year);
out.write(month);
out.write(day);
out.write(temperature);
}
2...z在本地从测试环境下,在windows Eclipse上运行hadoop程序时
总是报如下错误:
Diagnostics: Exception from container-launch.
Container id: container_1492698686128_0006_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2017-04-21 09:44:15,747 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0
找出 错误原因 是因为
项目中的classpath下不能有 服务器上的配置文件
至于为什么 就有点 不太懂了,可以看看网上其他的blog
参考:http://www.aboutyun.com/thread-8498-1-1.html
3.对了 我还遇到过一个问题 环境:服务器模式
就是MR程序状态一直在 JobRunnng状态 卡在这边 >=30min,后来在网上 看看有没有解决办法
在yarn-site.xml中加了:
- <property>
- <name>yarn.nodemanager.resource.memory-mb</name>
- <value>20480</value>
- </property>
- <property>
- <name>yarn.scheduler.minimum-allocation-mb</name>
- <value>2048</value>
- </property>
- <property>
- <name>yarn.nodemanager.vmem-pmem-ratio</name>
- <value>2.1</value>
- </property>
加了 以上这几句 就好了,可以正常运行
出现此故障的原因应该是,在每个Docker分配的内存和CPU资源太少,不能满足Hadoop和Hive运行所需的默认资源需求。
参考:http://blog.csdn.net/dai451954706/article/details/50464036