java hadoop 超时_hadoop map端的超时参数

Report progress

If your task reports no progress for 10 minutes (see the mapred.task.timeout property) then it will be killed by Hadoop. Most tasks don’t encounter this situation since they report progress implicitly by reading input and writing output. However, some jobs which don’t process records in this way may fall foul of this behavior and have their tasks killed. Simulations are a good example, since they do a lot of CPU-intensive processing in each map and typically only write the result at the end of the computation. They should be written in such a way as to report progress on a regular basis (more frequently than every 10 minutes). This may be achieved in a number of ways:

Call setStatus() on Reporter to set a human-readable description of

the task’s progress

Call incrCounter() on Reporter to increment a user counter

Call progress() on Reporter to tell Hadoop that your task is still there (and making progress)

但是，事情还没完，集群中会不定时地有任务卡死在某个点上导致任务无法继续下去：

"main" prio=10 tid=0x000000000293f000 nid=0x1e06 runnable [0x0000000041b20000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

- locked <0x00000006e243c3f0> (a sun.nio.ch.Util