测试下载一个大文件是否会出现Task attempt_201010291725_0041_m_000000_1 failed to report status for 602 seconds. Killing!
1、找一个比较大的文件:http://down1.520sys.cn/down/tomato/FQ_GhostXPsp3_1.5_%B4%BF%BE%BB%B0%E6.iso ,这个文件大约 600M.
2、做一个简单的MapReduce程序,实现下载-上述链接。
a: 创建一个文件,此文件就存放所要下载的URL.并且上传到HDFS.
b: 创建一个DownloadMapper类,该类extends Mapper,在这个类中实现下载URL,把所下载的内容直接写入到HDFS中。
c:在主函数DownloadMain中启动downloadJob.
3、启动download.job
root@se-test1-desktop:~/hadoop/program# tail -f dowload.log
10/11/09 12:58:55 INFO main.DownloadMain: downloadMain start .....
10/11/09 12:58:55 INFO main.DownloadMain: /tmp/ddow copy successfully
10/11/09 12:58:55 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/11/09 12:59:04 INFO input.FileInputFormat: Total input paths to process : 1
10/11/09 12:59:10 INFO mapred.JobClient: Running job: job_201010291725_0041
10/11/09 12:59:11 INFO mapred.JobClient: map 0% reduce 0%
10/11/09 12:59:29 INFO mapred.JobClient: map 100% reduce 0%
10/11/09 13:09:33 INFO mapred.JobClient: map 0% reduce 0%
10/11/09 13:09:35 INFO mapred.JobClient: Task Id : attempt_201010291725_0041_m_000000_0, Status : FAILED
Task attempt_201010291725_0041_m_000000_0 failed to report status for 602 seconds. Killing!
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0020320 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0017580 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0003120 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0001860 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0004730 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0015730 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0002630 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0004870 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005290 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0008870 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005540 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005600 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0009300 seconds
4、从日志中可以看出,当下载一个比较大的文件时,需要一定的时间,当这个时间超过配置文件所配置的(mapred.task.timeout--默认600)时间时,就会出现”failed to report status for 602 seconds. Killing!“。
总结 :经过上述的一个测试我们看出:
a:在我们自己的MapReduce程序当中所做的工作尽量不要超过默认配置的时间。
1、找一个比较大的文件:http://down1.520sys.cn/down/tomato/FQ_GhostXPsp3_1.5_%B4%BF%BE%BB%B0%E6.iso ,这个文件大约 600M.
2、做一个简单的MapReduce程序,实现下载-上述链接。
a: 创建一个文件,此文件就存放所要下载的URL.并且上传到HDFS.
b: 创建一个DownloadMapper类,该类extends Mapper,在这个类中实现下载URL,把所下载的内容直接写入到HDFS中。
c:在主函数DownloadMain中启动downloadJob.
3、启动download.job
root@se-test1-desktop:~/hadoop/program# tail -f dowload.log
10/11/09 12:58:55 INFO main.DownloadMain: downloadMain start .....
10/11/09 12:58:55 INFO main.DownloadMain: /tmp/ddow copy successfully
10/11/09 12:58:55 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/11/09 12:59:04 INFO input.FileInputFormat: Total input paths to process : 1
10/11/09 12:59:10 INFO mapred.JobClient: Running job: job_201010291725_0041
10/11/09 12:59:11 INFO mapred.JobClient: map 0% reduce 0%
10/11/09 12:59:29 INFO mapred.JobClient: map 100% reduce 0%
10/11/09 13:09:33 INFO mapred.JobClient: map 0% reduce 0%
10/11/09 13:09:35 INFO mapred.JobClient: Task Id : attempt_201010291725_0041_m_000000_0, Status : FAILED
Task attempt_201010291725_0041_m_000000_0 failed to report status for 602 seconds. Killing!
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0020320 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0017580 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0003120 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0001860 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0004730 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0015730 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0002630 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0004870 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005290 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0008870 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005540 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0005600 seconds
attempt_201010291725_0041_m_000000_0: Total time for which application threads were stopped: 0.0009300 seconds
4、从日志中可以看出,当下载一个比较大的文件时,需要一定的时间,当这个时间超过配置文件所配置的(mapred.task.timeout--默认600)时间时,就会出现”failed to report status for 602 seconds. Killing!“。
总结 :经过上述的一个测试我们看出:
a:在我们自己的MapReduce程序当中所做的工作尽量不要超过默认配置的时间。