solve hadoop streaming memory exceed

最新推荐文章于 2019-11-08 22:40:50 发布

robert_zhai

最新推荐文章于 2019-11-08 22:40:50 发布

阅读量651

点赞数

分类专栏： DM(data mining) php

本文链接：https://blog.csdn.net/robert198837/article/details/17452931

版权

php 同时被 2 个专栏收录

49 篇文章 0 订阅

订阅专栏

DM(data mining)

5 篇文章 0 订阅

订阅专栏

出现内存不够，指定mapper和reducer tasks数大点，就能避免这个问题，如指定

-jobconf mapred.map.tasks=500 -jobconf mapred.reduce.tasks=500

#!/bin/bash
/usr/bin/hadoop fs -rmr /user/hmthdfs/upload/out
/usr/bin/hadoop fs -lsr /user/hmthdfs/upload
/usr/bin/hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u4.jar  -input /user/hmthdfs/upload/allinone  -output /user/hmthdfs/upload/out  -file /search/hadoop/upload/mapper.php -file /search/hadoop/upload/reducer.php  -mapper mapper.php -reducer reducer.php -jobconf mapred.map.tasks=500 -jobconf mapred.reduce.tasks=500 
/usr/bin/hadoop fs -lsr /user/hmthdfs/upload/out