跑etl任务,报错:
Container [pid=31306,containerID=container_1479290736535_0004_01_000003] is running beyond physical memory limits. Current usage: 2.7 GB of 2.5 GB physical memory used; 4.4 GB of 7.5 GB virtual memory used. Killing container.
发现是一个小时任务打开了太多文件(大约7000多个,每个进程占用的buffer太多,导致内存不足)
解决办法:
修改程序设置,将每个buffer设置占用更小内存,以下是修改后成功运行mr程序的配置
"orc.strip.size":1024*1024,
"orc.block.size":16*1024*1024,
"orc.row.index.stride":500,
"orc.compress.size":8*1024,
转载于:https://blog.51cto.com/daixuan/1873967