swapniess and overcommit and other hadoop optimization

限制产生的文件大小,单位KB

ulimit -f 1
dd if=/dev/zero of=1g.txt bs=1023 count=1 可以
dd if=/dev/zero of=1g.txt bs=1024 count=1 可以
dd if=/dev/zero of=1g.txt bs=1025 count=1 不可以
ulimit -f ulimit

限制cpu的使用时间

ulimit -t 1

echo ‘ls -aRl /’ > 1.sh && bash 1.sh 发现程序运行一会儿后自动被kill

firefoxbug overcommit的解释

http://www.firefoxbug.com/index.php/archives/2800/
http://blog.csdn.net/houjixin/article/details/46412557
http://linuxstorm.blog.51cto.com/5218690/1609334

cat /proc/sys/vm/overcommit_memory
echo 0 > /proc/sys/vm/overcommit_memory
cat /proc/sys/vm/overcommit_memory

http://blog.csdn.net/metaxen/article/details/7188689
http://www.51testing.com/html/87/43487-204904.html
http://www.vpsee.com/2013/10/how-to-configure-the-linux-oom-killer/

动态链接库

http://m.blog.csdn.net/blog/zhangmiaoping23/45675119 # 写的非常棒

swapniess 更新

http://www.slideshare.net/technmsg/improving-hadoop-performancevialinux?next_slideshow=1
http://www.pythian.com/blog/what-exactly-is-swappiness/
http://www.linuxvox.com/2009/10/what-is-the-linux-kernel-parameter-vm-swappiness/
http://lwn.net/Articles/100978/ 不同的内核对swapniess的解释也不一样

# 个人认为修不修改,没有啥大的区别.因为在集群中,swap并没有大量使用,或者说非常少的swap的使用.
cat /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/swappiness
echo "vm.swapniess = 10" >> /etc.sysctl.conf

THP

cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

网卡升级

巨型帧打开后,可以提升10%~20%

page cache

echo 1 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=512M-file count=512000 bs=1k
time cat 512M-file &>/dev/null
echo 1 > /proc/sys/vm/drop_caches
time cat 512M-file &>/dev/null