CDH 集群日常管理&调优

调优无非从如上几个方面

• 硬件
• Linux
• JVM
• Hadoop

1 调优

1.1 Linux调优

调整Linux的最大文件打开数和最大进程数

vi /etc/security/limits.conf

	* soft nofile 65535
	* hard nofile 65535
	* soft nproc 65535
	* hard nproc 65535
	

网络参数

more /etc/sysctl.conf | grep net.core.somanconn\

[root@localhost security]# sysctl -w net.core.somaxconn=32768
net.core.somaxconn = 32768


[root@localhost security]# echo net.core.somaxconn=32768 >> /etc/sysctl.conf
[root@localhost security]# more /etc/sysctl.conf | grep net.core.somaxconn

网络参数txqueuelen

ifconfig
ens5f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.2.0.6  netmask 255.255.255.0  broadcast 10.2.0.255
        inet6 fe80::bb3:69b7:6a55:e0b6  prefixlen 64  scopeid 0x20<link>
        ether a0:36:9f:fa:4b:04  txqueuelen 1000  (Ethernet)
        RX packets 11132379  bytes 2037240247 (1.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17525655  bytes 4102894410 (3.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

### 发现txquenelen 值为1000 调整为4096

调整SWAP
设置SWAP为0 ，不是禁用 而是更多不用SWAP

more /etc/sysctl.conf | grep vm.swappiness
echo vm.swappiness=0 >> /etc/sysctl.conf

1.2 JVM参数的调优

Xms10240m -Xmx10240m
-XX:+UseParNewGC
-XX:+UseConcMarkSweePGC
-XX:-CmSConcurrentMTEnabled
-XX:CMSInitiatingOccupancyFration=70
-XX:+CMSParalleelRemarkEnabled

1.3 Hadoop

2 CDH 集群管理

2.1 配置参数

2.2 监控

2.3 慢磁盘 监控