一、zookeeper监控方式
根据官网介绍主要有两种监控方式,
1:使用ssh建立远程连接,然后使用"echo stat | nc ip port"命令返回结果。开源项目代表taokeeper。
2:使用JMX监控,文档[http://zookeeper.apache.org/doc/r3.4.6/zookeeperJMX.html]
我们考虑使用JMX监控是因为运维不希望在服务器之间配置ssh连接,或者openSSH,服务器太多难以管理用户密码和公钥,容易混乱。
二、zookeeper服务开启JMX监控
修改zookeeper的启动脚本vim zkServer.sh。找到启动参数ZOOMAIN,修改为下面值。
其中local.only=false,设为false才能在远程建立连接。
- ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false
- -Djava.rmi.server.hostname=127.0.0.1
- -Dcom.sun.management.jmxremote.port=9991
- -Dcom.sun.management.jmxremote.ssl=true
- -Dcom.sun.management.jmxremote.authenticate=true
- -Dcom.sun.management.jmxremote.access.file=/data/zookeeper/conf/jmxremote.access
- -Dcom.sun.management.jmxremote.password.file=/data/zookeeper/conf/jmxremote.password
- -Dzookeeper.jmx.log4j.disable=true
- org.apache.zookeeper.server.quorum.QuorumPeerMain"
在/data/zookeeper/conf目录下建立2个访问授权文件, 修改文件权限chmod 600 jmxremote.*
- -rw------- 1 deploy deploy 149 Aug 6 13:44 jmxremote.access
- -rw------- 1 deploy deploy 40 Aug 6 13:46 jmxremote.password
- [deploy@liutp conf]$ pwd
- /data/zookeeper/conf
- [deploy@liutp conf]$ cat jmxremote.access
- monitorRole readonly
- controlRole readwrite \
- create javax.management.monitor.*,javax.management.timer.* \
- unregister
- [deploy@liutp conf]$ cat jmxremote.password
- monitorRole 1234567
- controlRole 1234567
- [deploy@liutp conf]$
关于权限文件内容的设置可以参考jdk里的文件:C:\Java\jdk1.7.0_79\jre\lib\management\jmxremote.password.template
重启zk服务 :zkServer.sh restart /data/zookeeper/conf/zk1.cfg
三、使用Java自带的JConsole
在命令行输入JConsole,再回车。
在弹出的界面选择“远程进程”,输入“服务器IP:9991”(zookeeper服务器的IP和端口)
用户名:controlRole密码:1234567就能查看节点数,连接数,watch数
一、背景
上一篇通过Java自带的JConsole来获取zookeeper状态。主要有几个不方便的地方,zk集群一般会部署3或者5台,在多个JConsole窗口中切换比较麻烦,各个zk服务及历史数据之间,不能直观比较。一般会做一个WEB管理页面来展示集群状态,设置报警阀值来做报警。
二、JVM平台提供Mbeans
在Java5.0以上版本,有一组API可以让Java应用程序和允许的工具监视和管理Java虚拟机(JVM)和虚拟机所在的本机操作系统。该组API在 java.lang.management包。可以通过这些API可以监控local端JVM,同时也可以监控远端JVM。
通过静态工厂方法获取MXBean实例,从本地访问正在运行的虚拟机的MXBean接口。这些Bean我们从ManagementFactory类中定义的静态方法获取;如ManagementFactory.getOperatingSystemMXBean();其中不足就是只能获取本地的JVM状态。无法获取远程的虚拟机数据。
- ClassLoadingMXBean Java虚拟机的类加载系统
- CompilationMXBean Java虚拟机的编译系统
- MemoryMXBean Java虚拟机的内存系统
- RuntimeMXBean Java虚拟机的运行时系统
- OperatingSystemMXBean Java虚拟机在其上运行的操作系统
- GarbageCollectorMXBean Java虚拟机中的垃圾回收器
- MemoryManagerMXBean Java虚拟机中的内存管理器
- MemoryPoolMXBean Java虚拟机中的内存池
三、Zookeeper提供出来的Mbeans
构造MXBean代理实例,通过代理将方法调用转发到给定的MBeanServe。JConsole能够监控的项目,通过API都能获取到。
具体代码如下:
-
- import java.io.IOException;
- import java.lang.management.ClassLoadingMXBean;
- import java.lang.management.CompilationMXBean;
- import java.lang.management.ManagementFactory;
- import java.lang.management.OperatingSystemMXBean;
- import java.lang.management.ThreadMXBean;
- import java.util.ArrayList;
- import java.util.Arrays;
- import java.util.Collections;
- import java.util.HashMap;
- import java.util.Iterator;
- import java.util.List;
- import java.util.Map;
- import java.util.Set;
- import javax.management.InstanceNotFoundException;
- import javax.management.IntrospectionException;
- import javax.management.JMX;
- import javax.management.MBeanServerConnection;
- import javax.management.MalformedObjectNameException;
- import javax.management.ObjectName;
- import javax.management.ReflectionException;
- import javax.management.remote.JMXConnector;
- import javax.management.remote.JMXConnectorFactory;
- import javax.management.remote.JMXServiceURL;
- import org.apache.zookeeper.server.ConnectionMXBean;
- import org.apache.zookeeper.server.DataTreeMXBean;
- import org.apache.zookeeper.server.ZooKeeperServerMXBean;
- public class ZkJMXTest {
- static JMXConnector connector;
- /**
- * @param args
- * @throws IOException
- * @throws MalformedObjectNameException
- * @throws InstanceNotFoundException
- * @throws ReflectionException
- * @throws IntrospectionException
- */
- public static void main(String[] args) throws IOException, MalformedObjectNameException,
- InstanceNotFoundException, IntrospectionException, ReflectionException {
- OperatingSystemMXBean osbean = ManagementFactory.getOperatingSystemMXBean();
- System.out.println("体系结构:" + osbean.getArch());//操作系统体系结构
- System.out.println("处理器核数:" + osbean.getAvailableProcessors());///核数
- System.out.println("名字:" + osbean.getName());//名字
- System.out.println(osbean.getVersion());//操作系统版本
- ThreadMXBean threadBean=ManagementFactory.getThreadMXBean();
- System.out.println("活动线程:" + threadBean.getThreadCount());//总线程数
- ClassLoadingMXBean classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
- CompilationMXBean compilationMXBean = ManagementFactory.getCompilationMXBean();
- System.out.println("===========");
- // 通过 MBeanServer间接地访问 MXBean 接口
- MBeanServerConnection mbsc = createMBeanServer("192.168.1.100", "9991", "controlRole", "123456");
- // 操作系统
- ObjectName os = new ObjectName("java.lang:type=OperatingSystem");
- System.out.println("体系结构:" + getAttribute(mbsc, os, "Arch"));//体系结构
- System.out.println("处理器核数:" + getAttribute(mbsc, os, "AvailableProcessors"));//核数
- System.out.println("总物理内存:" + getAttribute(mbsc, os, "TotalPhysicalMemorySize"));//总物理内存
- System.out.println("空闲物理内存:" + getAttribute(mbsc, os, "FreePhysicalMemorySize"));//空闲物理内存
- System.out.println("总交换空间:" + getAttribute(mbsc, os, "TotalSwapSpaceSize"));//总交换空间
- System.out.println("空闲交换空间:" + getAttribute(mbsc, os, "FreeSwapSpaceSize"));//空闲交换空间
- System.out.println("操作系统:" + getAttribute(mbsc, os, "Name")+ getAttribute(mbsc, os, "Version"));//操作系统
- System.out.println("提交的虚拟内存:" + getAttribute(mbsc, os, "CommittedVirtualMemorySize"));//提交的虚拟内存
- System.out.println("系统cpu使用率:" + getAttribute(mbsc, os, "SystemCpuLoad"));//系统cpu使用率
- System.out.println("进程cpu使用率:" + getAttribute(mbsc, os, "ProcessCpuLoad"));//进程cpu使用率
- System.out.println("============");//
- // 线程
- ObjectName Threading = new ObjectName("java.lang:type=Threading");
- System.out.println("活动线程:" + getAttribute(mbsc, Threading, "ThreadCount"));// 活动线程
- System.out.println("守护程序线程:" + getAttribute(mbsc, Threading, "DaemonThreadCount"));// 守护程序线程
- System.out.println("峰值:" + getAttribute(mbsc, Threading, "PeakThreadCount"));// 峰值
- System.out.println("启动的线程总数:" + getAttribute(mbsc, Threading, "TotalStartedThreadCount"));// 启动的线程总数
- ThreadMXBean threadBean2 = ManagementFactory.newPlatformMXBeanProxy
- (mbsc, ManagementFactory.THREAD_MXBEAN_NAME, ThreadMXBean.class);
- System.out.println("活动线程:" + threadBean2.getThreadCount());// 活动线程
- ThreadMXBean threadBean3 = ManagementFactory.getThreadMXBean();
- System.out.println("本地活动线程:" + threadBean3.getThreadCount());// 本地活动线程
- System.out.println("============");//
- ObjectName Compilation = new ObjectName("java.lang:type=Compilation");
- System.out.println("总编译时间 毫秒:" + getAttribute(mbsc, Compilation, "TotalCompilationTime"));// 总编译时间 毫秒
- System.out.println("============");//
- ObjectName ClassLoading = new ObjectName("java.lang:type=ClassLoading");
- System.out.println("已加载类总数:" + getAttribute(mbsc, ClassLoading, "TotalLoadedClassCount"));// 已加载类总数
- System.out.println("已加装当前类:" + getAttribute(mbsc, ClassLoading, "LoadedClassCount"));// 已加装当前类
- System.out.println("已卸载类总数:" + getAttribute(mbsc, ClassLoading, "UnloadedClassCount"));// 已卸载类总数
- System.out.println("==========================================================");//
- // http://zookeeper.apache.org/doc/r3.4.6/zookeeperJMX.html
- // org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower
- ObjectName replica = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1");
- System.out.println("replica.1运行状态:" + getAttribute(mbsc, replica, "State"));// 运行状态
- mbsc = createMBeanServer("192.168.1.100", "9992", "controlRole", "123456");
- System.out.println("==============节点树对象===========");
- ObjectName dataTreePattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=InMemoryDataTree");
- Set<ObjectName> dataTreeSets = mbsc.queryNames(dataTreePattern, null);
- Iterator<ObjectName> dataTreeIterator = dataTreeSets.iterator();
- // 只有一个
- while (dataTreeIterator.hasNext()) {
- ObjectName dataTreeObjectName = dataTreeIterator.next();
- DataTreeMXBean dataTree = JMX.newMBeanProxy(mbsc, dataTreeObjectName, DataTreeMXBean.class);
- System.out.println("节点总数:" + dataTree.getNodeCount());// 节点总数
- System.out.println("Watch总数:" + dataTree.getWatchCount());// Watch总数
- System.out.println("临时节点总数:" + dataTree.countEphemerals());// Watch总数
- System.out.println("节点名及字符总数:" + dataTree.approximateDataSize());// 节点全路径和值的总字符数
- Map<String, String> dataTreeMap = dataTreeObjectName.getKeyPropertyList();
- String replicaId = dataTreeMap.get("name1").replace("replica.", "");
- String role = dataTreeMap.get("name2");// Follower,Leader,Observer,Standalone
- String canonicalName = dataTreeObjectName.getCanonicalName();
- int roleEndIndex = canonicalName.indexOf(",name3");
- ObjectName roleObjectName = new ObjectName(canonicalName.substring(0, roleEndIndex));
- System.out.println("==============zk服务状态===========");
- ZooKeeperServerMXBean ZooKeeperServer = JMX.newMBeanProxy(mbsc, roleObjectName, ZooKeeperServerMXBean.class);
- System.out.println(role + " 的IP和端口:" + ZooKeeperServer.getClientPort());// IP和端口
- System.out.println(role + " 活着的连接数:" + ZooKeeperServer.getNumAliveConnections());// 连接数
- System.out.println(role + " 未完成请求数:" + ZooKeeperServer.getOutstandingRequests());// 未完成的请求数
- System.out.println(role + " 接收的包:" + ZooKeeperServer.getPacketsReceived());// 收到的包
- System.out.println(role + " 发送的包:" + ZooKeeperServer.getPacketsSent());// 发送的包
- System.out.println(role + " 平均延迟(毫秒):" + ZooKeeperServer.getAvgRequestLatency());
- System.out.println(role + " 最大延迟(毫秒):" + ZooKeeperServer.getMaxRequestLatency());
- System.out.println(role + " 每个客户端IP允许的最大连接数:" + ZooKeeperServer.getMaxClientCnxnsPerHost());
- System.out.println(role + " 最大Session超时(毫秒):" + ZooKeeperServer.getMaxSessionTimeout());
- System.out.println(role + " 心跳时间(毫秒):" + ZooKeeperServer.getTickTime());
- System.out.println(role + " 版本:" + ZooKeeperServer.getVersion());// 版本
- // 三个重置操作
- // ZooKeeperServer.resetLatency(); //重置min/avg/max latency statistics
- // ZooKeeperServer.resetMaxLatency(); //重置最大延迟统计
- // ZooKeeperServer.resetStatistics(); // 重置包和延迟所有统计
- System.out.println("==============所有客户端的连接信息===========");
- ObjectName connectionPattern = new ObjectName("org.apache.ZooKeeperService:name0=ReplicatedServer_id?,name1=replica.?,name2=*,name3=Connections,*");
- Set<ObjectName> connectionSets = mbsc.queryNames(connectionPattern, null);
- List<ObjectName> connectionList = new ArrayList<ObjectName>(connectionSets.size());
- connectionList.addAll(connectionSets);
- Collections.sort(connectionList);
- for (ObjectName connectionON : connectionList) {
- System.out.println("=========================");
- ConnectionMXBean connectionBean = JMX.newMBeanProxy(mbsc, connectionON, ConnectionMXBean.class);
- System.out.println(" IP+Port:" + connectionBean.getSourceIP());//
- System.out.println(" SessionId:" + connectionBean.getSessionId());//
- System.out.println(" PacketsReceived:" + connectionBean.getPacketsReceived());// 收到的包
- System.out.println(" PacketsSent:" + connectionBean.getPacketsSent());// 发送的包
- System.out.println(" MinLatency:" + connectionBean.getMinLatency());//
- System.out.println(" AvgLatency:" + connectionBean.getAvgLatency());//
- System.out.println(" MaxLatency:" + connectionBean.getMaxLatency());//
- System.out.println(" StartedTime:" + connectionBean.getStartedTime());//
- System.out.println(" EphemeralNodes:" + connectionBean.getEphemeralNodes().length);//
- System.out.println(" EphemeralNodes:" + Arrays.asList(connectionBean.getEphemeralNodes()));//
- System.out.println(" OutstandingRequests:" + connectionBean.getOutstandingRequests());//
- //connectionBean.resetCounters();
- //connectionBean.terminateConnection();
- //connectionBean.terminateSession();
- }
- }
- // close connection
- if (connector != null) {
- connector.close();
- }
- }
- /**
- * 建立连接
- *
- * @param ip
- * @param jmxport
- * @return
- */
- public static MBeanServerConnection createMBeanServer(String ip,
- String jmxport, String userName, String password) {
- try {
- String jmxURL = "service:jmx:rmi:///jndi/rmi://" + ip + ":"
- + jmxport + "/jmxrmi";
- // jmxurl
- JMXServiceURL serviceURL = new JMXServiceURL(jmxURL);
- Map<String, String[]> map = new HashMap<String, String[]>();
- String[] credentials = new String[] { userName, password };
- map.put("jmx.remote.credentials", credentials);
- connector = JMXConnectorFactory.connect(serviceURL, map);
- MBeanServerConnection mbsc = connector.getMBeanServerConnection();
- return mbsc;
- } catch (IOException ioe) {
- ioe.printStackTrace();
- System.err.println(ip + ":" + jmxport + " 连接建立失败");
- }
- return null;
- }
- /**
- * 使用MBeanServer获取对象名为[objName]的MBean的[objAttr]属性值
- * <p>
- * 静态代码: return MBeanServer.getAttribute(ObjectName name, String attribute)
- *
- * @param mbeanServer
- * - MBeanServer实例
- * @param objName
- * - MBean的对象名
- * @param objAttr
- * - MBean的某个属性名
- * @return 属性值
- */
- private static String getAttribute(MBeanServerConnection mbeanServer,
- ObjectName objName, String objAttr) {
- if (mbeanServer == null || objName == null || objAttr == null)
- throw new IllegalArgumentException();
- try {
- return String.valueOf(mbeanServer.getAttribute(objName, objAttr));
- } catch (Exception e) {
- return null;
- }
- }
- }