虚拟机性能监控与故障处理工具

常见问题：
1、找出java进程中最耗时的线程
2、从dump文件中找出内存泄露的地方

jstatd

Java Statistics Monitoring Daemon

此命令的基本含义是远程主机信息收集
创建文件 jstatd.all.policy，jdk的目录可以通过一下命令获得

jinfo pid |grep java.home

文件内容为

grant codebase  "file:/usr/local/jdk1.8.0_131/jre/tools.jar"{
permission java.security.AllPermission;
};
// default permissions granted to all domains
grant {
// Allows any thread to stop itself using the java.lang.Thread.stop()
// method that takes no argument.
// Note that this permission is granted by default only to remain
// backwards compatible.
// It is strongly recommended that you either remove this permission
// from this policy file or further restrict it to code sources
// that you specify, because Thread.stop() is potentially unsafe.
// See the API specification of java.lang.Thread.stop() for more
        // information.
permission java.lang.RuntimePermission "stopThread";

// allows anyone to listen on un-privileged ports
permission java.net.SocketPermission "localhost:1024-", "listen";

// "standard" properies that can be read by anyone
permission java.util.PropertyPermission "java.version", "read";
permission java.util.PropertyPermission "java.vendor", "read";
permission java.util.PropertyPermission "java.vendor.url", "read";
permission java.util.PropertyPermission "java.class.version", "read";
permission java.util.PropertyPermission "os.name", "read";
permission java.util.PropertyPermission "os.version", "read";
permission java.util.PropertyPermission "os.arch", "read";
permission java.util.PropertyPermission "file.separator", "read";
permission java.util.PropertyPermission "path.separator", "read";
permission java.util.PropertyPermission "line.separator", "read";
permission java.util.PropertyPermission "java.specification.version", "read";
permission java.util.PropertyPermission "java.specification.vendor", "read";
permission java.util.PropertyPermission "java.specification.name", "read";
permission java.util.PropertyPermission "java.vm.specification.version", "read";
permission java.util.PropertyPermission "java.vm.specification.vendor", "read";
permission java.util.PropertyPermission "java.vm.specification.name", "read";
permission java.util.PropertyPermission "java.vm.version", "read";
permission java.util.PropertyPermission "java.vm.vendor", "read";
permission java.util.PropertyPermission "java.vm.name", "read";
permission java.security.AllPermission;
};

执行命令

jstatd -J-Djava.security.policy=jstatd.all.policy

打开

jvisualvm

jstack

java堆栈跟踪工具

找出最耗时的线程对应的堆栈

查找java进程中耗CPU的线程，分为以下三步：
列出进程id对应的线程id

ps -mp {pid} -o THREAD,tid,time

将cpu占用高的线程id找出并转换为16进制,或者去https://tool.lu/hexconvert进行转换

printf “%x\n” {tid}

打印出有问题的线程
jstack {pid} |grep {tid} -A 30

jinfo

查案虚拟机的信息

jps

虚拟机进程状况工具

jps [ options ] [ hostid ]

选项	作用
-q	只输出LVMID，省略主类的名称
-m	输出虚拟机进程启动时传递给主类main()函数的参数
-l
-v	输出虚拟机启动时的jvm参数

jmap

java内存映像工具

dump

导出整个JVM 中内存信息

jmap -dump:format=b,file=文件名 [pid]

只dump出存活的对象

jmap -dump:live,format=b,file=okb 83426

生成的文件，可以使用jhat进行分析，也可以使用eclipse Memory Analyzer
Eclipse 提供的一个用于分析JVM 堆Dump文件的插件。借助这个插件可查看对象的内存占用状况，引用关系，分析内存泄露等。
http://www.eclipse.org/mat/

histo

jmap -histo 23213

显示堆中对象统计信息，包括类，实例数量、合集容量

jstat

虚拟机统计信息监视工具

jstat -option pid

详细说明请查看

jhat

堆转储快找分析工具

jhat是sun 1.6及以上版本中自带的一个用于分析JVM 堆DUMP 文件的工具，基于此工具可分析JVM HEAP 中对象的内存占用情况

jhat -J-Xmx1024M [file]

执行后等待console 中输入start HTTP server on port 7000 即可使用浏览器访问 IP：7000,例如，利用上面生成的文件ok

jhat -J-Xmx1024M ok

一般这种方式，并不建议使用，而是使用jvisualvm进行分析

• jmap

打印共享对象的内存映射或者一个进程（或者文件，远程debug的服务器）的详细堆内存信息。这个命令是实验性的，不被官方支持的。

概要

jmap [ options ] pid
jmap [ options ] executable core
jmap [ options ] [ pid ] server-id@ ] remote-hostname-or-IP

参数	说明
options	命令的可选参数. See Options.
pid	需要被打印的进程ID，进程必须是java进程，如果需要列出一台机器上的java进程，可以使用 jps -l 命令
executable	堆转储的Java可执行文件。
core	需要被打印内存映射的一个核心文件
remote-hostname-or-IP	远程debug服务器的主机名称或者IP地址，查看：jsadebugd(1).
server-id	多个调试服务器在同一个远程主机上运行时使用的可选唯一标识。

Description
The jmap command prints shared object memory maps or heap memory details of a specified process, core file, or remote debug server. If the specified process is running on a 64-bit Java Virtual Machine (JVM), then you might need to specify the -J-d64 option, for example: jmap -J-d64 -heap pid.

Note: This utility is unsupported and might not be available in future releases of the JDK. On Windows Systems where the dbgeng.dll file is not present, Debugging Tools For Windows must be installed to make these tools work. The PATH environment variable should contain the location of the jvm.dll file that is used by the target process or the location from which the crash dump file was produced, for example: set PATH=%JDK_HOME%\jre\bin\client;%PATH%.

Options

如果没有option，jmap打印的是对象的映射。目标jvm中每一个共享的初始化的地址，映射的大小和共享对象文件的都会被打印。

[root@test jar]# jmap 27646
Attaching to process ID 27646, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11
0x0000000000400000	7K	/usr/local/jdk1.8.0_131/bin/java
0x00007f3eb2b5c000	86K	/usr/lib64/libgcc_s-4.8.5-20150702.so.1
0x00007f3eb2d72000	250K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libsunec.so
0x00007f3f25318000	90K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libnio.so
0x00007f3f25729000	49K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libmanagement.so
0x00007f3f25a33000	113K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libnet.so
0x00007f3f48921000	121K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libzip.so
0x00007f3f48b3c000	60K	/usr/lib64/libnss_files-2.17.so
0x00007f3f48d4f000	220K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libjava.so
0x00007f3f48f7b000	64K	/usr/local/jdk1.8.0_131/jre/lib/amd64/libverify.so
0x00007f3f49189000	42K	/usr/lib64/librt-2.17.so
0x00007f3f49391000	1114K	/usr/lib64/libm-2.17.so
0x00007f3f49693000	16597K	/usr/local/jdk1.8.0_131/jre/lib/amd64/server/libjvm.so
0x00007f3f4a686000	2067K	/usr/lib64/libc-2.17.so
0x00007f3f4aa47000	18K	/usr/lib64/libdl-2.17.so
0x00007f3f4ac4b000	100K	/usr/local/jdk1.8.0_131/lib/amd64/jli/libjli.so
0x00007f3f4ae61000	139K	/usr/lib64/libpthread-2.17.so
0x00007f3f4b07d000	151K	/usr/lib64/ld-2.17.so

-dump:[live,] format=b, file=filename
Dumps the Java heap in hprof binary format to filename. The live suboption is optional, but when specified, only the active objects in the heap are dumped. To browse the heap dump, you can use the jhat(1) command to read the generated file.

-finalizerinfo
Prints information about objects that are awaiting finalization.

-heap
Prints a heap summary of the garbage collection used, the head configuration, and generation-wise heap usage. In addition, the number and size of interned Strings are printed.

-histo[:live]
Prints a histogram of the heap. For each Java class, the number of objects, memory size in bytes, and the fully qualified class names are printed. The JVM internal class names are printed with an asterisk (*) prefix. If the live suboption is specified, then only active objects are counted.

-clstats
Prints class loader wise statistics of Java heap. For each class loader, its name, how active it is, address, parent class loader, and the number and size of classes it has loaded are printed.

-F
Force. Use this option with the jmap -dump or jmap -histo option when the pid does not respond. The live suboption is not supported in this mode.

-h
Prints a help message.

-help
Prints a help message.

-Jflag
Passes flag to the Java Virtual Machine where the jmap command is running.

java启动进程标记小技巧

为了方便精确找到控制台启动后的进程，可以带个无用的参数，这么杀的时候，可以根据参数杀掉进程，如下：

java -jar rocketmq-console-ng-1.0.0.jar -dprocess_Mark=rocketmq-console-zsj 
--server.port=12581 --rocketmq.config.namesrvAddr=192.168.211.134:9876 

-d 表示加参数,用下面的命令可以精确的找到这一条进程

ps -ef |grep java|grep "process_Mark=rocketmq-console-zsj"

CURR_SERVICE_PID=`ps -ef|grep java|grep "process_Mark=rocketmq-console-zsj"
|grep -v grep|awk '{print $2}'`

打开gc日志

jinfo -flag +PrintGCDetails 12278
jinfo -flag +PrintGC 12278

故障模拟

服务器进程奔溃
大量的请求，拖慢服务器，导致进程奔溃

解决方案，
生产者/消费者模式的消息队列