VisualVM: Monitoring Remote JVM Over SSH

VisualVM is a great tool for monitoring JVM (5.0+) regarding memory usage, threads, GC, MBeans etc. Let’s see how to use it over SSH to monitor (or even profile, using its sampler) a remote JVM either with JMX or without it.

This post is based on Sun JVM 1.6 running on Ubuntu 10 and VisualVM 1.3.3.

1. Communication: JStatD vs. JMX

There are two modes of communication between VisualVM and the JVM: Either over the Java Management Extensions (JMX) protocol or over jstatd.

jstatd

jstatd is a daemon that is distributed with JDK. You start it from the command line (it’s likely necessary to run it as the user running the target JVM or as root) on the target machine and VisualVM will contact it to fetch information about the remote JVMs.

• Advantages: Can connect to a running JVM, no need to start it with special parameters
• Disadvantages: Much more limited monitoring capabilities (f.ex. no CPU usage monitoring, not possible to run the Sampler and/or take thread dumps).

Ex.:

bash> cat jstatd.all.policy
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
bash> sudo /path/to/JDK/bin/jstatd -J-Djava.security.policy=jstatd.all.policy
# You can specify port with -p number and get more info with -J-Djava.rmi.server.logCalls=true

Note: Replace “${java.home}/../lib/tools.jar” with the absolute “/path/to/jdk/lib/tools.jar” if you have only copied but not installed the JDK.

If you get the failure

Could not create remote object
access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write)
java.security.AccessControlException: access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)

then jstatd likely hasn’t been started with the right java.security.policy file (try to provide fully qualified path to it).

More info about VisualVM and jstatd from Oracle.

JMX

• Advantages: Using JMX will give you the full power of VisualVM.
• Disadvantages: Need to start the JVM with some system properties.

You will generally want to use something like the following properties when starting the target JVM (though you could also enable SSL and/or require username and password):

yourJavaCommand... -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1098

See Remote JMX Connections.

2. Security: SSH

The easiest way to connect to the remote JMX or jstatd over ssh is to use a SOCKS proxy, which standard ssh clients can set up.

2.1 Set Up the SSH Tunnel With SOCKS

ssh -v -D 9696 my_server.example.com

2.2 Configure VisualVM to Use the Proxy

Tools->Options->Network – Manual Proxy Settings – check it and configure SOCKS Proxy at localhost and port 9696

2.3 Connect VisualVM to the Target

File -> Add Remote Host… – type the IP or hostname of the remote machine

JStatD Connection

You should see logs both in the ssh window (thanks to its “-v”, f.ex. “debug1: Connection to port 9696 forwarding to socks port 0 requested.” and “debug1: channel 3: free: direct-tcpip: listening port 9696 for 10.2.47.71 port 1099, connect from 127.0.0.1 port 61262, nchannels 6“) and in the console where you started jstatd (many, f.ex. “FINER: RMI TCP Connection(23)-10.2.47.71: …“)

Wait few minutes after having added the remote host, you should then see the JVMs running there.

Available stats: JVM arguments, Monitor: Heap, classes, threads monitoring (but not CPU). Sampler and MBeans require JMX.

JMX

Right-click on the remote host you have added and select Add JMX Connection …, type the JMX port you have chosen.

You should see similar logs as with jstatd.

Available stats: Also CPU usage, system properties, detailed Threads report with access to stack traces, CPU sampling (memory sampling not supported).

Note: Sampler vs. Profiler

The VisualVM’s Sampler excludes time spent in Object.wait and Thread.sleep (f.ex. waiting on I/O). Use the NetBeans Profiler to profile or sample a remote application if you want to have more control or want the possibility to include Object.wait and Thread.sleep time. It requires its Remote Pack (a java agent, i.e. a JAR file) to be in the target JVM (NetBeans’ Attach Wizard can generate the remote pack for you in step 4, Manual integration, and show you the options to pass to the target JVM to use it).

You can run the profiler over SSH by forwarding its default port (5140) and attaching to the forwarded port at localhost.

(NetBeans version 7.1.1.)