WebLogic Server: Analyzing stuck threads

A while ago I was confronted with a WebLogic server which did not want to start. The log files did not give an indication of what was wrong. I tried the usual suspects such as clearing the tmp and removing lok files but they did solve the issue. How to proceed? In this blog article I'll provide an example of how such an issue can be debugged.

A stuck thread is a thread which takes longer than the configured stuck thread max wait time (600 seconds by default). To simulate a stuck thread issue I created a simple servlet which does a Thread.sleep upon a request. I'll show how to use various tools to detect the servlet causing the problem and even identity the specific request which causes the code to malfunction. A similar mechanism can be used to debug for example JDBC issues and can help in the identification of which connection causes an issue. The tools used do not require the server to be fully started.

Overview monitoring tools Oracle JRockit

Several monitoring options for the JRockit JVM are available. jps, jstat and jconsole are monitoring tools provided by the Oracle (previously Sun) JDK. Jps and jstat are commandline tools for obtaining information about running JVM’s. JConsole is a GUI suitable to monitor JVM’s supporting Java Management Extensions (JMX) such as JRockit. Oracle provides JVM monitoring software for JRockit (which can also be used for other JVM's); Mission Control which consists of several tools which can help in for example detection of memory leaks.

To monitor the JRockit JVM, it has to be started with the -Xmanagement option. This option is the same as using the following startup options for the Oracle (previously Sun) JVM:
 -Dcom.sun.management.jmxremote.port=7091
 -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.ssl=false

Oracle (previously Sun) JVM tooling

Commandline tools

jps

The jps utility lists the virtual machines for the current user on the target system. The utility is useful in environments where the VM is started using the JNI Invocation API rather than the standard Java launcher. In these environments, it is not always easy to recognize the Java processes in the process list. The jps tool alleviates this problem.

The following example demonstrates the use of the jps utility. Enter jps on the command line, and the utility lists the virtual machines and process IDs for which the user has access rights

$ jps
16217 MyApplication
16342 jps

jstat

The jstat utility uses the JVM's built-in instrumentation to provide information on performance and resource consumption of running applications. The tool is useful for diagnosing performance issues, particularly issues related to heap sizing and garbage collection. jstatd is a daemon which can be used to use jstat to connect to a remote machine.

JConsole

If you start an application with JRockit in -Xmanagement mode, you'll be able to connect with Sun's jconsole tool to the JVM. See also in the introduction.

Overhead on the JVM for using JConsole is small; 3% to 4 %. See; http://weblogs.java.net/blog/emcmanus/archive/2006/07/how_much_does_i.html

Local monitoring

To monitor a local application, it must be running with the same user ID as jconsole.  The command syntax to start jconsole for local monitoring is:

jconsole [processID]
where processID is the application's process ID (PID). To determine an application's PID:

On Windows systems, use Task Manager to find the PID of java or javaw.
You can also use the jps command-line utility to determine PIDs.

For example, if you determined that the process ID of the application is 2956, then you would start jconsole as follows:

jconsole 2956

Both jconsole and the application must by executed by the same user name. The management and monitoring system uses the operating system's file permissions.

If you don't specify a process ID, jconsole will automatically detect all local Java applications, and display a dialog box that lets you select the one you want to monitor.

Remote monitoring

To start jconsole for remote monitoring, use this command syntax:

jconsole [hostName:portNum]
where hostName is the name of the system running the application and portNum is the port number you specified when you enabled the JMX agent when you started the JVM.

If you do not specify a host name/port number combination, then jconsole will display a connection dialog box enabling you to enter a host name and port number.

See below, Connection options under JRockit Mission Control for more information on the type of the connection.

JRockit Mission Control

Discovery options

Local

You can connect to locally discovered JRockits if the discovered JRockit is R27.1 or later and runs Java SE 5.0 or later.

JDP

JRockit instances are discovered automatically through JDP (JRockit Discovery Protocol) if the autodiscovery parameter is set to true (-Xmanagement:autodiscovery=true). The broadcast port for JDP can be specified in the JRockit Mission Control preferences and as a parameter upon starting the JVM.

Example parameters for starting JRockit (can be added in the Weblogic script which starts the relevant domain)
java -Xmanagement:ssl=false,authenticate=false,port=7091,autodiscovery=true

Connection options

Java Management Extensions (JMX) are accessed via RMI to obtain JVM instance information.

The JMX technology provides the tools for building distributed, Web-based, modular and dynamic solutions for managing and monitoring devices, applications, and service-driven networks. By design, this standard is suitable for adapting legacy systems, implementing new management and monitoring solutions, and plugging into those of the future. Starting with the J2SE platform 5.0, JMX technology is included in the Java SE platform.

RMI (remote method invocation) can be used to connect to JMX (MXBEANS or platform MBEANS) instances. In the references section there is a link to an example on how to implement this.

JRockit Mission control toolset 

Memory leak detection

BEA JRockit 1.4.2 (R26.2), BEA JRockit 5.0 sp1 and forward fully support the stand-alone Memory Leak Detector.

Memory leak detection is a tool which analyses the runtime of a JVM. Running the tool against a JVM has a small overhead cost most noticeable during garbage collection.

Memory leak detection can be used as followed;
- the Trend analyzer can be used to determine which objects have the fastest growth in bytes/second. The Trend analyzer can also help identifying targets which occupy a lot of heap space or have large numbers of instances
- once an interesting class has been identified, the Types tab can be used to identify causality and related objects
- to obtain more information about where the problem is occurring or which specific setting might cause it, specific instances can be analyzed of selected types and an allocation stack trace can be conducted. These cause more overhead to the JVM

JRockit Runtime Analyzer

The JRockit Runtime Analyzer consists of two parts. One is running inside the JVM and recording information about the JVM and the Java application currently running. This information is saved to a file which is then opened in the other part: the analyzer tool. This is a regular Java application used to visualize the information contained in the JRA recording file.

Recordings can be created from the Management Console, by using the jrcmd command line tool or by specifying recording options at the JRockit commandline.

On Windows, the commandline for jrcmd is as followed;

bin\jrcmd.exe <pid> jrarecording time=<jrarecording time>
filename=<filename>

The JRA tool provides the most extensive information on JVM usage, garbage collection, heap usage, etc. Since it is based on recording information during a set period and analyzing the data afterwards, timing becomes important if you want to use this tool to analyze runtime problems. During recording the overhead is small.

JRockit Management Console

The Management Console system and JVM behavior. It also has the option to set notifications for example, send an e-mail when a certain heap size is reached. Also the management console allows to increase the heap size at runtime to for example postpone an out of heapspace error. Also JRA recordings can be started as a result of for example high heap space usage to determine the problem.

The Management console has to be running in order to provide notifications, but using commandline options, it can be started in headless mode. Also using commandline options it can be set to connect to a JVM.

References

Jps and jstat
http://docs.oracle.com/javase/6/docs/technotes/tools/#monitor

JConsole documentation
http://download.oracle.com/javase/1.5.0/docs/tooldocs/share/jconsole.html

Oracle JRockit R28 documentation
http://www.oracle.com/technetwork/middleware/jrockit/documentation/index.html

BEA JRockit R27 documentation
http://www.oracle.com/technetwork/middleware/weblogic/documentation/weblogic-jrockit-089130.html

JMX Technology Home Page
http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html

Using RMI to connect to a JRockit JVM instance
https://blogs.oracle.com/pacogomez/entry/monitoring_jrockit_through_a_f

How much does it cost to monitor an app with jconsole?
http://weblogs.java.net/blog/emcmanus/archive/2006/07/how_much_does_i.html

JVM does not start on Windows Server 2008

A customer is using Windows Server 2008 64 bit and the Oracle JDK version 1.6.33 64 bit. Recently,  the following error occurred when a (any) Java application was started;

Problem signature:

Problem Event Name: APPCRASH

Application Name: java.exe

Application Version: 6.0.330.5

Application Timestamp: 4fed0a6a

Fault Module Name: jvm.dll

Fault Module Version: 20.8.0.3

Fault Module Timestamp: 4fed2ec7

Exception Code: c0000094

Exception Offset: 0000000000309132

OS Version: 6.1.7601.2.1.0.400.8

Locale ID: 1043

Additional Information 1: 9930

Additional Information 2: 993087b8a26c90b698b0aaa7c0c9a119

Additional Information 3: 45d8

Additional Information 4: 45d86e82cdfe3714a97bde810a82a78e

Even just starting java.exe -version caused this exception. De-installing all present JDK's/JRE's and installing different (32 bit and 64 bit) versions, did not solve this problem. When we downgraded to a JDK 1.5 version however, the problem did not occur anymore but using a version 1.5 JDK was not a viable option.

The  Exception Code: c0000094 is an INT_DIVIDE_BY_ZERO exception, so not much help there in clarifying the problem.

What we tried

There was no traceable change leading to this error.

We tried a clean install of the entire server (Windows reinstallation) and the JVM worked. Also after completely updating Windows, it still worked. We tried this locally with Virtualbox in a Virtual Machine instead of using the image/configuration of the machine on which the problem originally occurred. Multiple environments experienced the same problem. Also based on the below, our estimate is that this error is related to local (customer specific) policies (otherwise undoubtedly Google would have more information on this problem).

We encountered several errors when using Process Monitor; http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

What we saw was that in jvm.dll, different behavior started to occur. We could however not pinpoint the cause of this change of behavior. Decompiling the dll and running it with breakpoints on the server with Visual Studio, was not an option.


When we installed JRockit JDK, the problem was solved. To make the new JDK the default JDK used, several environment settings needed updating. We also use a Tomcat service which needed to be updated to use the new JDK.

Description

Install JRockit

First download the JRockit JDK;

http://www.oracle.com/technetwork/middleware/jrockit/downloads/index.html

Start the installer and specify the installation directory

Don't install Demo's, samples and sources (unless of course you will use them)

Install the public JRE (you might need the JRE some day and not the complete JDK)

Choose a directory to install the JRE

Update environment variables

Right mouse click the Computer icon, Properties (as you can see, I need a new laptop...).

Advanced system properties

Environment variables

Make sure no JAVA_HOME is present in the User variables. If JAVA_HOME is present under user variables, remove it.

If present, update and if not present, create the JAVA_HOME environment variable under System variables. Set it to be equal to the JDK installation directory. In this case; C:\Program Files\Java\jrockit-jdk1.6.0_33-R28.2.4-4.1.0. 

 

Update the Path System variable and set as the first entry the bin directory of the JDK. In this case; C:\Program Files\Java\jrockit-jdk1.6.0_33-R28.2.4-4.1.0\bin. Start a new command prompt (Start, cmd) and give java -version to test the Path  variable and the newly installed JDK.

Update Tomcat service

We use Tomcat 6 on the Windows 2008 server. When we changed the JDK, Tomcat wouldn't start anymore when we started it from the Windows Services screen. Tomcat usually autodetects the JDK present on the system based on the JAVA_HOME, but in case of JRockit, this auto setting wouldn't work.

The JVM can be specified manually though. See; http://tomcat.apache.org/tomcat-6.0-doc/windows-service-howto.html 

In case of JRockit, the following command updated the service to use the correct JDK;

tomcat6.exe //US//Tomcat6 --Jvm="C:\Program Files\Java\jrockit-jdk1.6.0_33-R28.2.4-4.1.0\jre\bin\jrockit\jvm.dll"

Tomcat6 here is the name of the service. The path to the jvm.dll file might differ depending on the location you specified when installing JRockit

Conclusion

When mysterious JVM errors start to occur when using Windows Server 2008, consider using JRockit and make sure the JVM is known to the programs using it.