You don’t need to be an expert in a particular application such as WebSphere Portal or IBM Forms to do a quick analysis of memory. Here’s how.
Get a Heapdump
The easiest way to create a heapdump is using the WebSphere Console.
- Start the administrative console.
- In the navigation pane, click Troubleshooting > Java dumps and cores.
- Select the server_name for which you want to generate the heap dump.
- Click Heap dump to generate the heap dump for your specified server.
Should this approach not work. Use wsadmin to generate the heapdump.
- Start the wsadmin scripting client. You have several options to run scripting commands, ranging from running them interactively to running them in a profile.
- Invoke the generateHeapDump operation on a JVM MBean.
For example, this is my script to launch the wsadmin client on the local machine.
C:\IBM\FormsExperienceBuilder\WebSphere\AppServer\bin\wsadmin.bat -host localhost -port 8071 -user wasadmin -password password
Next, I run the following commands on the wsadmin prompt.
set jvm [$AdminControl queryNames WebSphere:type=JVM,process=TranslatorServer,node=formsNode01,*] $AdminControl invoke $jvm generateHeapDump
There are a few settings you need to know.
- The SOAP port as seen in the first command.
- The process and node in the second command.
You can get these from the WebSphere Console. See the previous screenshot. The Server and Node are listed in the table. These values correspond to the process and node values substituted in the set command. You can find the SOAP port easily by viewing Servers -> Websphere application servers -> <your desired server> -> Communications -> Ports. Look for the SOAP_CONNECTOR_ADDRESS.
The output of these approaches will tell you where the heapdump is located. It will be a file with the PHD extension.
Use the Memory Analyzer Toolkit (MAT)
I’ve used MAT for years. It’s awesome at what it does, which is much more than looking at a pretty graph. Download it here https://eclipse.org/mat/. To open the PHD file generated by WebSphere, you’ll need to add the DTFJ plugin. Find instructions here http://www.ibm.com/developerworks/java/jdk/tools/dtfj.html.
Now it’s super simple to anaylze the PHD. Use File -> Open heap dump -> <select you PHD file>. Let’s take a look at a PHD I recently analyzed.
This is a leak suspect report (the default option in the wizard after opening a heap dump). It’s basically saying that there’s a possible memory leak. Why? Because the WebSphere server’s Java heap is currently set to 1 GB. Of that, 937MB is being used by one object. So 90% of the heap is being used by one object – that seems like a leak … or maybe it isn’t.
This is a server that is simply running out of memory because the workload is too high. You can drill down into how the heap is allocated. Use the Open Dominator Tree for entire heap button.
The FormCache object is using 90% of the total heap – we can see that in the percentage column and is expected given the graph in the leak suspects report. Look at what’s in this object. It’s a collection for CachedFormInstance private objects. And there are 860 of them! These range in size, from 15MB to 4.7MB (shown in the screenshot). Together these 860 entries make up that 983MB heap allocation.
This is where you ask the question of whether this is abonormal. Is this runaway code, poor caching behavior, or working as designed? A developer needs to answer this question, but at least you have background that something is not quite right.
Let’s say we’ve identified a problem and made an attempt to resolve. This could be adding more memory to the server, increasing the Java heap, or even application code changes. How can you proactively monitor the application server? One way is to take heapdumps at regular intervals and specifically review the object allocations. You can use MAT’s Object Query Language feature to select the object type and view the number of instances. For example, we saw 860 entries of the CachedFormInstance previously, here we see only 9. This could indicate that a code problem is resolved or the server is under relatively low load.
Another approach is to monitor the heap in real time. To do this, I use VisualVM, which can be downloaded at http://visualvm.java.net/. To monitor a remote application server, add the JMX settings to the WebSphere Application Server’s JVM process definition.
-Djavax.management.builder.initial= -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=3334
Now you can create a JMX connectin in VisualVM to forms.demos.ibm.com:3334 as seen in my example.
Notice that I can see the heap – both the current size and how much has been used. The sawtooth line shows heap usage over time. In this case, it looks like the heap is growing until garbage collection kicks in to reclaim the space. You would want to look at heap growth over time or large one time increases or drops and investigate accordingly.