Analyzing WebSphere Memory

You don’t need to be an expert in a particular application such as WebSphere Portal or IBM Forms to do a quick analysis of memory.  Here’s how.

Get a Heapdump

The easiest way to create a heapdump is using the WebSphere Console.

  1. Start the administrative console.
  2. In the navigation pane, click Troubleshooting > Java dumps and cores.
  3. Select the server_name for which you want to generate the heap dump.
  4. Click Heap dump to generate the heap dump for your specified server.

Heap Dump WAS Console

Should this approach not work.  Use wsadmin to generate the heapdump.

  1. Start the wsadmin scripting client. You have several options to run scripting commands, ranging from running them interactively to running them in a profile.
  2. Invoke the generateHeapDump operation on a JVM MBean.

For example, this is my script to launch the wsadmin client on the local machine.

C:\IBM\FormsExperienceBuilder\WebSphere\AppServer\bin\wsadmin.bat -host localhost -port 8071 -user wasadmin -password password

Next, I run the following commands on the wsadmin prompt.

set jvm [$AdminControl queryNames WebSphere:type=JVM,process=TranslatorServer,node=formsNode01,*]
$AdminControl invoke $jvm generateHeapDump

There are a few settings you need to know.

  • The SOAP port as seen in the first command.
  • The process and node in the second command.

You can get these from the WebSphere Console.  See the previous screenshot. The Server and Node are listed in the table.  These values correspond to the process and node values substituted in the set command. You can find the SOAP port easily by viewing Servers -> Websphere application servers -> <your desired server> -> Communications -> Ports. Look for the SOAP_CONNECTOR_ADDRESS.

The output of these approaches will tell you where the heapdump is located. It will be a file with the PHD extension.

Use the Memory Analyzer Toolkit (MAT)

I’ve used MAT for years. It’s awesome at what it does, which is much more than looking at a pretty graph.  Download it here To open the PHD file generated by WebSphere, you’ll need to add the DTFJ plugin.  Find instructions here

Now it’s super simple to anaylze the PHD. Use File -> Open heap dump -> <select you PHD file>. Let’s take a look at a PHD I recently analyzed.

Memory Analyzer

This is a leak suspect report (the default option in the wizard after opening a heap dump). It’s basically saying that there’s a possible memory leak. Why? Because the WebSphere server’s Java heap is currently set to 1 GB.  Of that, 937MB is being used by one object.  So 90% of the heap is being used by one object – that seems like a leak … or maybe it isn’t.

This is a server that is simply running out of memory because the workload is too high. You can drill down into how the heap is allocated. Use the Open Dominator Tree for entire heap button.

Dominator Tree

The FormCache object is using 90% of the total heap – we can see that in the percentage column and is expected given the graph in the leak suspects report. Look at what’s in this object. It’s a collection for CachedFormInstance private objects. And there are 860 of them! These range in size, from 15MB to 4.7MB (shown in the screenshot). Together these 860 entries make up that 983MB heap allocation.

This is where you ask the question of whether this is abonormal. Is this runaway code, poor caching behavior, or working as designed? A developer needs to answer this question, but at least you have background that something is not quite right.

Proactive Monitoring

Let’s say we’ve identified a problem and made an attempt to resolve. This could be adding more memory to the server, increasing the Java heap, or even application code changes. How can you proactively monitor the application server? One way is to take heapdumps at regular intervals and specifically review the object allocations.  You can use MAT’s Object Query Language feature to select the object type and view the number of instances. For example, we saw 860 entries of the CachedFormInstance previously, here we see only 9. This could indicate that a code problem is resolved or the server is under relatively low load.


Another approach is to monitor the heap in real time. To do this, I use VisualVM, which can be downloaded at  To monitor a remote application server, add the JMX settings to the WebSphere Application Server’s JVM process definition.

Seen here.

VisualVM WebSphere

Now you can create a JMX connectin in VisualVM to as seen in my example.


Notice that I can see the heap – both the current size and how much has been used. The sawtooth line shows heap usage over time. In this case, it looks like the heap is growing until garbage collection kicks in to reclaim the space. You would want to look at heap growth over time or large one time increases or drops and investigate accordingly.


IBM Script Portlet with IBM Sametime

Let’s build a web application using IBM Sametime and the newly released IBM Script Portlet for WebSphere Portal.

IBM Sametime integrates with WebSphere Portal already, and there’s ample documentation in the Knowledge Base on configuring the two products.  But assume you either want to quickly build upon the existing integration or forego the tight integration for a more ad hoc application approach.  To do this, I’ll use the Sametime Proxy samples (stproxy), which provides a rich client-side API – perfect for a web application.
This is a basic web sample straight from the SDK.

<html xmlns="" xml:lang="en" lang="en">
   <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
   <title>IBM Sametime 9 Example 9</title>
   <link type="text/css" rel="stylesheet" href="sametime.css" />
   <script type="text/javascript">
      // Settings for the proxy
      var stproxyConfig = {
         server: "http://<YourProxyServer>:9080",      // Replace with the appropriate server & port
         tunnelURI: "http://<ThisServer>/tunnel.html", // The url of the openAjax Tunnel which must be on the same server as this file
         isConnectClient: false
      djConfig = {
         parseOnLoad: true,
          locale: "<%=request.getLocale().toString().replaceAll("_","-")%>"
   <script type="text/javascript" src="http://<YourProxyServer>:9080/stwebclient/latest/"></script>
   <script type="text/javascript" src="http://<YourProxyServer>:9080/stwebclient/latest/include.js?widget=widgets&auto=false"></script>
    <img src="logo.png" />
       <h1>Example 9</h1>
       <p>This example shows how to embed the entire web client into a web page</p>
        <div style="width:330px;height:550px;border:1px solid black;" id="myClient"

What’s going on in this code?  First, there are two configuration that get created in the head tag of the page.  Next, two script includes are loaded.  The include.js file will use the configuration objects – so order is important.  And finally, the div specifies where to render the Sametime component, which is the web client.

Using the new script portlet, you might create an application by plugging in the respective components.

Script Portlet Editor

This won’t work (besides the need to specify actual servers and remove the JSP scriptlet).  The reason this won’t work is due to how the script portlet emits the final markup.  To see this, you can review the WCM presentation template.

Script Portlet WCM Presentation Template
The [Component name=”script portlet library/links generator”] is what injects the dependencies (CSS and Javascript) into the head tag of the page.  Then another head tag is created to contain the key=Head content (not seen in screenshot).  The style tag encapsulates the key=CSS script editor content, the script tag contains the key=Javascript content, and finally the key=HTML content is created at the end.

The order of the final markup is at fault.  The external Javascript dependencies load before the configuration objects are evaluated in the script tag.  And due to the design of the portlet, I can’t move a <script src=”somescript.js”/> tag into the head editor control.  The design is such that only Javascript code can be placed there – not a reference to a Javascript file.
My point in detailing this is that you should understand what the script portlet does with the web artifacts.  You can then either refactor your code to accommodate or even create a new presentation template to re-order the emitted markup.  The latter is the value of using WCM.  I chose to simply refactor code.  The final version looks like this.

Script Portlet Sametime Code

I’ve kept the configuration objects as Javascript.  The external dependencies have moved to <script> tags in the HTML editor.  And I’m also calling the login function inside the HTML editor rather than as evaluated Javascript in the respective Javascript editor.  Again, minor code changes, but ones that required knowledge of both the Sametime code as well as the Script Portlet design.

Now you can build Sametime widgets with just a bit or markup.

Sametime ExamplesHappy coding!

IBM at SugarCon 2013

Last week, I attended SugarCon in New York to demonstrate “Social Selling” in the IBM booth.  This was a great opportunity to showcase the potential to create social sales experiences by combining products such as IBM Connections and SugarCRM.  A demonstration of the social selling experience is available on YouTube.

Some also asked whether the live demo could be provided.  I’ve uploaded our live demo, which is a story board of web pages – just enough to tell the social sales story and see it in action.

Also at the event were several IBM business partners.  VoiceRite showcased their Sametime Connector for SugarCRM.  Alacrinet created an exceptional web sales experience by combining WebSphere Portal, IBM Web Content Manager, IBM Connections, and SugarCRM.  And Highland Solutions demonstrated “Semantic CRM”, which uses IBM Connections and SugarCRM to surface sales collateral in Sugar dashlets.  A demonstration of Semantic CRM is available on YouTube.

And what visit to New York would be complete without a view of the skyline?

SugarCon Skyline


Customizing the SmartCloud for Social Business Login

Creating a custom login form for SmartCloud for Social Business is pretty basic – like “I once took a class on HTML at a coffee shop” basic.  It’s just an HTML form that anyone with a little bit of web development can achieve.  For example, let’s consider the following custom login form.

Business Partner Login Form

I’ve copied the login form (View -> Page Source on the login page), hosted it on another server, and added a custom graphic.  The Javascript files, style sheets, and submission location still reside on SmartCloud.  Take a look; it’s basic HTML.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="" xml:lang="en" lang="en">
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="viewport" content="user-scalable=no, width=device-width" />
<link rel="shortcut icon"
href="" />
<link type="text/css"
rel="stylesheet" />
<link rel="stylesheet"
type="text/css" />
<script type="text/javascript"
<script type="text/javascript"
<body class="tundra" style="text-align: center;"
<div id="frame">
<div id="joinBox" style="text-align: left;">
<div id="mainBox" role="main" aria-labelledby="pageTitle">
<div class="mainBox_inner">
<!-- need to translate the alt tag contents -->
<div id="company_logo" role="banner">
<a class="ibm_login" href=""
style="text-align: left;" id="loginLogo"><img
src="ibm-bp.png" alt="IBM" /></a>
<h1 id="pageTitle">Host Sametime Meeting</h1>
<div id="errorMessage" class="error_login" style="display: none;"
role="alert" aria-live="assertive">Your username &amp;
password are not recognized by the service. Please check your
login credentials and try again.</div>
<div id="suspenderrorMessage" class="error_login"
style="display: none;" role="alert" aria-live="assertive">
Your account has been expired or suspended.</div>
<form id="loginActionForm" name="login" onsubmit="return false;"
method="post" role="form">
Log in to <span id="productName1"></span>
<input type="hidden" name="login-form-type" value="pwd" /> <input
type="hidden" id="error-code" name="error-code" value="" /> <label
id="loginActionForm_eMail" for="username">E-mail
address:</label> <input type="text" name="username" id="username"
class="text" aria-required="true" /> <label
id="loginActionForm_Password" for="password">Password:</label> <input
type="password" name="password" id="password" class="text"
aria-required="true" />
<div style="padding: 15px 0px 0px 0px;">
<input type="submit" class="lotusBtn ok okLeft" id="submit_form"
onclick="submitLoginForm();" value="Log In" /> <a role="button"
class="forgot" id="forgot" href="#"
<div style="clear: both;">&nbsp;</div>
<div style="clear: both;">&nbsp;</div>
<!-- Do Not Remove For Gomez Scripts -->
<div style="display: none;">6h5k7fQJ1VSa6qi5dxgr</div>
<script type="text/javascript">
<iframe src=""width="0" height="0" style="visibility: hidden"></iframe></body>

Notice the highlighted lines?  This is a hidden IFRAME containing the host URL for Sametime Meetings.  “So, what?” you ask.

Think about what would happen if you pasted any SmartCloud URL into your browser. Unless you’re already logged in, you are likely to be issued the login page by Tivoli Access Manager.  TAM is also setting a cookie in your browser to your pasted URL. This way – after you login – TAM reads the cookie, and you get sent back to the URL you pasted. Otherwise, you just end up on SmartCloud for Social Business’ home page.

By including the hidden IFRAME, the form is also telling TAM where we want to go after login. It’s doing two functions: login and redirect. In the above code, a user logs in, and he or she is entered immediately into a Sametime Meeting as the host.  This bypasses the SmartCloud user interface and alleviates the need to manually press the ‘Host Meeting’ button.

Removing the SmartCloud user experience in this manner might seem counter-intuitive, but there are use cases for ISVs that make this solution desireable.

Sametime 8.5.2 Install Failure

While installing a new Sametime 8.5.2 server, I ran into the following APAR LO71871. Unfortunately, the APAR says it was not reproduced and presumably never fixed. The validate buttons fail in the Installation Manager wizard – again presumably – on IIM 1.5+.  You will see an error in the log.

Unhandled event loop exception
java.lang.NoSuchMethodError: com/ibm/cic/common/logging/Logger.debug(Ljava/lang/String;Ljava/lang/Object;)

I am using IIM 1.6.1 since I have other servers previously installed, and I cannot simply uninstall 1.6.1 unless I also uninstall the other servers (not going to happen). I tried a silent install, which had issues. Attempting to hack the class and plugin was a dead end. The “fix” I settled on was to rename the Installation Manager program folder (C:\IBM\Installation Manager) and the data folder (C:\ProgramData\IBM\Installation Manager), and then install version 1.4.2 that ships with the install media. In the future, I’ll likely need to switch between my 1.6.1 and 1.4.2 IIM installations to install fixes and etc.

Troubleshooting Notes and Sametime

Often IBM support will request trace data to further investigate a problem. What is this data, how is it enabled, and how can it be used to pro-actively solve problems before contacting IBM? The Expeditor Wiki contains detailed information related to enabling and collecting trace. For information regarding trace (log) files available and their usage, see the article IBM Lotus Expeditor Logs-What are they good for. This post is intended to be a practical top-down overview for administrators investigating problems with deployed Notes and Sametime applications.

Tracing is debug information written to the platform’s log file. For performance reasons, tracing is not enabled by default; it is normally enabled after a problem has been encountered.  Trace messages are added during the development of an application in anticipation of problems encountered in the field. Such messages may include configuration settings, runtime values, or output of problems encountered during the operation of an application. Most Notes and Sametime applications use JSR47 logging. Because such tracing uses JSR47, you can follow a consistent approach to debugging most problems. A simple snippet on JSR47 tracing is below.

private final String clazz = ViewPart.class.getName(); // this Class' name
private Logger logger = Logger.getLogger(clazz); // the logger for this Class
private void foo() {
logger.entering(clazz, "foo");
try {
SECRET.charAt(1); // java.lang.StringIndexOutOfBoundsException
logger.log(Level.FINE, "The secret is " + SECRET);
} catch (java.lang.StringIndexOutOfBoundsException e) {
"foo()'s SECRET did not have a character at index 1", e);
logger.exiting(clazz, "foo");

Tracing has varying levels that control the amount of data written to the trace files. In order of information, they are: CONFIG, FINE, FINER and FINEST. In the above example, the message “The secret is” will be written to the trace file if we enable trace at the FINE level. For practical reasons, it’s generally accepted to simply start with the FINEST level to ensure no relevant messages are accidentally omitted.

IBM support may ask you to enable trace for a particular plugin. For example, the request may be stated as, “Please enable trace for at the FINEST level”. Or support may refer to the specification “ FINEST”.

The preferred manner is using dynamic tracing. Support often prefers dynamic tracing because it can be enabled just prior to reproducing a problem. By enabling trace only when needed, the amount of unnecessary information is omitted. Enabling trace in this manner can be done with the below command.

setlogrlev FINEST

To enable trace for multiple packages, simply issue the setlogrlev command with the package and level on multiple lines (use enter at the end of each command).

Tracing may be also enabled statically – meaning the trace level will persist after the client restarts. Because dynamic tracing ends when the client is restarted, static tracing is helpful when the client is expected to be restarted (such as provisioning or installation problems) or when the problem may not occur for some time. Update the file with the following.

For multiple packages, add the trace specifications on separate lines. A restart must occur after the has been updated.

Application users and administrators can be pro-active in problem solving using the following steps.

  1. Identify a problem has occurred. This can be done either when an application dialog appears with an error message or the application simply fails to respond appropriately.
  2. Consult the trace file. Use the Viewing the trace file article to review the log file. Identify any suspicious entries and enable trace in the relevant area. Suspicious entries may include any message severity with WARNING or SEVERE. A stack trace (seen in the message below) is a good indication of abnormal behavior.
  3. Enable tracing. Below is a snippet of a WARNING message. The code encountering a “problem” is within the package. Enabling trace for at the FINEST level is a good start, but it is too narrow in scope. This would obtain information only for the package and the problem may exist “higher” in the stack. As a general rule of thumb, expand the scope by removing one or two packages. For example, instead of, use or But be careful, not to go too high such as or or you may receive too much logging, which is difficult to review.

Finding a Plugin’s ‘Owner’

How much importance do you give to a plugin’s name?

Most developers new to rich client platform development generally name their projects not realizing the importance of the project name. You might start a new software feature with the project name MyRemoteWebServiceClient. The “problem” is that Eclipse’s plugin development environment inherits the bundle’s symbolic name from the project name. So both the project name and plugin name are MyRemoteWebServiceClient. Why is this a problem?

If you look at Lotus Notes, Sametime, or Expeditor plugin names, they appear verbose. A plugin named seems more unwieldily than just MyRemoteWebServiceClient. Naming convention becomes more useful after deployment.

Generally, plugin naming convention mimics the Java package naming within the source code. For any plugin, I’d expect the top level package in the plugin to be named the same. For example, should have a top level package called Why is this useful? Let’s assume that a product fails with the exception:

Caused by: org.xml.sax.SAXParseException: The root element is required in a well-formed document. Message being parsed:
at $Proxy28.getSystemTime(Unknown Source)

Once you move beyond the Sun packages, you’re in the plugin developer’s code . The class threw the exception, but what feature contains this plugin?  How can I find the plugin owner to fix the defect? You can start by inspecting the client platform using some OSGI commands from a previous post. To find the bundle, you might use the class’ package hierarchy and issue the command:


This might output the following.

Framework is launched.
id State Bundle
24330 ACTIVE

Now you know there are actually two bundles sharing the identifier. They’re probably related, but more importantly the bundle with ID 24329 matches the offending class’ package seen in the stack trace. There’s the offending plugin – or at least a highly suspicious one.

Using the OSGI Console in IBM Notes and Sametime

The OSGI console can assist with development investigation and troubleshooting runtime errors.  It’s worth becoming familiar with a few basic commands.

When using the Expeditor or Sametime client, append -console to the client shortcut. If you are using Notes, append -RPARAMS -console. For example, here is the Notes shortcut C:\notes\notes.exe -RPARAMS -console.  After starting the client, the OSGI console will display in a separate window.  Eclipse developers may already be familiar with the OSGI console.  It’s often used during development, and the same commands listed below are applicable to Eclipse.

Notes, Sametime, or Expeditor OSGI Console
Notes, Sametime, or Expeditor OSGI Console
Eclipse OSGI Console
Eclipse OSGI Console

Commands can be input to the platform by typing the command next to the osgi> prompt. If you do not see the prompt simply press enter. For a complete list of commands available, type ? and then press enter.

If the client experiences a platform hang or crash, the following commands provide useful data to locate the cause.

  • dump heap generates a JVM heap dump which is useful when analyzing heap memory usage or memory leaks.
  • dump threads generates a javacore which can be used to analyze the stack trace of various threads; it’s recommended to run dump threads several times and compare threads over time.

See IBM Thread and Monitor Dump Analyzer for Java for analyzing output from the dump commands. The location of dump output can be redirected by editing the file.


When trying to isolate why a particular feature may be failing, it’s useful to start by understanding the bundle’s state.  Is it installed, is it started, or has it failed to start? The OSGI console accepts either the bundle’s symbolic name or the bundle ID.

  • ss or short status list matching bundles and their state.
  • start starts a bundle.
  • stop stops the bundle.
  • diag determines if the bundle has unresolved dependencies which could lead to runtime problems or an inability to use the bundle.
  • bundle displays a verbose set of information related to exported packages, dependencies, etc.

To enable trace output or limit the amount use the following command.

  • setlogrlev dynamically adjusts log output provide the logger ID (normally the bundle’s symbolic name) and a level (FINEST, FINER, FINE, INFO, WARNING, CONFIG).

Here are a few examples of the above commands.

setlogrlev FINEST
ss sync
start 606