Debugging class loaders in WebSphere

We typically change the classloader from the default of PARENT_FIRST (i.e. get WebSphere to find/load the classes) to PARENT_LAST (i.e the application is responsible for finding/loading classes). However, when we did it for this application, we still got the same error: "java.lang.NoSuchMethodError". So, what we needed was some classloading debugging to find out what was going on...
Setting up debugging of classloaders in Websphere
If you have a dev instance of WebSphere 5.1/6 then the following process should work and allow you to get debugging going. (Note, sorry all you Firefox lovers, the WebSphere admin console needs IE to work for this as it will not allow you to set the right debugging modules.) To setup debugging, go to the Application Server settings (for MDF, this will be "server1")
1. Go to PROCESS DEFINITION > JVM and tick the "Verbose JNI" and "Verbose Classloading" and click on OK.
2. Click at the bottom, CUSTOM PROPERTIES and a new custom property called "ws.ext.debug" set to "true" and again click on OK
3. You have now set up the debugging flags for the bootstrap and system classloaders. You now need to do the same thing for the app servers three classloaders. From the Application server settings (again, "server" for MDF instances) click on LOGGING AND TRACING > DIAGNOSTIC TRACE and click on MODIFY button.
4. A window will pop up (this is the bit that does not work on Firefox) and you need to find (at the bottom) the "Websphere ClassLoader" and click and set to ENABLE ALL. When you click on OK, the text in the Trace Specfication will change to "WebSphere ClassLoader=all=enabled". Now change the log size (default is 20Mb, which is way to small) to around 200Mb. Click on OK. Now SAVE and Syncronise your changes and restart your app.
5. When you restart WAS, you will get a new log file called "trace.log" which will contain all the classloading debugging info, with the boostrap and system classloading going into the native_stdout/err logs.
You can see below the boostrap classloading extract - its a very big file as it outlines all classes WebSphere boots and loads.

...

|Opened /opt/websphere/asvr/java/jre/lib/jce.jar|

|Opened /opt/websphere/asvr/java/jre/lib/charsets.jar|

|Loaded java.lang.Object from /opt/websphere/asvr/java/jre/lib/rt.jar|

|Loaded java.io.Serializable from /opt/websphere/asvr/java/jre/lib/rt.jar|

...

The following sample shows the output in the trace.log (a very very big file, so you are going to need to use text manipulation tools like grep, sed or awk realistically)

...

6/21/07 12:44:47:994 BST]  1e88c7f CompoundClass > findClass name=org.w3c.dom.Node

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

6/21/07 12:44:47:996 BST]  1e88c7f CompoundClass d class org.w3c.dom.Node found in

SinglePathClassProvider : com.ibm.ws.classloader.SinglePathClassProvider@ccd65d classpath =

/u01/websphere/asvr/installedApps/cell/app_war.ear/app.war/WEB-INF/lib/NCSO-6.jar

6/21/07 12:44:47:997 BST]  1e88c7f CompoundClass d loaded org.w3c.dom.Node from this=

6/21/07 12:44:47:998 BST]  1e88c7f CompoundClass d loaded org.w3c.dom.Node using classloader=

6/21/07 12:44:48:162 BST]  1e88c7f CompoundClass > loadClass name=org.w3c.dom.NodeList

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

6/21/07 12:44:48:162 BST]  1e88c7f CompoundClass > findClass name=org.w3c.dom.NodeList

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

...

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21228700%3C/p%3E
http://websphere.sys-con.com/read/196105.htm%3C/p%3E
https://issues.apache.org/bugzilla/show_bug.cgi?id=38719#c2