Websphere EJB Optimization: Convert pass by value calls to pass by reference without Local Interface

I've encountered a specific situation in the project where EJB calls should be optimized to reduce response time and memory footprint without modifying code.

Every J2EE programmer know about EJB local interfaces that enables EJB calls like local method calls as long as client and EJB resides in same physical machine. This option reduces overhead of data serialization and transfer between client and EJB layer and also reduces memory foot print since the data should be copied from one layer to another layer.

How do we enable the same feature for any existing project missing Local Interfaces ?

IBM websphere provides the following options to optimize EJB calls without adding local interfaces.

1) Set "Pass by reference" property  in ORB service to true. Navigate to Application servers --> server_name --> Container services --> ORB service. Find the property "pass by reference" and set the value to TRUE.

2)Specify noLocalCopies in servant.jvm.options (-Dcom.ibm.CORBA.iiop.noLocalCopies=1).

Caution: We must ensure that the application is not negatively impacted by using pass-by-reference semantics. For example, if one bean passes another bean an object, expecting that the second bean can modify the object without affecting the first bean's object.

Unable to upload the TIF image of large dimensions 1920x1080 to Java Application

I got an incident from customer saying that we are unable to upload TIF images to application and how ever they are able to upload when it was opened in Photoshop and saved.

I have raised this issue with third party and we were not having exact application logs to look into

Reason:

Application could not load this because it was missing the dimension information (width x height). This information is required in order for STEP to be able to load a TIFF image.

The dimension information is metadata on the file that is normally provided by whatever program the file was created in. Unfortunately, I cannot tell what program created this file, as it did not leave its name in the metadata either. Re-saving this image in any Photoshop application will supply the required metadata and then you should be able to import it into STEP. Just be aware that Photoshop will default to 72 dpi unless a different resolution is set.

Workaround:

1. Select all the images into Microsoft Picture Manager.
2. Click on Edit Pictures -- >Re-size and then select the required dimensions and press OK.
3. Click on File -- >Save All to save all the re-sized images

Please note that original images will be re-sized and make sure that back up original images in case if required

Java DNS Cache issues

Java 1.5 DNS Cache issues:

Recently I encountered an issue with the application and things went pretty ugly. One of the database server is migrated from one hardware to another (copy DB to new HW) and DNS change was part of the implementation. The permissions on old database tables were revoked, but we are able to connect to that database.

Once all migration activities are completed I started testing our application (J2EE) connectivity to new database server (after couple of hours) I encountered permission issue on selecting database tables. 

DBA's were telling that I'm still connecting to old servers and requested to point JDBC connection to new servers, but the fact is we have the new host name already in our JDBC string which is used in other part of the application.

Finally we figured it out the issue would have happened because of DNS cache and I found out that DNS queries to our new database server has TTL value of 3600 seconds, so Linux DNS cache is not an issue since I started testing couple of hours after DNS change.

use dig +ttlid <host-name> command in Linux to find out TTL value returned by DNS servers.

;; ANSWER SECTION:
google.com.             300     IN      A       74.125.225.39
google.com.             300     IN      A       74.125.225.34

The value of 300 is TTL for google.com and DNS entries for google.com will be flushed out from DNS cache after 300 seconds.

So where is the issue ?

Root Cause:

Java (1.4 & 1.5) has internal DNS cache whose entries never expires ( TTL =-1) which caused our application to connect to old servers.

Resolution:

Change the TTL value of Java DNS cache:

The SDK provides the following properties to control network address caching:
networkaddress.cache.ttl (default: -1)

You can use the following four properties to override the default behavior.

• networkaddress.cache.ttl (default: -1)
  Specified in the java.security file to indicate the caching policy for successful name lookups from the name service. The value is specified as an integer to indicate the number of seconds to cache the successful lookup. A value of -1 indicates "cache forever".
  
• networkaddress.cache.negative.ttl (default: 10)
  Specified in the java.security file to indicate the caching policy for unsuccessful name lookups from the name service. The value is specified as an integer to indicate the number of seconds to cache the failure for unsuccessful lookups. A value of 0 indicates "never cache". A value of -1 indicates "cache forever".
  
• sun.net.inetaddr.ttl
  This property is a sun private system property that corresponds to networkaddress.cache.ttl. It takes the same value and has the same meaning, but you can set it as a command-line option. However, the preferred way is to use the security property mentioned above.
  
• sun.net.inetaddr.negative.ttl
  This property is a sun private system property that corresponds to networkaddress.cache.negative.ttl. It takes the same value and has the same meaning, but you can set it as a command-line option. However, the preferred way is to use the security property mentioned above.
  
  You can disable caching by adding -Dsun.net.inetaddr.ttl=0 on the command line starting the JVM. However, you cannot set the value of networkaddress.cache.ttl on the command line. You can set the required value in the java.security file, which is located in the %JRE%\lib\security directory.
  

Workarounds:

• Upgrade to JDK 1.6 / JDK 1.7 which introduced changes to default DNS caching behavior. The TTL values are set to 30 seconds without a security manager. 
• JVM restart after DNS changes.

References:

1. http://www-01.ibm.com/support/docview.wss?uid=swg21207534
2. http://www-01.ibm.com/support/docview.wss?uid=swg21439843

Websphere connection pool errors:ORA-02396: exceeded maximum idle time, please connect again

Today one of our application experienced Oracle time out errors “ORA-02396: exceeded maximum idle time, please connect again” when ever user submits long running operations and was able to re produce the same issue in my local environment with the help of debugger .

Oracle profile for the app user has idle time out of 45 minutes and when ever application took more than 45 minutes for processing the request and tries to update the results to DB, the application is throwing this error. Oracle discards idle connections after 45 minutes and for some reason websphere is not discarding this stale connections. Application is getting connection from Connection pool via JNDI lookup and closing out resources during method exits and not keeping the connection open.


The data source has the default values for connection pool properties.
Reap time: 180 seconds
Unused timeout: 1800 seconds
Aged timeout: 0 seconds
Global Transactions :No
EJB Transaction Attribute:Default (required) -CMT
Transaction started : Not in Web Layer/EJB Layer. DAO classes are opening connections.
Pattern: open/use/close pattern for DB connections.

I’ve changed “Aged timeout parameter” to 1200 seconds in my local environment which didn’t help. Not sure why Connection pool is keeping stale connections.

Finally I decided to do more research why stale connections are not discarded from Websphere connection pool even with appropriate settings.

The application specifies “required” transaction attribute for all container managed transactions and resource sharing scope is sharable for all local transactions (LTC).

According to websphere documentation when ever new JDBC connection is allocated under LTC (Local transaction containment) it is marked as having an affinity to the current LTC and when application closes the connection it is not returned to the pool until transaction ends and connection remains used by this transaction so connection pool mechanism can’t discard this connection even though it becomes stale connection.

Please refer http://www.ibm.com/developerworks/websphere/library/techarticles/0506_johnsen/0506_johnsen.html for more details.

I have verified this in my local websphere and when ever LTC has started the same logical connection is returned every time application tries to acquire db connection.


Here are some resolutions for this issue

1) Change the resource sharing scope of JDBC connections to “UnSharable” which creates more physical connection to database instead of sharing single physical connection using multiple logical connections and connection will be released to connection pool after application closes the connection.

2Add exception handler to specific part of the code and obtain new connection gracefully

3)Optimize application processing or long running queries

I did more research on the possible implementation of option (1) and it seems that option (1) is not feasible to implement with our application DB access pattern.

Resource sharing property can be specified either in web.xml or ejb-jar.xml along with resource reference (Datasource) and application code (servlet or EJB) should lookup the data source using this reference container will utilize sharable property (res-sharing-scope).How ever our app code is not using resource references to obtain database connection and instead obtain connections in DAO class.
Since the sharable property in CMConfigData (Connection Manager config Data) object in websphere object and it is not exposed outside (no setters) , the possibility of using IBM helper classes to set the sharable property in the opened database connection is blocked


We can implement Option 2 and 3 together and IBM also suggests handling StaleConnection exceptions in the code and more information is available at http://www.redbooks.ibm.com/redbooks/SG246688/axa.htm

Debugging class loaders in WebSphere

We typically change the classloader from the default of PARENT_FIRST (i.e. get WebSphere to find/load the classes) to PARENT_LAST (i.e the application is responsible for finding/loading classes). However, when we did it for this application, we still got the same error: "java.lang.NoSuchMethodError". So, what we needed was some classloading debugging to find out what was going on...
Setting up debugging of classloaders in Websphere
If you have a dev instance of WebSphere 5.1/6 then the following process should work and allow you to get debugging going. (Note, sorry all you Firefox lovers, the WebSphere admin console needs IE to work for this as it will not allow you to set the right debugging modules.) To setup debugging, go to the Application Server settings (for MDF, this will be "server1")
1. Go to PROCESS DEFINITION > JVM and tick the "Verbose JNI" and "Verbose Classloading" and click on OK.
2. Click at the bottom, CUSTOM PROPERTIES and a new custom property called "ws.ext.debug" set to "true" and again click on OK
3. You have now set up the debugging flags for the bootstrap and system classloaders. You now need to do the same thing for the app servers three classloaders. From the Application server settings (again, "server" for MDF instances) click on LOGGING AND TRACING > DIAGNOSTIC TRACE and click on MODIFY button.
4. A window will pop up (this is the bit that does not work on Firefox) and you need to find (at the bottom) the "Websphere ClassLoader" and click and set to ENABLE ALL. When you click on OK, the text in the Trace Specfication will change to "WebSphere ClassLoader=all=enabled". Now change the log size (default is 20Mb, which is way to small) to around 200Mb. Click on OK. Now SAVE and Syncronise your changes and restart your app.
5. When you restart WAS, you will get a new log file called "trace.log" which will contain all the classloading debugging info, with the boostrap and system classloading going into the native_stdout/err logs.
You can see below the boostrap classloading extract - its a very big file as it outlines all classes WebSphere boots and loads.

...

|Opened /opt/websphere/asvr/java/jre/lib/jce.jar|

|Opened /opt/websphere/asvr/java/jre/lib/charsets.jar|

|Loaded java.lang.Object from /opt/websphere/asvr/java/jre/lib/rt.jar|

|Loaded java.io.Serializable from /opt/websphere/asvr/java/jre/lib/rt.jar|

...

The following sample shows the output in the trace.log (a very very big file, so you are going to need to use text manipulation tools like grep, sed or awk realistically)

...

6/21/07 12:44:47:994 BST]  1e88c7f CompoundClass > findClass name=org.w3c.dom.Node

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

6/21/07 12:44:47:996 BST]  1e88c7f CompoundClass d class org.w3c.dom.Node found in

SinglePathClassProvider : com.ibm.ws.classloader.SinglePathClassProvider@ccd65d classpath =

/u01/websphere/asvr/installedApps/cell/app_war.ear/app.war/WEB-INF/lib/NCSO-6.jar

6/21/07 12:44:47:997 BST]  1e88c7f CompoundClass d loaded org.w3c.dom.Node from this=

6/21/07 12:44:47:998 BST]  1e88c7f CompoundClass d loaded org.w3c.dom.Node using classloader=

6/21/07 12:44:48:162 BST]  1e88c7f CompoundClass > loadClass name=org.w3c.dom.NodeList

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

6/21/07 12:44:48:162 BST]  1e88c7f CompoundClass > findClass name=org.w3c.dom.NodeList

this=com.ibm.ws.classloader.CompoundClassLoader@42d227

...

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21228700%3C/p%3E
http://websphere.sys-con.com/read/196105.htm%3C/p%3E
https://issues.apache.org/bugzilla/show_bug.cgi?id=38719#c2

How to change the users passwords from J2EE Application ?

There are no generic J2EE API's to manipulate J2EE application user details.How ever each Application servers exposes MBeans to perform these operations through console or via scripting tools.
Eg: While creating the users from admin console, specifc mbeans will be called behind the scenes.

I wrote an example to reset the password for particular user using weblogic MBeans from Servlet.Eventhough this is not the exhaustive example which provides the basic steps for listing users/groups and modifying the user passwords


import java.io.IOException;
import java.io.PrintWriter;

import javax.naming.Context;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import weblogic.jndi.Environment;
import weblogic.management.MBeanHome;
import weblogic.management.security.RealmMBean;
import weblogic.management.security.authentication.AuthenticationProviderMBean;
import weblogic.management.security.authentication.GroupEditorMBean;
import weblogic.management.security.authentication.GroupMemberListerMBean;
import weblogic.management.security.authentication.MemberGroupListerMBean;
import weblogic.management.security.authentication.UserEditorMBean;
import weblogic.management.security.authentication.UserPasswordEditorMBean;
import weblogic.management.security.authentication.UserReaderMBean;
import weblogic.management.security.credentials.CredentialMapperMBean;
import weblogic.management.security.credentials.UserPasswordCredentialMapEditorMBean;
import weblogic.management.security.credentials.UserPasswordCredentialMapReaderMBean;
import weblogic.management.utils.InvalidParameterException;
import weblogic.management.utils.NotFoundException;

public class PasswordServlet extends HttpServlet
{

protected transient MBeanHome home;


/**
* The doGet method of the servlet.

*
* This method is called when a form has its tag value method equals to get.
*
* @param request the request send by the client to the server
* @param response the response send by the server to the client
* @throws ServletException if an error occurred
* @throws IOException if an error occurred
*/
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{

doPost(request, response);
}

/**
* The doPost method of the servlet.

*
* This method is called when a form has its tag value method equals to post.
*
* @param request the request send by the client to the server
* @param response the response send by the server to the client
* @throws ServletException if an error occurred
* @throws IOException if an error occurred
*/
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
response.setContentType("text/html");
findAdminHome();
PrintWriter out = response.getWriter();
RealmMBean[]realms= home.getActiveDomain().getSecurityConfiguration().findRealms();
for(int n=0;n< authProviders.length; i++) { if (authProviders[i] instanceof UserEditorMBean) { out.println("UserEditorMBean: " + authProviders[i].wls_getDisplayName()); } if (authProviders[i] instanceof GroupEditorMBean) { out.println("GroupEditorMBean: " + authProviders[i].wls_getDisplayName()); } if (authProviders[i] instanceof MemberGroupListerMBean) { out.println("MemberGroupListerMBean: " + authProviders[i].wls_getDisplayName()); } if (authProviders[i] instanceof GroupMemberListerMBean) { out.println("GroupMemberListerMBean: " + authProviders[i].wls_getDisplayName()); } if (authProviders[i] instanceof UserReaderMBean) { UserReaderMBean users=(UserReaderMBean)authProviders[i]; try { //String listUser=users.listUsers("USR*", 100); out.println("Users List: " +users.userExists("USR333")); out.println("Users List: " +users.userExists("USR330")); } catch (InvalidParameterException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } for (int i = 0; i < credMappers.length; i++) { if (authProviders[i] instanceof UserPasswordCredentialMapReaderMBean) { UserPasswordCredentialMapReaderMBean user=(UserPasswordCredentialMapReaderMBean)authProviders[i]; out.println("UserPasswordCredentialMapReaderMBean: " + credMappers[i].wls_getDisplayName()); } if (authProviders[i] instanceof UserPasswordCredentialMapEditorMBean) { out.println("UserPasswordCredentialMapEditorMBean: " + credMappers[i].wls_getDisplayName()); } if (authProviders[i] instanceof UserPasswordEditorMBean) { UserPasswordEditorMBean pwd=(UserPasswordEditorMBean)authProviders[i]; try { pwd.changeUserPassword("all", "allusers", "ALLUSERS"); out.println("Password Chnaged : " + credMappers[i].wls_getDisplayName()); } catch (NotFoundException e) { e.printStackTrace(); } catch (InvalidParameterException e) { e.printStackTrace(); } } } } out.flush(); out.close(); } private void findAdminHome() { Environment env = new Environment(); env.setProviderUrl("t3://localhost:7001"); env.setSecurityPrincipal("weblogic"); env.setSecurityCredentials("weblogic"); try { Context ctx = env.getInitialContext(); home = (MBeanHome) ctx.lookup(MBeanHome.ADMIN_JNDI_NAME); } catch (javax.naming.NamingException e) { e.printStackTrace(); } } }

Connect to Websphere JDBC datasource from standalone Client:

We may hit a situation to connect to configured Websphere JDBC datasource from standalone java client and need appropriate drivers for target database we are connecting to.

There are no additional jars need to be downloaded from IBM website except database driver jar and we will be using IBM command(setupCmdLine.sh) to setup environment and connect to Oracle database.

Script to setup WAS environment and call Java Client:

# Use IBM script to initialize common WebSphere runtime settings

REPLACE_WAS_HOME=${WAS_HOME}

. $WAS_HOME/bin/setupCmdLine.sh

# Add oracle drivers to classpath

CLASSPATH="$WAS_CLASSPATH:/tmp/JDBC/ojdbc5.jar:."

# Specify JVM

JAVA="$JAVA_HOME/bin/java"

# Specify JVM command line options

JAVA_OPTS="-Djava.ext.dirs=$JAVA_HOME/jre/lib/ext:$WAS_EXT_DIRS:$WAS_HOME/plugins"

JAVA_OPTS="$JAVA_OPTS  -cp $CLASSPATH"

$JAVA $JAVA_OPTS ConnectJDBC

Java Client:

import javax.naming.Context;

import javax.naming.InitialContext;

import javax.naming.NamingException;

import java.io.Serializable;

import java.sql.Connection;

import java.util.ArrayList;

import java.util.Hashtable;

import java.util.List;

import java.io.*;

public class ConnectJDBC {

public static void connect() {

final Hashtable env = new Hashtable();

env.put(Context.INITIAL_CONTEXT_FACTORY,"com.ibm.websphere.naming.WsnInitialContextFactory");

env.put(Context.PROVIDER_URL, "corbaloc:iiop::,::,::");

final Context jndiContext;

try {

jndiContext = new InitialContext(env);

} catch (NamingException e) {

System.err.println("Could not create JNDI API context: "+ e.toString());

System.exit(1);

return;

}

try {

javax.sql.DataSource ds =  ((javax.sql.DataSource)jndiContext.lookup(""));

Connection conn =ds.getConnection(,);

String SQL = "{call . }";

} catch (Exception e) {

System.err.println("Could not perfrom JDBC Procedure Call: "+ e.toString());

System.exit(2);

}

}

public static void main(String[] args) {

connect();

}

}

Note: Please make sure the script #setupCmdLine.sh has appropriate permissions to avoid any failures. Note that standalone client will not be able use component-managed authentication for database and it's responsibility of client to provide authentication details during database connection..

Dependency Jars:

The standalone client need the following jars to connect to JDBC datasource configured in websphere when we are unable to use #setupCmdLine.sh script to setup the environment.

com.ibm.ws.runtime_6.1.0.jar

ojdbc14.jar

rsadbutils.jar

rsahelpers.jar

sibc.jndi.jar

sibc.jms.jar

sibc_install-o0902.06.jar

com.ibm.jaxws.thinclient_6.1.0.jar

Note: I performed all these tests using eclipse editor and Websphere 6.1 server.

Post JMS messages to WebSphere queues from stand alone Client:

We may encounter a situation to send JMS messages to queues/topics configured in Websphere from standalone java program and unit/functional testing would be one good scenario for our case.

We need additional client libraries to lookup resources from JNDI and connect to websphere SIB and queues which can be downloaded from here.

Client Program:

import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Hashtable;
import java.util.List;
import java.io.*;


public class PostJMSMessage {

public static void publish() {
final Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY,"com.ibm.websphere.naming.WsnInitialContextFactory");
env.put(Context.PROVIDER_URL, "iiop://:");

final Context jndiContext;
try {
jndiContext = new InitialContext(env);
} catch (NamingException e) {
System.err.println("Could not create JNDI API context: "+ e.toString());
System.exit(1);
return;
}

try {
ConnectionFactory connectionFactory = (ConnectionFactory) jndiContext.lookup("");
Connection qConn = connectionFactory.createConnection();
Session qSession = qConn.createSession(false,Session.AUTO_ACKNOWLEDGE);
Queue q = (Queue) jndiContext.lookup("");

MessageProducer producer = qSession.createProducer(q);
ObjectMessage message = qSession.createObjectMessage();
message.setObject((Serializable) );
producer.send(message);

producer.close();
qSession.close();
qConn.close();
in.close();
} catch (Exception e) {
System.err.println("Could not perfrom JMS operation: "+ e.toString());
System.exit(2);
}
}

public static void main(String[] args) {
publish();
}
}

Jars required in classpath:

The following jars are required in classpath along with any application specific jars to compile and run the program

sibc.jms.jar

sibc.jndi.jar

sibc_install-o0902.06.jar

Compile & Run:

Navigate to the directory where the client program and jars are copied and find out IBM jdk path.

/opt/IBM/WebSphere/AppServer/java/bin/javac -cp :sibc.jms.jar:sibc.jndi.jar:. PostJMSMessage.java

/opt/IBM/WebSphere/AppServer/java/bin/java -cp :sibc.jms.jar:sibc.jndi.jar:. PostJMSMessage