Java DNS Cache issues

Java 1.5 DNS Cache issues:

Recently I encountered an issue with the application and things went pretty ugly. One of the database server is migrated from one hardware to another (copy DB to new HW) and DNS change was part of the implementation. The permissions on old database tables were revoked, but we are able to connect to that database.

Once all migration activities are completed I started testing our application (J2EE) connectivity to new database server (after couple of hours) I encountered permission issue on selecting database tables. 

DBA's were telling that I'm still connecting to old servers and requested to point JDBC connection to new servers, but the fact is we have the new host name already in our JDBC string which is used in other part of the application.

Finally we figured it out the issue would have happened because of DNS cache and I found out that DNS queries to our new database server has TTL value of 3600 seconds, so Linux DNS cache is not an issue since I started testing couple of hours after DNS change.

use dig +ttlid <host-name> command in Linux to find out TTL value returned by DNS servers.

;; ANSWER SECTION:
google.com.             300     IN      A       74.125.225.39
google.com.             300     IN      A       74.125.225.34

The value of 300 is TTL for google.com and DNS entries for google.com will be flushed out from DNS cache after 300 seconds.

So where is the issue ?

Root Cause:

Java (1.4 & 1.5) has internal DNS cache whose entries never expires ( TTL =-1) which caused our application to connect to old servers.

Resolution:

Change the TTL value of Java DNS cache:

The SDK provides the following properties to control network address caching:
networkaddress.cache.ttl (default: -1)

You can use the following four properties to override the default behavior.

• networkaddress.cache.ttl (default: -1)
  Specified in the java.security file to indicate the caching policy for successful name lookups from the name service. The value is specified as an integer to indicate the number of seconds to cache the successful lookup. A value of -1 indicates "cache forever".
  
• networkaddress.cache.negative.ttl (default: 10)
  Specified in the java.security file to indicate the caching policy for unsuccessful name lookups from the name service. The value is specified as an integer to indicate the number of seconds to cache the failure for unsuccessful lookups. A value of 0 indicates "never cache". A value of -1 indicates "cache forever".
  
• sun.net.inetaddr.ttl
  This property is a sun private system property that corresponds to networkaddress.cache.ttl. It takes the same value and has the same meaning, but you can set it as a command-line option. However, the preferred way is to use the security property mentioned above.
  
• sun.net.inetaddr.negative.ttl
  This property is a sun private system property that corresponds to networkaddress.cache.negative.ttl. It takes the same value and has the same meaning, but you can set it as a command-line option. However, the preferred way is to use the security property mentioned above.
  
  You can disable caching by adding -Dsun.net.inetaddr.ttl=0 on the command line starting the JVM. However, you cannot set the value of networkaddress.cache.ttl on the command line. You can set the required value in the java.security file, which is located in the %JRE%\lib\security directory.
  

Workarounds:

• Upgrade to JDK 1.6 / JDK 1.7 which introduced changes to default DNS caching behavior. The TTL values are set to 30 seconds without a security manager. 
• JVM restart after DNS changes.

References:

1. http://www-01.ibm.com/support/docview.wss?uid=swg21207534
2. http://www-01.ibm.com/support/docview.wss?uid=swg21439843