JBoss clustering Quick Reference Guide

So i was involved in clustering two JBoss instances and for some reason i had lost my guide i made for myself last time i did a clustering implementation. This time i thought ill blog it so as to keep it as a reference for myself as well as anyone who might need to quickly do JBoss Clustering as we all know how time critical all IT projects are. I will put this in a step by step approach to make it easier to read and comprehend. Note that this approach assumes you are clustering the all configuration in JBoss.

• Locate the cluster-service.xml located within the deploy folder. Open it and locate the attribute PartitionName within the first mbean you see in that file and change the value ${jboss.partition.name:DefaultPartition} to ${jboss.partition.name}. As in delete the part DefaultParition. 
•  Find the mbean code which equals to org.jboss.ha.jndi.HANamingService and org.jboss.ha.hasessionstate.server.HASessionStateService. Within that locate the tag   . Chane the value jboss:service=${jboss.partition.name:DefaultPartition} to jboss:service=${jboss.partition.name}. Also add the following to both the mbeans mentioned above ${jboss.partition.name}.

• In your application jndi reference make sure to change the port to 1100 because in a clustered environment HA-JNDI(High availability JNDI) is used to look up resource. So localhost:1099 should be localhost:1100. Also you have to give the jndi in a comma separated way indicating all servers being clustered. This is needed in a case where if one server goes down still look ups will not fail as you have specified all servers. Say for example you have two servers named ABC and XYZ. Then if you only specify your jndi url as ABC:1100, this will work but in the event that ABC goes down all your other look ups will also fail. So to avoid that kind of situations make sure to define it as ABC:1100,XYZ:1100. This will make sure to try the other server in the case one fails.
• In all your EJBs make sure to put the Annotation @Clustered(partition="mypartition"). The name we specify should be specified in your run script which i will give in the end of this article.
• If you running on Redhat make sure the HA-JNDI port is open by issuing the following commands to open port 1100;
  
  
  
  
   

iptables -A INPUT -i eth0 -p tcp --sport 1100 -m state --state ESTABLISHED -j ACCEPT

/etc/init.d/iptables restart


service iptables restart

Next i would like to get your attention on a draw back in JNDI failover and how to overcome it. Say for example you have two servers named serverA and serverB, If serverA shuts down then serverB should handle the whole load. This works out of the box from JBoss. But if serverA starts again and serverB goes down then you are screwed because you will be getting an error as java.lang.RuntimeException: Unreachable?: Service unavailable.

To overcome this there is a work around. Open the file standardjboss.xml locate in the all/conf directory. Within that file locate the text clustered-stateless-rmi-invoker. Now within that you will see number of elements. In one of these you will find something called org.jboss.proxy.ejb.SingleRetryInterceptor. You will find two instances of this. You need to change it to org.jboss.proxy.ejb.RetryInterceptor. What this does is it will keep retrying to connect infinitely until the other server comes up. Ofcourse this is not a perfect solution but just a work around. Note that this is with respect to Jboss 4.2.3 that i am talking about. This might have been fixed in later releases which im not aware of.

So the change will look as below;

<invoker-proxy-binding>
      <name>clustered-stateless-rmi-invoker</name>
      <invoker-mbean>jboss:service=invoker,type=jrmpha</invoker-mbean>
      <proxy-factory>org.jboss.proxy.ejb.ProxyFactoryHA</proxy-factory>
      <proxy-factory-config>
        <client-interceptors>
          <home>
            <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
            <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
            <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
            <interceptor call-by-value="false">org.jboss.invocation.InvokerInterceptor</interceptor>
            <interceptor call-by-value="true">org.jboss.invocation.MarshallingInvokerInterceptor</interceptor>
          </home>
          <bean>
            <interceptor>org.jboss.proxy.ejb.StatelessSessionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
            <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
            <interceptor call-by-value="false">org.jboss.invocation.InvokerInterceptor</interceptor>
            <interceptor call-by-value="true">org.jboss.invocation.MarshallingInvokerInterceptor</interceptor>
          </bean>
        </client-interceptors>
      </proxy-factory-config>
    </invoker-proxy-binding> 

Lastly i present to you the jboss_redhat_init.sh file which sets the jboss partition name to be used across the application which is the same name used in the @Clustered annotation.

#!/bin/sh
#
# $Id: jboss_init_redhat.sh 71252 2008-03-25 17:52:00Z dbhole $
#
# JBoss Control Script
#
# To use this script run it as root - it will switch to the specified user
#
# Here is a little (and extremely primitive) startup/shutdown script
# for RedHat systems. It assumes that JBoss lives in /usr/local/jboss,
# it's run by user 'jboss' and JDK binaries are in /usr/local/jdk/bin.
# All this can be changed in the script itself. 
#
# Either modify this script for your requirements or just ensure that
# the following variables are set correctly before calling the script.

#define where jboss is - this is the directory containing directories log, bin, conf etc
JBOSS_HOME=${JBOSS_HOME:-"/opt/xyz/jboss-4.2.3.GA"}

#define the user under which jboss will run, or use 'RUNASIS' to run as the current user
JBOSS_USER=${JBOSS_USER:-"RUNASIS"}

#make sure java is in your path
JAVAPTH=${JAVAPTH:-"/usr/java/jdk1.6.0_16/bin"}

#configuration to use, usually one of 'minimal', 'default', 'all'
JBOSS_CONF=${JBOSS_CONF:-"all"}

#if JBOSS_HOST specified, use -b to bind jboss services to that address
JBOSS_BIND_ADDR=${JBOSS_HOST:+"-b $JBOSS_HOST"}

#define the script to use to start jboss
JBOSSSH=${JBOSSSH:-"$JBOSS_HOME/bin/run.sh -b 172.2.23.2 -c $JBOSS_CONF $JBOSS_BIND_ADDR -Djboss.partition.name=mypartition"}

if [ "$JBOSS_USER" = "RUNASIS" ]; then
  SUBIT=""
else
  SUBIT="su - $JBOSS_USER -c "
fi

if [ -n "$JBOSS_CONSOLE" -a ! -d "$JBOSS_CONSOLE" ]; then
  # ensure the file exists
  touch $JBOSS_CONSOLE
  if [ ! -z "$SUBIT" ]; then
    chown $JBOSS_USER $JBOSS_CONSOLE
  fi 
fi

if [ -n "$JBOSS_CONSOLE" -a ! -f "$JBOSS_CONSOLE" ]; then
  echo "WARNING: location for saving console log invalid: $JBOSS_CONSOLE"
  echo "WARNING: ignoring it and using /dev/null"
  JBOSS_CONSOLE="/dev/null"
fi

#define what will be done with the console log
JBOSS_CONSOLE=${JBOSS_CONSOLE:-"/dev/null"}

JBOSS_CMD_START="cd $JBOSS_HOME/bin; $JBOSSSH"

if [ -z "`echo $PATH | grep $JAVAPTH`" ]; then
  export PATH=$PATH:$JAVAPTH
fi

if [ ! -d "$JBOSS_HOME" ]; then
  echo JBOSS_HOME does not exist as a valid directory : $JBOSS_HOME
  exit 1
fi

echo JBOSS_CMD_START = $JBOSS_CMD_START

function procrunning() {
   procid=0
   JBOSSSCRIPT=$(echo $JBOSSSH | awk '{print $1}' | sed 's/\//\\\//g')
   for procid in `/sbin/pidof -x "$JBOSSSCRIPT"`; do
       ps -fp $procid | grep "${JBOSSSH% *}" > /dev/null && pid=$procid
   done
}


stop() {
    pid=0
    procrunning
    if [ $pid = '0' ]; then
        echo -n -e "\nNo JBossas is currently running\n"
        exit 1
    fi

    RETVAL=1

    # If process is still running

    # First, try to kill it nicely
    for id in `ps --ppid $pid | awk '{print $1}' | grep -v "^PID$"`; do
       if [ -z "$SUBIT" ]; then
           kill -15 $id
       else
           $SUBIT "kill -15 $id"
       fi
    done

    sleep=0
    while [ $sleep -lt 120 -a $RETVAL -eq 1 ]; do
        echo -n -e "\nwaiting for processes to stop";
        sleep 10
        sleep=`expr $sleep + 10`
        pid=0
        procrunning
        if [ $pid == '0' ]; then
            RETVAL=0
        fi
    done

    # Still not dead... kill it

    count=0
    pid=0
    procrunning

    if [ $RETVAL != 0 ] ; then
        echo -e "\nTimeout: Shutdown command was sent, but process is still running with PID $pid"
        exit 1
    fi

    echo
    exit 0
}

case "$1" in
start)
    cd $JBOSS_HOME/bin
    if [ -z "$SUBIT" ]; then
        eval $JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &
    else
        $SUBIT "$JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &" 
    fi
    ;;
stop)
    stop
    ;;
restart)
    $0 stop
    $0 start
    ;;
*)
    echo "usage: $0 (start|stop|restart|help)"
esac

The part to note here is -Djboss.partition.name=mypartition which sets the name of the cluster as an application wide property.

Thats it guys. If you have any issues or improvements please leave a comment which is as always much appreciated.

Its time for some code review

I know some of us just hate it when we hear the word code review. The thought of some one else going through our code is somewhat offensive to some people. I myself have come across many. But for me this is a good thing given that my reviewer knows what to look for and have a set of parameters set out as a base for him/her to review on. So in this article i would like to highlight a few mistakes i have come across when doing code reviews. Most of them are common but we tend to often miss out on those. This is not the end of them. So if you guys have come across any thing other than what is highlighted here please do leave a comment as it would be beneficial for me as well as the other readers.

Integer myVal = new Integer(10);

Ofcourse by the look of this code there is nothing intrinsically wrong with it. The thing is in this code snippet you create an instance of a new Integer wrapper class. You can enhance this as follows;

Integer myVal = Integer.valueOf(10);

Using the valueOf method is better performance wise as this method returns the value from the in memory cache rather than creating a new object. Check here for what the java api says in this regard.

String records = "";

        for (int i = 0; i < 10; i++) {
            records+="new record"+i;
        }

Ok the issue here is as some of you already might have guessed is string concatenation. Though this works there is an issue with performance with respect to string concatenation. For those of you who do not know, when you concatenate two strings what really happens internally is a new string buffer is created and the new string is assigned to it and toString() is called on that string buffer object. Now this happens on every loop call. So you can understand the performance impact this will have.

Ofcourse prior to java 1.6 you would have argued about so many objects being created on the heap as a result of this. But with java 1.6 escape analysis, any object created within the scope of a method is created on the stack and not on the heap. And hence that argument is invalid. So a better way to handle this situation is to use StringBuilder as follows;

StringBuilder records = new StringBuilder(100);

        for (int i = 0; i < 10; i++) {
            records.append("new record").append(i);
        }

Note that i have also specified an initial size to the String Builder within the constructor. We do this for efficiency because if you do not define an initial size the default size of 16 characters is given by default. Whenever your buffer exceeds this limit a new StringBuilder object is created and the whole String that was in the old buffer is copied to the new one. To avoid such transactions taking place we give an initial capacity that we deem appropriate.

int i=0;
       .......
       .......
       .......
       .......
       .......
       .......
       .......
       .......
       
       i = 2+4;

This is a common mistake i see many people make. You define a variable at one point and afterwards you do various other method calls and at a latter point in that method you manipulate that variable you defined early on in the method. As a best practice always use the variable as soon as it is defined. This provides much clarity to your code and keeps it clean.

Using the same example i would like to point out another thing. See that the code above defines an integer called "i". This is ok to use within a for loop but if this is in the context of a method then always use meaningful names as this will self describe your code minimizing the code commenting you need to do.

/**
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * Blah blah blah blah blah blah blah
     * 
     */
    public void myTestMethod() {
        
    }

Please stop writing lengthy code comments. In my point of view is this is just an utter waste of time. The fact that you need to write such a lengthy explanation to me disseminates the fact that this method does more than what it should do. Write a short description on what your method does. A simple explanation of business logic is more than enough. Make your code readable which reduces the time you have to comment.

/**
     * Sets the name
     * @param name
     */
    public void setMyName(String name) {
        this.name = name;
    }

Wow what a code comment. The author says it sets the name variable. Wow really? Thank goodness for that comment else i would not have known what was going on in that method :) . Seriously stop making useless code comments. It just clutters your code and does no use for anyone reading it. Dont comment on getter/setter methods as the name it self implies what is happening.

public void doProcess()throws MyServiceException {
        
        try {
            service.doProcess();
        }catch(MyServiceException e) {
            log.error("Error occured");
            throw e;
        }
    }

Ok whats wrong here? Problem here is that this method catches a MyServiceException and logs the error and rethrows the same exception back from the method. Why would you catch an exception if you are going to rethrow the same? It just does not make sense. Proper way to handle such a thing is to let the caller handle the exception and log the error at that end rather than you trying to handle it. Anyway it is recommended now to throw Runtime exceptions rather than checked exceptions if the error you throw is like a DB exception or IO exception as these kind of exceptions does not make sense to the client.

public void doProcess()throws MyServiceException {
        
            log.debug("Started method");
            service.doProcess();
            log.debug("Method processed");


    }

Here the issue lies in the debug log statement. Often times in a production environment your log level will mostly be set to INFO or ERROR. But when your code has debug logs the Logger will try to process this which results in unnecessary executions. You can avoid this by checkin if debug is enabled as shown in the below code snippet;

public void doProcess() throws MyServiceException {

        if (log.isDebugEnabled()) {
            log.debug("Started method");
        }
        service.doProcess();
        if (log.isDebugEnabled()) {
            log.debug("Method processed");
        }

    }

if(server.equals("TEST")) {
            //do something
        }else if(server.equals("TEST123")) {
            //do something
        }else if(server.equals("TEST1234")) {
            //do something
        }else if(server.equals("TEST12345")) {
            //do something
        }else if(server.equals("TEST123456")) {
            //do something
        }

In an instance where you are tangled in multiple if/else statement it is always much cleaner to use switch statements and use enums to hold the constants.With JDK 1.7 you are allowed to use Strings in a switch statement. So your code above will look like this with use of enums and switch statement;

switch (server) {
        case TEST:
            // do something
            break;

        case TEST123:
            // do something
            break;
        case TEST1234:
            // do something
            break;
        case TEST12345:
            // do something
            break;
        case TEST123456:
            // do something
            break;
        }

Those are few of the things that came to my mind at the time of writing this post. Ofcourse there are many more which i will follow in subsequent posts. Pls do leave a comment which will be an additions to this post that you deem appropriate for a code review session.


Cheers Guys

Time to get rid of EARs

We all know what EARs(Enteprise Archive) are. When talking about enterprise software this is a common term used where we bundle up our application with all the required JAR(Java Archive) files needed by our application at runtime. This has been done for many years and people have adapted to it. Bundling up WARs(Web Archive) inside EARs is a common practice too.

In the initial stage even our current application was structured in such a way that in the end an EAR is deployment to our Application server along with the configs needed. Note that i am talking with respective to JBoss Application Server 4.2.3 ( Refereed as Jboss from here onwards). Our application is hosted within a Jboss A/S and hence it goes into the server/default folder in development.

The problem we faced was when we were doing a production release. Our servers were remotely hosted at a local internet service provider's(ISP) data warehouse. And our typical EAR came to about 40MB in size which took about 45min - 1 hr to copy to the remote servers. This posed a problem and a probable bottleneck in the overall process of pushing a build as i saw it. So i was looking at any other way we can achieve some efficiency in this regard.

The solution i came up with was influenced by Julius Ceasar many years ago when He devised the Divide and Conquer Mechanism. Our problem as i saw was that our application code was only a mere 5-8MB where as the dependent JAR files amounted around 30MB in size.

So what i did was i took our the EAR building(Note that we are using Maven as our build tool) out of the process and bundled up our application as shown in the diagram below. Note that the diagram below depicts my approach to the problem at hand.

Note that now as we have separated the thirdparty JARs into a separate JAR file we only deploy this once and whenever a new dependency adds up (which is rare because the project is matured into a stable position). Now we only have to deploy the Other JAR files including the WAR which is only around 8-10MB of size overall. So overall our deployment time is now reduced to 10-15 mins which took almost an hour before with the EAR approach. I wrote a separate assembly descriptor to bundle up the required JAR files in our project into a single JAR file. If any of you guys want this i can share the assembly descriptor file i wrote.

One downside i have to note with this approach is that we cannot use JPA as JPA requires you to have a persistence.xml inside your EARs/WARs META-INF directory. But still it might work if we put the persistence.xml inside the domain/dto jar file but i have not tested this. But this was not an issue for us because we initially decided to go with Hibernate using Spring's Hibernate Template as the abstraction layer.

With the advent of jee6 now you can have your EJBs running inside your WAR. But the application server i have talked about here does not support jee6 and hence this is the only viable solution i could come up with.

That's about it. I do not know how this will scale with other application servers as i can only guarantee this working in JBoss. But for me its time to say good by to EARs which is too bloated as per my experience.

Twitter Based Alert System

We were in need of an alert system for our system in order to notify relevant parties of any errors which would occur in our production servers. There was already an email alert system in place but hey you have to admit no one is going to be checking emails 24x7 yea? :) ... We needed a solution like getting an SMS to our support team's phone, but we didnt have any SMS gateway available.

So thinking in the same lines i came up with a solution which enabled us to receive an SMS whenever an error occurred in our production systems. The solution was not a complex one and was very easy to implement with Spring 3 support.

The implemented solution was first we created a twitter account for our application and made the tweets private and followers to be authenticated. We dont want the whole world to know of the errors in our system now do we :) ... Then there was a very comprehensive tutorial available in Spring which you can find here which shows how to easily publish messages to your twitter account using Spring Integration. I honestly felt it was very easy to adapt with minimal changes.

Afterwards we created a twitter account for our support staff and followed the twitter account created for the application and enabled mobile alerts for that account. So now whenever an error occurs the application publishes the message via Spring's twitter plugin and consequently the support staff receives an update to their phones because we enabled mobile alerts on that specific account.

Now we have a fully fledged SMS based alert system without any cost involved. Someone somewhere said the best solution was almost always the simplest solution, i got to admit they were right on the money on that..

Cheers folks!!!!