JBoss clustering Quick Reference Guide

So i was involved in clustering two JBoss instances and for some reason i had lost my guide i made for myself last time i did a clustering implementation. This time i thought ill blog it so as to keep it as a reference for myself as well as anyone who might need to quickly do JBoss Clustering as we all know how time critical all IT projects are. I will put this in a step by step approach to make it easier to read and comprehend. Note that this approach assumes you are clustering the all configuration in JBoss.

• Locate the cluster-service.xml located within the deploy folder. Open it and locate the attribute PartitionName within the first mbean you see in that file and change the value ${jboss.partition.name:DefaultPartition} to ${jboss.partition.name}. As in delete the part DefaultParition. 
•  Find the mbean code which equals to org.jboss.ha.jndi.HANamingService and org.jboss.ha.hasessionstate.server.HASessionStateService. Within that locate the tag   . Chane the value jboss:service=${jboss.partition.name:DefaultPartition} to jboss:service=${jboss.partition.name}. Also add the following to both the mbeans mentioned above ${jboss.partition.name}.

• In your application jndi reference make sure to change the port to 1100 because in a clustered environment HA-JNDI(High availability JNDI) is used to look up resource. So localhost:1099 should be localhost:1100. Also you have to give the jndi in a comma separated way indicating all servers being clustered. This is needed in a case where if one server goes down still look ups will not fail as you have specified all servers. Say for example you have two servers named ABC and XYZ. Then if you only specify your jndi url as ABC:1100, this will work but in the event that ABC goes down all your other look ups will also fail. So to avoid that kind of situations make sure to define it as ABC:1100,XYZ:1100. This will make sure to try the other server in the case one fails.
• In all your EJBs make sure to put the Annotation @Clustered(partition="mypartition"). The name we specify should be specified in your run script which i will give in the end of this article.
• If you running on Redhat make sure the HA-JNDI port is open by issuing the following commands to open port 1100;
  
  
  
  
   

iptables -A INPUT -i eth0 -p tcp --sport 1100 -m state --state ESTABLISHED -j ACCEPT

/etc/init.d/iptables restart


service iptables restart

Next i would like to get your attention on a draw back in JNDI failover and how to overcome it. Say for example you have two servers named serverA and serverB, If serverA shuts down then serverB should handle the whole load. This works out of the box from JBoss. But if serverA starts again and serverB goes down then you are screwed because you will be getting an error as java.lang.RuntimeException: Unreachable?: Service unavailable.

To overcome this there is a work around. Open the file standardjboss.xml locate in the all/conf directory. Within that file locate the text clustered-stateless-rmi-invoker. Now within that you will see number of elements. In one of these you will find something called org.jboss.proxy.ejb.SingleRetryInterceptor. You will find two instances of this. You need to change it to org.jboss.proxy.ejb.RetryInterceptor. What this does is it will keep retrying to connect infinitely until the other server comes up. Ofcourse this is not a perfect solution but just a work around. Note that this is with respect to Jboss 4.2.3 that i am talking about. This might have been fixed in later releases which im not aware of.

So the change will look as below;

<invoker-proxy-binding>
      <name>clustered-stateless-rmi-invoker</name>
      <invoker-mbean>jboss:service=invoker,type=jrmpha</invoker-mbean>
      <proxy-factory>org.jboss.proxy.ejb.ProxyFactoryHA</proxy-factory>
      <proxy-factory-config>
        <client-interceptors>
          <home>
            <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
            <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
            <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
            <interceptor call-by-value="false">org.jboss.invocation.InvokerInterceptor</interceptor>
            <interceptor call-by-value="true">org.jboss.invocation.MarshallingInvokerInterceptor</interceptor>
          </home>
          <bean>
            <interceptor>org.jboss.proxy.ejb.StatelessSessionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
            <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
            <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
            <interceptor call-by-value="false">org.jboss.invocation.InvokerInterceptor</interceptor>
            <interceptor call-by-value="true">org.jboss.invocation.MarshallingInvokerInterceptor</interceptor>
          </bean>
        </client-interceptors>
      </proxy-factory-config>
    </invoker-proxy-binding> 

Lastly i present to you the jboss_redhat_init.sh file which sets the jboss partition name to be used across the application which is the same name used in the @Clustered annotation.

#!/bin/sh
#
# $Id: jboss_init_redhat.sh 71252 2008-03-25 17:52:00Z dbhole $
#
# JBoss Control Script
#
# To use this script run it as root - it will switch to the specified user
#
# Here is a little (and extremely primitive) startup/shutdown script
# for RedHat systems. It assumes that JBoss lives in /usr/local/jboss,
# it's run by user 'jboss' and JDK binaries are in /usr/local/jdk/bin.
# All this can be changed in the script itself. 
#
# Either modify this script for your requirements or just ensure that
# the following variables are set correctly before calling the script.

#define where jboss is - this is the directory containing directories log, bin, conf etc
JBOSS_HOME=${JBOSS_HOME:-"/opt/xyz/jboss-4.2.3.GA"}

#define the user under which jboss will run, or use 'RUNASIS' to run as the current user
JBOSS_USER=${JBOSS_USER:-"RUNASIS"}

#make sure java is in your path
JAVAPTH=${JAVAPTH:-"/usr/java/jdk1.6.0_16/bin"}

#configuration to use, usually one of 'minimal', 'default', 'all'
JBOSS_CONF=${JBOSS_CONF:-"all"}

#if JBOSS_HOST specified, use -b to bind jboss services to that address
JBOSS_BIND_ADDR=${JBOSS_HOST:+"-b $JBOSS_HOST"}

#define the script to use to start jboss
JBOSSSH=${JBOSSSH:-"$JBOSS_HOME/bin/run.sh -b 172.2.23.2 -c $JBOSS_CONF $JBOSS_BIND_ADDR -Djboss.partition.name=mypartition"}

if [ "$JBOSS_USER" = "RUNASIS" ]; then
  SUBIT=""
else
  SUBIT="su - $JBOSS_USER -c "
fi

if [ -n "$JBOSS_CONSOLE" -a ! -d "$JBOSS_CONSOLE" ]; then
  # ensure the file exists
  touch $JBOSS_CONSOLE
  if [ ! -z "$SUBIT" ]; then
    chown $JBOSS_USER $JBOSS_CONSOLE
  fi 
fi

if [ -n "$JBOSS_CONSOLE" -a ! -f "$JBOSS_CONSOLE" ]; then
  echo "WARNING: location for saving console log invalid: $JBOSS_CONSOLE"
  echo "WARNING: ignoring it and using /dev/null"
  JBOSS_CONSOLE="/dev/null"
fi

#define what will be done with the console log
JBOSS_CONSOLE=${JBOSS_CONSOLE:-"/dev/null"}

JBOSS_CMD_START="cd $JBOSS_HOME/bin; $JBOSSSH"

if [ -z "`echo $PATH | grep $JAVAPTH`" ]; then
  export PATH=$PATH:$JAVAPTH
fi

if [ ! -d "$JBOSS_HOME" ]; then
  echo JBOSS_HOME does not exist as a valid directory : $JBOSS_HOME
  exit 1
fi

echo JBOSS_CMD_START = $JBOSS_CMD_START

function procrunning() {
   procid=0
   JBOSSSCRIPT=$(echo $JBOSSSH | awk '{print $1}' | sed 's/\//\\\//g')
   for procid in `/sbin/pidof -x "$JBOSSSCRIPT"`; do
       ps -fp $procid | grep "${JBOSSSH% *}" > /dev/null && pid=$procid
   done
}


stop() {
    pid=0
    procrunning
    if [ $pid = '0' ]; then
        echo -n -e "\nNo JBossas is currently running\n"
        exit 1
    fi

    RETVAL=1

    # If process is still running

    # First, try to kill it nicely
    for id in `ps --ppid $pid | awk '{print $1}' | grep -v "^PID$"`; do
       if [ -z "$SUBIT" ]; then
           kill -15 $id
       else
           $SUBIT "kill -15 $id"
       fi
    done

    sleep=0
    while [ $sleep -lt 120 -a $RETVAL -eq 1 ]; do
        echo -n -e "\nwaiting for processes to stop";
        sleep 10
        sleep=`expr $sleep + 10`
        pid=0
        procrunning
        if [ $pid == '0' ]; then
            RETVAL=0
        fi
    done

    # Still not dead... kill it

    count=0
    pid=0
    procrunning

    if [ $RETVAL != 0 ] ; then
        echo -e "\nTimeout: Shutdown command was sent, but process is still running with PID $pid"
        exit 1
    fi

    echo
    exit 0
}

case "$1" in
start)
    cd $JBOSS_HOME/bin
    if [ -z "$SUBIT" ]; then
        eval $JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &
    else
        $SUBIT "$JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &" 
    fi
    ;;
stop)
    stop
    ;;
restart)
    $0 stop
    $0 start
    ;;
*)
    echo "usage: $0 (start|stop|restart|help)"
esac

The part to note here is -Djboss.partition.name=mypartition which sets the name of the cluster as an application wide property.

Thats it guys. If you have any issues or improvements please leave a comment which is as always much appreciated.