Wednesday, 14 November 2012

Setting up a Jboss 7.1 Cluster

Sadly, setting up clustering in jboss 7.1.1 is not as straight forward as it should be.  After hours of googling and numerous false starts, I finally got it all working.  Here are some of the gotcha's which tripped me up.

My starting assumption is that we're running a domain.  Out of the box, the Master node defines 3 servers:
 Server-1    - main-server-group
 Server-2    - other-server-group
 Server-3    - main-server-group

Server-1 and Server-2 are configured to start automatically.  In a cluster these are of no use - disable them.  We're only interested in Server-3 which uses the full-ha profile.

Before we go on, a quick heads up:

  • You only need to amend the domain.xml on the master node
Some helpfull links:


Gotcha #1

When you start up the nodes using the domain.bat scripts, always provide the bind ip and ports.  If you don't then you seem to get some strange behaviours:

domain.sh-b 192.168.0.24 -bmanagement 192.168.0.24     (this is the master)
and
domain.sh-b 192.168.0.25 -bmanagement 192.168.0.25


Gotcha #2

Server-3 does not start cleanly out of the box, due to a mismatch between the full-ha progile and the socket bindings.  In domain.xml, amend as follows:


<server-group name="other-server-group" profile="full-ha">
  <jvm name="default"
    <heap size="64m" max-size="512m"/>
  </jvm>
  <socket-binding-group ref="full-ha-sockets"/>
</server-group>


This will allow Server-2 to start cleanly.

Gotcha #3

The hornetq-server needs configuring with a user name and password.  If you don't you will get an exception along the lines of:

[Server:server-three] 16:24:36,875 ERROR [org.hornetq.core.protocol.core.impl.Ho
rnetQPacketHandler] (Old I/O server worker (parentId: 2128830893, [id: 0x7ee361a
d, /192.168.0.24:5695])) Failed to create session : HornetQException[errorCode=1
05 message=Unable to validate user: HORNETQ.CLUSTER.ADMIN.USER]

Update domain.xml with:
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
  <hornetq-server>                
  <cluster-user>fred</cluster-user>
  <cluster-password>password</cluster-password>


It looks as though you can use any user and password.  It doesn't appear to be authenticated against anything!

Gotcha #4

Use the correct version of @Clustered annotation!   If you don't then in a failover situation you will get the "No EJBReceiver" exception.  The version of @Clustered your'e after is org.jboss.ejb3.annotation.Clustered.  This comes from the jboss-ejb3-ext-api dependency, which is only available from the jboss repo.  Assuming that you are using maven add the following to the pom.xml:

<dependencies>
 ...

  <dependency>
    <groupId>org.jboss.ejb3</groupId>
    <artifactId>jboss-ejb3-ext-api</artifactId>
    <version>2.0.0-beta-3</version>
    <scope>provided</scope>
  </dependency>

<dependencies>
...
<repositories>
  <repository>
    <id>JBoss repository</id>
    <url>http://repository.jboss.org/nexus/content/groups/public/</url>
  </repository>
<repositories>

Then correctly annotate the bean:

import org.jboss.ejb3.annotation.Clustered;

@Stateful
@Clustered
@Remote( coder36.demo.service.MyService.class )
public class MyServiceImpl implements MyService {
...

Gotcha #5

When injecting the EJB do not use the @Inject annotation!   Use @EJB on it's own.  No JNDI name is required:


@ManagedBean
public class PersonController implements Serializable {
  @EJB
  private MyService myService;


Note - the @EJB annotation is available via the following dependency:


<dependency>
  <groupId>org.jboss.spec.javax.ejb</groupId>
  <artifactId>jboss-ejb-api_3.1_spec</artifactId>
</dependency>


Gotcha #6

To make the web application cluster-able ensure that webapp/WEB-INF/web.xml exists, and has the <distributable/> tag:


<web-app xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
    http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
    version="3.0"> 

  <distributable/> 
</web-app>


To be extra helpful, I've provided the web.xml namespaces :)

Gotcha #7

Assuming you have downloaded and installed mod_cluster 1,2.Final from here.

Out of the box, mod_cluster does't  not work with jboss 7.1.1. A few tweaks are needed.  Keeping things as simple as possible, I've stripped the provided httpd.conf down the bare minimum:


LoadModule authz_host_module modules/mod_authz_host.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule manager_module modules/mod_manager.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule advertise_module modules/mod_advertise.so

Listen 192.168.0.24:80
Listen 192.168.0.24:10001

<VirtualHost 192.168.0.24:10001>
  <Location />
    Order deny,allow
    Deny from all
    Allow from all
  </Location>

  KeepAliveTimeout 300
  MaxKeepAliveRequests 0

  EnableMCPMReceive

  <Location /mod_cluster_manager>
    SetHandler mod_cluster-manager
    Order deny,allow
    Deny from all
    Allow from all
  </Location>
</VirtualHost>
ErrorLog "logs/error_log"


I used port 10001 as the default port 6666 didn't work for me.  This httpd.conf should be enough to get you going.

The domain.xml also needs updating to link it up with the mod_cluster instance:


<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
  <mod-cluster-config advertise-socket="modcluster" 
                      proxy-list="192.168.0.24:10001">
  ...


and

<subsystem xmlns="urn:jboss:domain:web:1.1" 
            default-virtual-server="default-host" 
            instance-id="${jboss.node.name}" 
            native="false">
...


Testing

To test everything is working, fire up the domain, deploy an application and start httpd (on windows 7 I needed to run it as administrator otherwise it complained about file permissions).  Navigate to:  http://192.168.0.24:10001/mod_cluster_manager.  You should see:



The web application can be now accessed through the mod_cluster proxy sitting on http://192.168.0.24 and http://192.168.0.24:10001:

ie. http://192.168.0.24/demo/index/jsf

10 comments:

  1. Mark,

    I was struggling for a while to get mod cluster working with JBOSS AS 7.1.1. After reading ur post, I was able to fix the issue straight away.

    Thanks a lot..

    ReplyDelete
  2. I setup everything accord to what you have but I have to comment out this line:
    #Listen 10.0.0.24:80
    Listen 10.0.0.24:10001
    to make httpd runs. I am not sure why. Can you please help see what's I did wrong, here are my httpd.conf stuffs:
    #Listen 10.0.0.24:80
    Listen 10.0.0.24:10001




    Order deny,allow
    Deny from all
    Allow from all



    # This directive allows you to view mod_cluster status at URL http://10.0.0.24:10001/mod_cluster-manager

    SetHandler mod_cluster-manager
    Order deny,allow
    Deny from all
    Allow from all


    KeepAliveTimeout 60
    MaxKeepAliveRequests 0
    EnableMCPMReceive
    # ManagerBalancerName iriocluster
    AdvertiseFrequency 5

    ReplyDelete
  3. Perhaps port 80 is already in use, or reserved in some way by the OS (you may need to run as root to use it).

    Try and narrow down your problem - try another port, see if that works. On Ubuntu you can use:
    sudo netstat -anltp to view open ports. On windows its: netstat -a.

    ReplyDelete
  4. Thanks a lot for this post. I was able to successfully setup my jboss as 7 cluster.

    Cheers

    Parag

    ReplyDelete
  5. The only issue I am facing is that my slave is unable to deploy the war and says that it could not find the datasource. I have created the datasource under full-ha profile and master is able to access it seamlessly. Can you please help?

    ReplyDelete
  6. A few things to try:

    1) If you are deploying the JDBC driver as a module (under the /modules folder), then you will need to make sure that you physically copy the module to each of your slaves/nodes.

    2) The jboss web console is not particularly trust worthy and sometimes gets confused. Double check your setup via jboss-cli.

    3) Make sure you are deploying your war file in the same server group as your datasource.

    4) Try deploying a type 4 jdbc driver (deploy it as you would a war file) and recreate your datasource binding to this driver.

    ReplyDelete
    Replies
    1. Thanks Mark! yes I was able to get the datasource correctly after I deployed the mysql module on slave server. Now checking the session replication :) .. hope I will be able to crack that as it was giving error in acquiring lock issue. Thanks once again for your help. Its really helpful.

      Delete
  7. This comment has been removed by the author.

    ReplyDelete
  8. hi mark,
    i done the configuration as mentioned, After starting the server am getting the error like

    org.jboss.modcluster.mcmp.impl.Default
    MCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [null: null: {4}] sending command INFO to proxy ip:port , configuration will be reset

    continuously.
    please help to solve this issue

    thanks

    ReplyDelete