Creating WSO2 ESB Cluster with Nginx LB

This article is about creating WSO2 ESB Cluster setup (1 Manager, 2 Workers) with Nginx Load Balancer.

If you want to create same type of cluster without using load balancer, refer blog post [1] where Ravindra has explained it in detail.

WSO2 ESB Cluster will be created according to the below setup. There will be a manager node with two worker nodes.

Ports Configuration Summary

Following port configurations will be used in manager and worker nodes in the cluster.

Configuration	Manager	Worker1	Worker2
offset	1	2	3
localMemberPort	4100	4200	4300
nhttp	8281	8282	8283
nhttps	8244	8245	8246

Extract ESB packs in Manager, Worker1 and Worker2 locations and edit configurations as follows;

Setting up user management and registry databases

1. Open a mysql shell and create a database as follows

mysql> create database wso2conum_db;

2. Create database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory

mysql> use wso2conum_db;

mysql> source CARBON_HOME/dbscripts/mysql.sql;

3. Now, create another database which will be used as the shared governance and configuration registry database.

mysql> create database wso2conreg_db;

4. Create registry database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory

mysql> use wso2conreg_db;

mysql> source CARBON_HOME/dbscripts/mysql.sql;

Update SVNKit libraries

In order to work with Deployment Synchronization[2], we need to add SVNKit jar files as follows to the ESB runtime in all managers and workers.

1. Download SVNKit from http://product-dist.wso2.com/tools/svnkit-all-1.8.7.wso2v1.jar and put into the <ESB_HOME>/repository/components/dropins folder.
2. Download http://maven.wso2.org/nexus/content/groups/wso2-public/com/trilead/trilead-ssh2/1.0.0-build215/trilead-ssh2-1.0.0-build215.jar and put into <ESB_HOME>/repository/components/lib folder.

Configure the Manager node

1. Unzip the WSO2 ESB binary distribution. Consider the extracted directory as <ESB_HOME>.

2. Configure user management database and shared registry database by editing <WSO2_ESB_MGR_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<datasource>

<name>WSO2_SHARED_REG_DB</name>

<description>The datasource used for shared config and governance registry</description>

<jndiConfig>

<name>jdbc/WSO2SharedDB</name>

</jndiConfig>

<definition type="RDBMS">

<configuration>

<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>

<username>root</username>

<password>root</password>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<testOnBorrow>true</testOnBorrow>

<validationQuery>SELECT 1</validationQuery>

<validationInterval>30000</validationInterval>

</configuration>

</definition>

</datasource>

<datasource>

<name>WSO2_UM_DB</name>

<description>The datasource used for registry and user manager</description>

<jndiConfig>

<name>jdbc/WSO2UmDB</name>

</jndiConfig>

<definition type="RDBMS">

<configuration>

<url>jdbc:mysql://localhost:3306/wso2conum_db</url>

<username>root</username>

<password>root</password>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<testOnBorrow>true</testOnBorrow>

<validationQuery>SELECT 1</validationQuery>

<validationInterval>30000</validationInterval>

</configuration>

</definition>

</datasource>

Make sure to replace username and password with your mysql database username and password

3. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.

<Property name="dataSource">jdbc/WSO2UmDB</Property>

4. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.

<dbConfig name="sharedregistry">

<dataSource>jdbc/WSO2SharedDB</dataSource>

</dbConfig>

<remoteInstance url="https://localhost:9444/registry">

<id>instanceid</id>

<dbConfig>sharedregistry</dbConfig>

<readOnly>false</readOnly>

<enableCache>true</enableCache>

<registryRoot>/</registryRoot>

</remoteInstance>

<mount path="/_system/config" overwrite="true">

<instanceId>instanceid</instanceId>

<targetPath>/_system/esbnodes</targetPath>

</mount>

<mount path="/_system/governance" overwrite="true">

<instanceId>instanceid</instanceId>

<targetPath>/_system/governance</targetPath>

</mount>

5. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to <ESB_HOME>

/repository/components/lib

6. Setting up the cluster configurations edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

<parameter name="membershipScheme">wka</parameter>

<parameter name="domain">wso2.esb.domain</parameter>

<parameter name="localMemberHost">mgt.esb.wso2.com</parameter>

<parameter name="localMemberPort">4100</parameter>
........

7. Specify the well known member.Here, the well known member is a worker1 node

<members>

 <member>

       <hostName>esb.wso2.com</hostName>

       <port>4200</port>     

 </member>

</members>

8. Configuring the port offset and host name, edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;

<Offset>1</Offset>

<HostName>esb.wso2.com</HostName>

<MgtHostName>mgt.esb.wso2.com</MgtHostName>

9. Enable SVN based deployment synchronizer with AutoCommit property marked ‘true’.

Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows

<DeploymentSynchronizer>

      <Enabled>true</Enabled>

      <AutoCommit>true</AutoCommit>

      <AutoCheckout>true</AutoCheckout>

      <RepositoryType>svn</RepositoryType>

      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>

      <SvnUser>svnuser</SvnUser>

      <SvnPassword>xxxxxx</SvnPassword>

      <SvnUrlAppendTenantId>true</SvnUrlAppen dTenantId>

</DeploymentSynchronizer>

If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

10. Mapping the host names to the IP. Add the below host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the hostnames with the IP address of the Load balancer machine.

<IP-of-LB> esb.wso2.com

<IP-of-LB> mgt.esb.wso2.com

11. Allow access the management console only through LB, configure the HTTP proxy ports to communicate through the load balancer.

12. Edit “<ESB_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows

<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"

             port="9763"

             proxyPort="80"

Configure the worker node

1. Unzip the WSO2 ESB binary distribution

2. Configure user management database and shared registry database by editing

<ESB_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<datasource>

<name>WSO2_SHARED_REG_DB</name>

<description>The datasource used for shared config and governance registry</description>

<jndiConfig>

<name>jdbc/WSO2SharedDB</name>

</jndiConfig>

<definition type="RDBMS">

<configuration>

<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>

<username>root</username>

<password>root</password>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<testOnBorrow>true</testOnBorrow>

<validationQuery>SELECT 1</validationQuery>

<validationInterval>30000</validationInterval>

</configuration>

</definition>

</datasource>

<datasource>

<name>WSO2_UM_DB</name>

<description>The datasource used for registry and user manager</description>

<jndiConfig>

<name>jdbc/WSO2UmDB</name>

</jndiConfig>

<definition type="RDBMS">

<configuration>

<url>jdbc:mysql://localhost:3306/wso2conum_db</url>

<username>root</username>

<password>root</password>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<testOnBorrow>true</testOnBorrow>

<validationQuery>SELECT 1</validationQuery>

<validationInterval>30000</validationInterval>

</configuration>

</definition>

</datasource>

Make sure to replace username and password with your mysql database username and password

3. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.

<Property name="dataSource">jdbc/WSO2UmDB</Property>

4. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.

<dbConfig name="sharedregistry">

<dataSource>jdbc/WSO2SharedDB</dataSource>

</dbConfig>

<remoteInstance url="https://localhost:9444/registry">

<id>instanceid</id><dbConfig>sharedregistry</dbConfig>

<readOnly>false</readOnly>

<enableCache>true</enableCache>

<registryRoot>/</registryRoot>

</remoteInstance>

<mount path="/_system/config" overwrite="true">

<instanceId>instanceid</instanceId>

<targetPath>/_system/esbnodes</targetPath>

</mount>

<mount path="/_system/governance" overwrite="true">

<instanceId>instanceid</instanceId>

<targetPath>/_system/governance</targetPath>

</mount>

5. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to

<WSO2_ESB_WORKER_HOME>/repository/components/lib

6. Setting up the cluster configurations

Edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

<parameter name="membershipScheme">wka</parameter>

<parameter name="domain">wso2.esb.domain</parameter>

<parameter name="localMemberHost">esb.wso2.com</parameter>

<parameter name="localMemberPort">4200</parameter>

7. Specify the well known member. Here, the well known member is manager node

<members>

 <member>

       <hostName>mgt.esb.wso2.com</hostName>

       <port>4100</port>     

 </member>

</members>

8. Uncomment and edit WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and

<parameter name="WSDLEPRPrefix" locked="false">http://esb.wso2.com:80</parameter>

9. Configuring the port offset and host name.Edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;

<Offset>2</Offset>

<HostName>esb.wso2.com</HostName>

10. Enable SVN based deployment synchronizer with AutoCommit property marked ‘false’.

Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows

<DeploymentSynchronizer>

      <Enabled>true</Enabled>

      <AutoCommit>false</AutoCommit>

      <AutoCheckout>true</AutoCheckout>

      <RepositoryType>svn</RepositoryType>

      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>

      <SvnUser>svnuser</SvnUser>

      <SvnPassword>xxxxxx</SvnPassword>

      <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>

  </DeploymentSynchronizer>

If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

Create the worker2 node

Get a copy of worker1 node and change the following

In axis2.xml file

<parameter name="localMemberPort">4300</parameter>

In carbon.xml file

<Offset>3</Offset>

Setup Nginx load balancer

1. Install Nginx

$sudo apt-get install nginx

2. Locate the http{} block of the /etc/nginx/nginx.conf file and add the following

# load balancing among the worker nodes - HTTP

upstream wso2.esb.com {

         server esb.wso2.com:8281;

         server esb.wso2.com:8282;

         server esb.wso2.com:8283;

}

server {

         listen 80;

         server_name esb.wso2.com;

         location / {

                  proxy_pass http://wso2.esb.com;

         }

}

3. Mapping the host names to the IP, update the “/etc/hosts” file

<IP-of-worker> esb.wso2.com

<IP-of-manager> mgt.esb.wso2.com

4. Validate the configurations

$sudo nginx -t

5. Start the Nginx

$sudo /etc/init.d/nginx start

Create a Local SVN Repo

This is required if a local SVN repo is used for testing which is in same server as ESB Cluster is deployed.

1. Create folder for the local svn repos

$mkdir -p /localsvn

2. Make the folder SVN repo

$cd localsvn

$svnadmin create samplerepo

3. Grant read/write access to created folders and sub-folders/files

$chmod 755 -R *

Testing the cluster

Start the manager node

sudo <ESB_HOME>/bin/wso2server.sh

Start the worker1 and worker2 nodes

sudo <ESB_HOME>/bin/wso2server.sh -DworkerNode=true

Check member joined messages in all consoles.

Access management console through LB https://mgt.esb.wso2.com:9444/carbon

Test load distribution via http://esb.wso2.com using Simple Stock Quote Client and Pass Through Proxy as follows;

Create Proxy

Create new pass through proxy using below configuration.

<proxy xmlns="http://ws.apache.org/ns/synapse"

      name="StockQuoteProxy"

      transports="https,http"

      statistics="disable"

      trace="disable"

      startOnLoad="true">

  <target>

     <inSequence>

        <log level="full"/>

        <send>

           <endpoint>

              <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>

           </endpoint>

        </send>

     </inSequence>

     <outSequence>

        <send/>

     </outSequence>

  </target>

  <description/>

</proxy>

Deploy and Run Stock Quote client

We need some background services to be available for testing purposes. For that, run an ANT task to deploy the “SimpleStockQuoteService” in the simple axis2server as follows.

• Navigate to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService and run ‘ant’ to build the sample and deploy background services.
• Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh (on Linux) to start the Axis2 server.
• Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
• Go to <ESB_HOME>/samples/axis2client/ and execute below command in terminal.

ant stockquote -Dtrpurl=http://esb.wso2.com/services/StockQuoteProxy

• It will show message is received to one of worker/manager nodes. Executing multiple times of this will result the message to receive in different nodes in cluster.

References:

[1] - http://ravindraranwala.blogspot.com/2015/09/wso2-esb-worker-manager-cluster-without.html 

[2] - https://docs.wso2.com/display/CLUSTER420/SVN-Based+Deployment+Synchronizer+for+Carbon+4.3.0+and+4.4.0-Based+Products

[3] - http://www.thinkplexx.com/learn/howto/scm/svn/how-to-create-and-use-local-svn-subversion-repository-on-windows-or-linux-simple-and-fast-step-by-step