Sunday, June 7, 2015

Creating WSO2 ESB Cluster with Nginx LB


This article is about creating WSO2 ESB Cluster setup (1 Manager, 2 Workers) with Nginx Load Balancer.

If you want to create same type of cluster without using load balancer, refer blog post [1] where Ravindra has explained it in detail.

WSO2 ESB Cluster will be created according to the below setup. There will be a manager node with two worker nodes.


Ports Configuration Summary



Following port configurations will be used in manager and worker nodes in the cluster.


Configuration
Manager
Worker1
Worker2
offset
1
2
3
localMemberPort
4100
4200
4300
nhttp
8281
8282
8283
nhttps
8244
8245
8246


Extract ESB packs in Manager, Worker1 and Worker2 locations and edit configurations as follows;

Setting up user management and registry databases


1. Open a mysql shell and create a database as follows
mysql> create database wso2conum_db;
2. Create database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory
mysql> use wso2conum_db;
mysql> source CARBON_HOME/dbscripts/mysql.sql;
3. Now, create another database which will be used as the shared governance and configuration registry database.
mysql> create database wso2conreg_db;
4. Create registry database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory
mysql> use wso2conreg_db;
mysql> source CARBON_HOME/dbscripts/mysql.sql;


Update SVNKit libraries

In order to work with Deployment Synchronization[2], we need to add SVNKit jar files as follows to the ESB runtime in all managers and workers.

  1. Download SVNKit from http://product-dist.wso2.com/tools/svnkit-all-1.8.7.wso2v1.jar and put into the <ESB_HOME>/repository/components/dropins folder.
  2. Download http://maven.wso2.org/nexus/content/groups/wso2-public/com/trilead/trilead-ssh2/1.0.0-build215/trilead-ssh2-1.0.0-build215.jar and put into <ESB_HOME>/repository/components/lib folder.

Configure the Manager node



  1. Unzip the WSO2 ESB binary distribution. Consider the extracted directory as <ESB_HOME>.


  1. Configure user management database and shared registry database by editing <WSO2_ESB_MGR_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<datasource>
<name>WSO2_SHARED_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conum_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>


Make sure to replace username and password with your mysql database username and password


  1. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.
<Property name="dataSource">jdbc/WSO2UmDB</Property>


  1. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.


<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id>
<dbConfig>sharedregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>


  1. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to <ESB_HOME>
/repository/components/lib


  1. Setting up the cluster configurations edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.esb.domain</parameter>
<parameter name="localMemberHost">mgt.esb.wso2.com</parameter>
<parameter name="localMemberPort">4100</parameter>
........


  1. Specify the well known member.Here, the well known member is a worker1 node


<members>
 <member>
       <hostName>esb.wso2.com</hostName>
       <port>4200</port>     
 </member>
</members>
  1. Configuring the port offset and host name, edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;


<Offset>1</Offset>
<HostName>esb.wso2.com</HostName>
<MgtHostName>mgt.esb.wso2.com</MgtHostName>


  1. Enable SVN based deployment synchronizer with AutoCommit property marked ‘true’.
Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows


<DeploymentSynchronizer>
      <Enabled>true</Enabled>
      <AutoCommit>true</AutoCommit>
      <AutoCheckout>true</AutoCheckout>
      <RepositoryType>svn</RepositoryType>
      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>
      <SvnUser>svnuser</SvnUser>
      <SvnPassword>xxxxxx</SvnPassword>
      <SvnUrlAppendTenantId>true</SvnUrlAppen dTenantId>
</DeploymentSynchronizer>


If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.


  1. Mapping the host names to the IP. Add the below host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the hostnames with the IP address of the Load balancer machine.


<IP-of-LB> esb.wso2.com
<IP-of-LB> mgt.esb.wso2.com


  1. Allow access the management console only through LB, configure the HTTP proxy ports to communicate through the load balancer.


  1. Edit “<ESB_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows


<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
             port="9763"
             proxyPort="80"


Configure the worker node



  1. Unzip the WSO2 ESB binary distribution


  1. Configure user management database and shared registry database by editing
<ESB_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.


<datasource>
<name>WSO2_SHARED_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conum_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>


Make sure to replace username and password with your mysql database username and password


  1. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.
<Property name="dataSource">jdbc/WSO2UmDB</Property>
  1. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.


<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id><dbConfig>sharedregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>


  1. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to
<WSO2_ESB_WORKER_HOME>/repository/components/lib


  1. Setting up the cluster configurations
Edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.esb.domain</parameter>
<parameter name="localMemberHost">esb.wso2.com</parameter>
<parameter name="localMemberPort">4200</parameter>


  1. Specify the well known member. Here, the well known member is manager node


<members>
 <member>
       <hostName>mgt.esb.wso2.com</hostName>
       <port>4100</port>     
 </member>
</members>


  1. Uncomment and edit WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and


<parameter name="WSDLEPRPrefix" locked="false">http://esb.wso2.com:80</parameter>
  1. Configuring the port offset and host name.Edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;


<Offset>2</Offset>
<HostName>esb.wso2.com</HostName>


  1. Enable SVN based deployment synchronizer with AutoCommit property marked ‘false’.
Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows


<DeploymentSynchronizer>
      <Enabled>true</Enabled>
      <AutoCommit>false</AutoCommit>
      <AutoCheckout>true</AutoCheckout>
      <RepositoryType>svn</RepositoryType>
      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>
      <SvnUser>svnuser</SvnUser>
      <SvnPassword>xxxxxx</SvnPassword>
      <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
  </DeploymentSynchronizer>


If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

Create the worker2 node


Get a copy of worker1 node and change the following
In axis2.xml file
<parameter name="localMemberPort">4300</parameter>
In carbon.xml file
<Offset>3</Offset>


Setup Nginx load balancer



  1. Install Nginx
$sudo apt-get install nginx


  1. Locate the http{} block of the /etc/nginx/nginx.conf file and add the following
# load balancing among the worker nodes - HTTP
upstream wso2.esb.com {
         server esb.wso2.com:8281;
         server esb.wso2.com:8282;
         server esb.wso2.com:8283;
}

server {
         listen 80;
         server_name esb.wso2.com;
         location / {
                  proxy_pass http://wso2.esb.com;
         }
}


  1. Mapping the host names to the IP, update the “/etc/hosts” file
<IP-of-worker> esb.wso2.com
<IP-of-manager> mgt.esb.wso2.com


  1. Validate the configurations
$sudo nginx -t


  1. Start the Nginx
$sudo /etc/init.d/nginx start


Create a Local SVN Repo


This is required if a local SVN repo is used for testing which is in same server as ESB Cluster is deployed.


  1. Create folder for the local svn repos
$mkdir -p /localsvn


  1. Make the folder SVN repo

$cd localsvn
$svnadmin create samplerepo


  1. Grant read/write access to created folders and sub-folders/files
$chmod 755 -R *



Testing the cluster


Start the manager node
sudo <ESB_HOME>/bin/wso2server.sh


Start the worker1 and worker2 nodes
sudo <ESB_HOME>/bin/wso2server.sh -DworkerNode=true


Check member joined messages in all consoles.


Access management console through LB https://mgt.esb.wso2.com:9444/carbon


Test load distribution via http://esb.wso2.com using Simple Stock Quote Client and Pass Through Proxy as follows;


Create Proxy



Create new pass through proxy using below configuration.


<proxy xmlns="http://ws.apache.org/ns/synapse"
      name="StockQuoteProxy"
      transports="https,http"
      statistics="disable"
      trace="disable"
      startOnLoad="true">
  <target>
     <inSequence>
        <log level="full"/>
        <send>
           <endpoint>
              <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
           </endpoint>
        </send>
     </inSequence>
     <outSequence>
        <send/>
     </outSequence>
  </target>
  <description/>
</proxy>


Deploy and Run Stock Quote client



We need some background services to be available for testing purposes. For that, run an ANT task to deploy the “SimpleStockQuoteService” in the simple axis2server as follows.
  • Navigate to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService and run ‘ant’ to build the sample and deploy background services.
  • Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh (on Linux) to start the Axis2 server.
  • Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
  • Go to <ESB_HOME>/samples/axis2client/ and execute below command in terminal.

  • It will show message is received to one of worker/manager nodes. Executing multiple times of this will result the message to receive in different nodes in cluster.


8 comments:

  1. Good Post, U have missed adding svn jars to libs and dropings directories....:)

    ReplyDelete
  2. Hi Yasassri,

    Updated missing details. Thanks for the feedback.

    ReplyDelete
  3. very good explanation details..

    ReplyDelete
  4. Nice Article , It will be great if you can list down the benefits/views of Master-Worker cluster, worker-worker cluster and multiple nodes(workers) with No clustering

    ReplyDelete
  5. Very useful article. This can be referred to set up ESB 4.8.1 cluster as well. But the svnkit jar's version should be 1.3.9,
    Can be downloaded in http://dist.wso2.org/tools/svnkit-1.3.9.wso2v2.jar

    ReplyDelete
  6. This is working perfectly. Really helpful for a beginner.

    ReplyDelete