Thursday, June 18, 2015

JMS Inbound Coordination with WSO2 ESB and WSO2 MB

JMS Inbound coordination is introduced with WSO2 ESB 4.9.0, which allows only one active Inbound Endpoint will poll messages from a JMS queue created in WSO2 Message Broker. The Inbound Endpoint should be created with protocol 'jms', then it will be started as polling inbound by default.


Following setup will demonstrate Inbound Coordination where only one active Inbound will poll Message Store.

To create WSO2 ESB cluster for this scenario, follow my previous blog post Creating WSO2 ESB Cluster with Nginx LB.






Since we have already configured ESB cluster, follow below steps to configure WSO2 MB as the message broker.


  1. In ESB Cluster, following configurations needs to be done to establish the connection to Message Broker.
  2. Update jndi.properties file with new queue details. This should be done in all nodes in cluster.
Open the <ESB_HOME>/repository/conf/JNDI.proerties file.
Define a new queue called ‘ordersQueue’.
# register some connection factories
# connectionfactory.[jndiname] = [ConnectionURL]
connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5682'
connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5682'
# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]
queue.ordersQueue = ordersQueue
  1. Download WSO2 Message Broker binary file from WSO2 Web Site and follow instructions in Installation Guide to setup the server.
  2. Start WSO2 MB by running <MB_HOME>/bin/wso2server.sh (on Linux) or <MB_HOME>/bin/wso2server.bat (on Windows).
  3. Start WSO2 ESB Manager Node by running <ESB_HOME>/bin/wso2server.sh
and Worker nodes by <ESB_HOME>/bin/wso2server.sh -DworkerNode=true
  1. Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh to start the Axis2 server.
  2. Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
  3. Next create below proxy in ESB Manager Node. It will be used to send messages to back-end which were polled by Inbound Endpoint from MB Queue. This sequence will be specified in Inbound Endpoint configuration.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="request" onError="fault">
  <log level="full"/>
  <call>
     <endpoint>
        <address uri="http://localhost:9000/services/SimpleStockQuoteService"
                 format="soap12"/>
     </endpoint>
  </call>
  <drop/>
</sequence>
  1. Now create Inbound Endpoint using this configuration in ESB Manager node, which will connect to ‘ordersQueue’ in WSO2 MB and start polling messages in every 4 seconds. Polled messages will be injected to the sequence named 'request' which we created above.

<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"
                name="jms_inbound"
                sequence="request"
                onError="fault"
                protocol="jms"
                suspend="false">
  <parameters>
     <parameter name="interval">4000</parameter>
     <parameter name="coordination">true</parameter>
     <parameter name="transport.jms.Destination">ordersQueue</parameter>
     <parameter name="transport.jms.CacheLevel">3</parameter>
     <parameter name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter>
     <parameter name="java.naming.factory.initial">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter>
     <parameter name="sequential">true</parameter>
     <parameter name="java.naming.provider.url">repository/conf/jndi.properties</parameter>
     <parameter name="transport.jms.SessionAcknowledgement">AUTO_ACKNOWLEDGE</parameter>
     <parameter name="transport.jms.SessionTransacted">false</parameter>
     <parameter name="transport.jms.ConnectionFactoryType">queue</parameter>
  </parameters>
</inboundEndpoint>


  1. Go to admin console of WSO2 MB,https://esb.wso2.com:9453/carbon/ it will show the new queue ‘ordersQueue’ is created in Queues>Browse window.




  1. To add a message to the ‘ordersQueue’, click on ‘PublishMessage’ link of ‘ordersQueue’ and enter below message as the Message Body in new window. Set Number of Messages as 1. Then click on ‘Send Message’, then the message will be published in the queue.


<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing">
   <soapenv:Body>
       <m:placeOrder xmlns:m="http://services.samples">
    <m:order>
        <m:price>3.141593E0</m:price>
        <m:quantity>4</m:quantity>
        <m:symbol>IBM</m:symbol>
    </m:order>
</m:placeOrder>
   </soapenv:Body>
</soapenv:Envelope>
  1. One of Inbound Endpoints running in worker nodes will pick this message and send to the backend. In one of worker ESB consoles, below log entry can be observed which confirms the message polling.
[2015-05-28 15:00:36,753] DEBUG - Axis2SynapseEnvironment Creating Message Context
[2015-05-28 15:00:36,841] DEBUG - Axis2SynapseEnvironment Injecting MessageContext for inbound mediation using the : request Sequence
[2015-05-28 15:00:36,851] DEBUG - SequenceMediator Start : Sequence <request>
[2015-05-28 15:00:36,851] DEBUG - SequenceMediator Setting the onError handler : fault for the sequence : request
[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Sequence <SequenceMediator> :: mediate()
[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Mediation started from mediator position : 0
[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Building message. Sequence <SequenceMediator> is content aware
[2015-05-28 15:00:36,898] DEBUG - LogMediator Start : Log mediator
[2015-05-28 15:00:36,912] DEBUG - MiscellaneousUtil Loading a file 'synapse.properties' from classpath
[2015-05-28 15:00:36,932]  INFO - LogMediator To: , MessageID: urn:uuid:29D7910394F77252861432805436798, Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing"><soapenv:Body>
       <m:placeOrder xmlns:m="http://services.samples">
   <m:order>
       <m:price>3.141593E0</m:price>
       <m:quantity>4</m:quantity>
       <m:symbol>IBM</m:symbol>
   </m:order>
</m:placeOrder>
   </soapenv:Body></soapenv:Envelope>

To test the coordination functionality (coordination is already enabled in Inbound Configuration as <parameter name="coordination">true</parameter>), shutdown the worker node which contained above log (node that polled last message) and re-do step 9. Now the message will be polled by the other node.

Sunday, June 7, 2015

Creating WSO2 ESB Cluster with Nginx LB


This article is about creating WSO2 ESB Cluster setup (1 Manager, 2 Workers) with Nginx Load Balancer.

If you want to create same type of cluster without using load balancer, refer blog post [1] where Ravindra has explained it in detail.

WSO2 ESB Cluster will be created according to the below setup. There will be a manager node with two worker nodes.


Ports Configuration Summary



Following port configurations will be used in manager and worker nodes in the cluster.


Configuration
Manager
Worker1
Worker2
offset
1
2
3
localMemberPort
4100
4200
4300
nhttp
8281
8282
8283
nhttps
8244
8245
8246


Extract ESB packs in Manager, Worker1 and Worker2 locations and edit configurations as follows;

Setting up user management and registry databases


1. Open a mysql shell and create a database as follows
mysql> create database wso2conum_db;
2. Create database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory
mysql> use wso2conum_db;
mysql> source CARBON_HOME/dbscripts/mysql.sql;
3. Now, create another database which will be used as the shared governance and configuration registry database.
mysql> create database wso2conreg_db;
4. Create registry database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory
mysql> use wso2conreg_db;
mysql> source CARBON_HOME/dbscripts/mysql.sql;


Update SVNKit libraries

In order to work with Deployment Synchronization[2], we need to add SVNKit jar files as follows to the ESB runtime in all managers and workers.

  1. Download SVNKit from http://product-dist.wso2.com/tools/svnkit-all-1.8.7.wso2v1.jar and put into the <ESB_HOME>/repository/components/dropins folder.
  2. Download http://maven.wso2.org/nexus/content/groups/wso2-public/com/trilead/trilead-ssh2/1.0.0-build215/trilead-ssh2-1.0.0-build215.jar and put into <ESB_HOME>/repository/components/lib folder.

Configure the Manager node



  1. Unzip the WSO2 ESB binary distribution. Consider the extracted directory as <ESB_HOME>.


  1. Configure user management database and shared registry database by editing <WSO2_ESB_MGR_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<datasource>
<name>WSO2_SHARED_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conum_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>


Make sure to replace username and password with your mysql database username and password


  1. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.
<Property name="dataSource">jdbc/WSO2UmDB</Property>


  1. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.


<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id>
<dbConfig>sharedregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>


  1. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to <ESB_HOME>
/repository/components/lib


  1. Setting up the cluster configurations edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.esb.domain</parameter>
<parameter name="localMemberHost">mgt.esb.wso2.com</parameter>
<parameter name="localMemberPort">4100</parameter>
........


  1. Specify the well known member.Here, the well known member is a worker1 node


<members>
 <member>
       <hostName>esb.wso2.com</hostName>
       <port>4200</port>     
 </member>
</members>
  1. Configuring the port offset and host name, edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;


<Offset>1</Offset>
<HostName>esb.wso2.com</HostName>
<MgtHostName>mgt.esb.wso2.com</MgtHostName>


  1. Enable SVN based deployment synchronizer with AutoCommit property marked ‘true’.
Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows


<DeploymentSynchronizer>
      <Enabled>true</Enabled>
      <AutoCommit>true</AutoCommit>
      <AutoCheckout>true</AutoCheckout>
      <RepositoryType>svn</RepositoryType>
      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>
      <SvnUser>svnuser</SvnUser>
      <SvnPassword>xxxxxx</SvnPassword>
      <SvnUrlAppendTenantId>true</SvnUrlAppen dTenantId>
</DeploymentSynchronizer>


If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.


  1. Mapping the host names to the IP. Add the below host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the hostnames with the IP address of the Load balancer machine.


<IP-of-LB> esb.wso2.com
<IP-of-LB> mgt.esb.wso2.com


  1. Allow access the management console only through LB, configure the HTTP proxy ports to communicate through the load balancer.


  1. Edit “<ESB_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows


<Connector  protocol="org.apache.coyote.http11.Http11NioProtocol"
             port="9763"
             proxyPort="80"


Configure the worker node



  1. Unzip the WSO2 ESB binary distribution


  1. Configure user management database and shared registry database by editing
<ESB_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.


<datasource>
<name>WSO2_SHARED_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://localhost:3306/wso2conum_db</url>
<username>root</username>
<password>root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>


Make sure to replace username and password with your mysql database username and password


  1. Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.
<Property name="dataSource">jdbc/WSO2UmDB</Property>
  1. Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.


<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id><dbConfig>sharedregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>


  1. Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to
<WSO2_ESB_WORKER_HOME>/repository/components/lib


  1. Setting up the cluster configurations
Edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows


<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>
<parameter name="domain">wso2.esb.domain</parameter>
<parameter name="localMemberHost">esb.wso2.com</parameter>
<parameter name="localMemberPort">4200</parameter>


  1. Specify the well known member. Here, the well known member is manager node


<members>
 <member>
       <hostName>mgt.esb.wso2.com</hostName>
       <port>4100</port>     
 </member>
</members>


  1. Uncomment and edit WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and


<parameter name="WSDLEPRPrefix" locked="false">http://esb.wso2.com:80</parameter>
  1. Configuring the port offset and host name.Edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;


<Offset>2</Offset>
<HostName>esb.wso2.com</HostName>


  1. Enable SVN based deployment synchronizer with AutoCommit property marked ‘false’.
Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows


<DeploymentSynchronizer>
      <Enabled>true</Enabled>
      <AutoCommit>false</AutoCommit>
      <AutoCheckout>true</AutoCheckout>
      <RepositoryType>svn</RepositoryType>
      <SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>
      <SvnUser>svnuser</SvnUser>
      <SvnPassword>xxxxxx</SvnPassword>
      <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
  </DeploymentSynchronizer>


If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

Create the worker2 node


Get a copy of worker1 node and change the following
In axis2.xml file
<parameter name="localMemberPort">4300</parameter>
In carbon.xml file
<Offset>3</Offset>


Setup Nginx load balancer



  1. Install Nginx
$sudo apt-get install nginx


  1. Locate the http{} block of the /etc/nginx/nginx.conf file and add the following
# load balancing among the worker nodes - HTTP
upstream wso2.esb.com {
         server esb.wso2.com:8281;
         server esb.wso2.com:8282;
         server esb.wso2.com:8283;
}

server {
         listen 80;
         server_name esb.wso2.com;
         location / {
                  proxy_pass http://wso2.esb.com;
         }
}


  1. Mapping the host names to the IP, update the “/etc/hosts” file
<IP-of-worker> esb.wso2.com
<IP-of-manager> mgt.esb.wso2.com


  1. Validate the configurations
$sudo nginx -t


  1. Start the Nginx
$sudo /etc/init.d/nginx start


Create a Local SVN Repo


This is required if a local SVN repo is used for testing which is in same server as ESB Cluster is deployed.


  1. Create folder for the local svn repos
$mkdir -p /localsvn


  1. Make the folder SVN repo

$cd localsvn
$svnadmin create samplerepo


  1. Grant read/write access to created folders and sub-folders/files
$chmod 755 -R *



Testing the cluster


Start the manager node
sudo <ESB_HOME>/bin/wso2server.sh


Start the worker1 and worker2 nodes
sudo <ESB_HOME>/bin/wso2server.sh -DworkerNode=true


Check member joined messages in all consoles.


Access management console through LB https://mgt.esb.wso2.com:9444/carbon


Test load distribution via http://esb.wso2.com using Simple Stock Quote Client and Pass Through Proxy as follows;


Create Proxy



Create new pass through proxy using below configuration.


<proxy xmlns="http://ws.apache.org/ns/synapse"
      name="StockQuoteProxy"
      transports="https,http"
      statistics="disable"
      trace="disable"
      startOnLoad="true">
  <target>
     <inSequence>
        <log level="full"/>
        <send>
           <endpoint>
              <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
           </endpoint>
        </send>
     </inSequence>
     <outSequence>
        <send/>
     </outSequence>
  </target>
  <description/>
</proxy>


Deploy and Run Stock Quote client



We need some background services to be available for testing purposes. For that, run an ANT task to deploy the “SimpleStockQuoteService” in the simple axis2server as follows.
  • Navigate to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService and run ‘ant’ to build the sample and deploy background services.
  • Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh (on Linux) to start the Axis2 server.
  • Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
  • Go to <ESB_HOME>/samples/axis2client/ and execute below command in terminal.

  • It will show message is received to one of worker/manager nodes. Executing multiple times of this will result the message to receive in different nodes in cluster.