JMS Inbound coordination is introduced with WSO2 ESB 4.9.0, which allows only one active Inbound Endpoint will poll messages from a JMS queue created in WSO2 Message Broker. The Inbound Endpoint should be created with protocol 'jms', then it will be started as polling inbound by default.

Following setup will demonstrate Inbound Coordination where only one active Inbound will poll Message Store.

To create WSO2 ESB cluster for this scenario, follow my previous blog post Creating WSO2 ESB Cluster with Nginx LB.

Since we have already configured ESB cluster, follow below steps to configure WSO2 MB as the message broker.

In ESB Cluster, following configurations needs to be done to establish the connection to Message Broker.
Update jndi.properties file with new queue details. This should be done in all nodes in cluster.

Open the <ESB_HOME>/repository/conf/JNDI.proerties file.

Define a new queue called ‘ordersQueue’.

# register some connection factories
# connectionfactory.[jndiname] = [ConnectionURL]

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5682'

connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5682'

# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]

queue.ordersQueue = ordersQueue

Download WSO2 Message Broker binary file from WSO2 Web Site and follow instructions in Installation Guide to setup the server.
Start WSO2 MB by running <MB_HOME>/bin/wso2server.sh (on Linux) or <MB_HOME>/bin/wso2server.bat (on Windows).
Start WSO2 ESB Manager Node by running <ESB_HOME>/bin/wso2server.sh

and Worker nodes by <ESB_HOME>/bin/wso2server.sh -DworkerNode=true

Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh to start the Axis2 server.
Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
Next create below proxy in ESB Manager Node. It will be used to send messages to back-end which were polled by Inbound Endpoint from MB Queue. This sequence will be specified in Inbound Endpoint configuration.

<call>

<address uri="http://localhost:9000/services/SimpleStockQuoteService"

format="soap12"/>

</endpoint>

</call>

<drop/>

</sequence>

Now create Inbound Endpoint using this configuration in ESB Manager node, which will connect to ‘ordersQueue’ in WSO2 MB and start polling messages in every 4 seconds. Polled messages will be injected to the sequence named 'request' which we created above.

<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"

name="jms_inbound"

sequence="request"

onError="fault"

protocol="jms"

suspend="false">

<parameter name="transport.jms.Destination">ordersQueue</parameter>

<parameter name="transport.jms.ConnectionFactoryJNDIName">QueueConnectionFactory</parameter>

<parameter name="java.naming.factory.initial">org.wso2.andes.jndi.PropertiesFileInitialContextFactory</parameter>

<parameter name="java.naming.provider.url">repository/conf/jndi.properties</parameter>

<parameter name="transport.jms.SessionAcknowledgement">AUTO_ACKNOWLEDGE</parameter>

<parameter name="transport.jms.SessionTransacted">false</parameter>

<parameter name="transport.jms.ConnectionFactoryType">queue</parameter>

</parameters>

</inboundEndpoint>

Go to admin console of WSO2 MB,https://esb.wso2.com:9453/carbon/ it will show the new queue ‘ordersQueue’ is created in Queues>Browse window.

To add a message to the ‘ordersQueue’, click on ‘PublishMessage’ link of ‘ordersQueue’ and enter below message as the Message Body in new window. Set Number of Messages as 1. Then click on ‘Send Message’, then the message will be published in the queue.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing">

<soapenv:Body>

<m:placeOrder xmlns:m="http://services.samples">

<m:order>

<m:price>3.141593E0</m:price>

<m:quantity>4</m:quantity>

<m:symbol>IBM</m:symbol>

</m:order>

</m:placeOrder>

</soapenv:Body>

</soapenv:Envelope>

One of Inbound Endpoints running in worker nodes will pick this message and send to the backend. In one of worker ESB consoles, below log entry can be observed which confirms the message polling.

[2015-05-28 15:00:36,753] DEBUG - Axis2SynapseEnvironment Creating Message Context

[2015-05-28 15:00:36,841] DEBUG - Axis2SynapseEnvironment Injecting MessageContext for inbound mediation using the : request Sequence

[2015-05-28 15:00:36,851] DEBUG - SequenceMediator Start : Sequence <request>

[2015-05-28 15:00:36,851] DEBUG - SequenceMediator Setting the onError handler : fault for the sequence : request

[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Sequence <SequenceMediator> :: mediate()

[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Mediation started from mediator position : 0

[2015-05-28 15:00:36,858] DEBUG - SequenceMediator Building message. Sequence <SequenceMediator> is content aware

[2015-05-28 15:00:36,898] DEBUG - LogMediator Start : Log mediator

[2015-05-28 15:00:36,912] DEBUG - MiscellaneousUtil Loading a file 'synapse.properties' from classpath

[2015-05-28 15:00:36,932] INFO - LogMediator To: , MessageID: urn:uuid:29D7910394F77252861432805436798, Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing"><soapenv:Body>

<m:placeOrder xmlns:m="http://services.samples">

<m:order>

<m:price>3.141593E0</m:price>

<m:quantity>4</m:quantity>

<m:symbol>IBM</m:symbol>

</m:order>

</m:placeOrder>

</soapenv:Body></soapenv:Envelope>

To test the coordination functionality (coordination is already enabled in Inbound Configuration as <parameter name="coordination">true</parameter>), shutdown the worker node which contained above log (node that polled last message) and re-do step 9. Now the message will be polled by the other node.

This article is about creating WSO2 ESB Cluster setup (1 Manager, 2 Workers) with Nginx Load Balancer.

If you want to create same type of cluster without using load balancer, refer blog post [1] where Ravindra has explained it in detail.

WSO2 ESB Cluster will be created according to the below setup. There will be a manager node with two worker nodes.

Ports Configuration Summary

Following port configurations will be used in manager and worker nodes in the cluster.

Configuration	Manager	Worker1	Worker2
offset	1	2	3
localMemberPort	4100	4200	4300
nhttp	8281	8282	8283
nhttps	8244	8245	8246

Extract ESB packs in Manager, Worker1 and Worker2 locations and edit configurations as follows;

Setting up user management and registry databases

1. Open a mysql shell and create a database as follows

mysql> create database wso2conum_db;

2. Create database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory

mysql> use wso2conum_db;

mysql> source CARBON_HOME/dbscripts/mysql.sql;

3. Now, create another database which will be used as the shared governance and configuration registry database.

mysql> create database wso2conreg_db;

4. Create registry database schema and populate tables as follows. Make sure to replace CARBON_HOME with the absolute path of WSO2 ESB directory

mysql> use wso2conreg_db;

mysql> source CARBON_HOME/dbscripts/mysql.sql;

Update SVNKit libraries

In order to work with Deployment Synchronization[2], we need to add SVNKit jar files as follows to the ESB runtime in all managers and workers.

Download SVNKit from http://product-dist.wso2.com/tools/svnkit-all-1.8.7.wso2v1.jar and put into the <ESB_HOME>/repository/components/dropins folder.
Download http://maven.wso2.org/nexus/content/groups/wso2-public/com/trilead/trilead-ssh2/1.0.0-build215/trilead-ssh2-1.0.0-build215.jar and put into <ESB_HOME>/repository/components/lib folder.

Configure the Manager node

Unzip the WSO2 ESB binary distribution. Consider the extracted directory as <ESB_HOME>.

Configure user management database and shared registry database by editing <WSO2_ESB_MGR_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<name>WSO2_SHARED_REG_DB</name>

<description>The datasource used for shared config and governance registry</description>

<name>jdbc/WSO2SharedDB</name>

</jndiConfig>

<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<validationQuery>SELECT 1</validationQuery>

</configuration>

</definition>

</datasource>

<description>The datasource used for registry and user manager</description>

</jndiConfig>

<url>jdbc:mysql://localhost:3306/wso2conum_db</url>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<validationQuery>SELECT 1</validationQuery>

</configuration>

</definition>

</datasource>

Make sure to replace username and password with your mysql database username and password

Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.

Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.

<dataSource>jdbc/WSO2SharedDB</dataSource>

</dbConfig>

<id>instanceid</id>

<dbConfig>sharedregistry</dbConfig>

<readOnly>false</readOnly>

</remoteInstance>

<instanceId>instanceid</instanceId>

<targetPath>/_system/esbnodes</targetPath>

</mount>

<instanceId>instanceid</instanceId>

<targetPath>/_system/governance</targetPath>

</mount>

Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to <ESB_HOME>

/repository/components/lib

Setting up the cluster configurations edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows

<parameter name="domain">wso2.esb.domain</parameter>

<parameter name="localMemberPort">4100</parameter>
........

Specify the well known member.Here, the well known member is a worker1 node

</member>

</members>

Configuring the port offset and host name, edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;

Enable SVN based deployment synchronizer with AutoCommit property marked ‘true’.

Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows

<SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>

<SvnUser>svnuser</SvnUser>

<SvnPassword>xxxxxx</SvnPassword>

</DeploymentSynchronizer>

If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

Mapping the host names to the IP. Add the below host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You have to map the hostnames with the IP address of the Load balancer machine.

<IP-of-LB> esb.wso2.com

<IP-of-LB> mgt.esb.wso2.com

Allow access the management console only through LB, configure the HTTP proxy ports to communicate through the load balancer.

Edit “<ESB_HOME>/repository/conf/tomcat/catalina-server.xml” file as follows

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

port="9763"

proxyPort="80"

Configure the worker node

Unzip the WSO2 ESB binary distribution

Configure user management database and shared registry database by editing

<ESB_HOME>/repository/conf/datasoruces/master-datasources.xml as shown below. Add this as a new entry to the config file.

<name>WSO2_SHARED_REG_DB</name>

<description>The datasource used for shared config and governance registry</description>

<name>jdbc/WSO2SharedDB</name>

</jndiConfig>

<url>jdbc:mysql://localhost:3306/wso2conreg_db</url>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<validationQuery>SELECT 1</validationQuery>

</configuration>

</definition>

</datasource>

<description>The datasource used for registry and user manager</description>

</jndiConfig>

<url>jdbc:mysql://localhost:3306/wso2conum_db</url>

<driverClassName>com.mysql.jdbc.Driver</driverClassName>

<validationQuery>SELECT 1</validationQuery>

</configuration>

</definition>

</datasource>

Make sure to replace username and password with your mysql database username and password

Update <ESB_HOME>/repository/conf/user-mgt.xml as shown below under UserManager>Realm>Configuration.

Configure shared registry database and mounting details in <ESB_HOME>/repository/ conf/registry.xml as follows. Keep existing <dbConfig> and add following as a new entry. This will be used as the shared registry among the cluster nodes.

<dataSource>jdbc/WSO2SharedDB</dataSource>

</dbConfig>

<id>instanceid</id><dbConfig>sharedregistry</dbConfig>

<readOnly>false</readOnly>

</remoteInstance>

<instanceId>instanceid</instanceId>

<targetPath>/_system/esbnodes</targetPath>

</mount>

<instanceId>instanceid</instanceId>

<targetPath>/_system/governance</targetPath>

</mount>

Copy mysql jdbc driver (mysql-connector-java-5.1.19-bin.jar) to

<WSO2_ESB_WORKER_HOME>/repository/components/lib

Setting up the cluster configurations

Edit “<ESB_HOME>/repository/conf/axis2/axis2.xml” file as follows

<parameter name="domain">wso2.esb.domain</parameter>

Specify the well known member. Here, the well known member is manager node

</member>

</members>

Uncomment and edit WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and

Configuring the port offset and host name.Edit “<ESB_HOME>/repository/conf/ carbon.xml” file as follows;

Enable SVN based deployment synchronizer with AutoCommit property marked ‘false’.

Edit “<ESB_HOME>/repository/conf/carbon.xml” file as follows

<AutoCommit>false</AutoCommit>

<SvnUrl>https://svn.wso2.org/repos/samplerepo</SvnUrl>

<SvnUser>svnuser</SvnUser>

<SvnPassword>xxxxxx</SvnPassword>

</DeploymentSynchronizer>

If it is a local SVN, use file:///<svn folder path> as the SvnUrl and no need of username/password.

Create the worker2 node

Get a copy of worker1 node and change the following

In axis2.xml file

In carbon.xml file

Setup Nginx load balancer

Install Nginx

$sudo apt-get install nginx

Locate the http{} block of the /etc/nginx/nginx.conf file and add the following

# load balancing among the worker nodes - HTTP

upstream wso2.esb.com {

server esb.wso2.com:8281;

server esb.wso2.com:8282;

server esb.wso2.com:8283;

}

server {

listen 80;

server_name esb.wso2.com;

location / {

proxy_pass http://wso2.esb.com;

}

Mapping the host names to the IP, update the “/etc/hosts” file

<IP-of-worker> esb.wso2.com

<IP-of-manager> mgt.esb.wso2.com

Validate the configurations

$sudo nginx -t

Start the Nginx

$sudo /etc/init.d/nginx start

Create a Local SVN Repo

This is required if a local SVN repo is used for testing which is in same server as ESB Cluster is deployed.

Create folder for the local svn repos

$mkdir -p /localsvn

Make the folder SVN repo

$cd localsvn

$svnadmin create samplerepo

Grant read/write access to created folders and sub-folders/files

$chmod 755 -R *

Testing the cluster

Start the manager node

sudo <ESB_HOME>/bin/wso2server.sh

Start the worker1 and worker2 nodes

sudo <ESB_HOME>/bin/wso2server.sh -DworkerNode=true

Check member joined messages in all consoles.

Access management console through LB https://mgt.esb.wso2.com:9444/carbon

Test load distribution via http://esb.wso2.com using Simple Stock Quote Client and Pass Through Proxy as follows;

Create Proxy

Create new pass through proxy using below configuration.

<proxy xmlns="http://ws.apache.org/ns/synapse"

name="StockQuoteProxy"

transports="https,http"

statistics="disable"

trace="disable"

startOnLoad="true">

<send>

</endpoint>

</send>

</inSequence>

<send/>

</outSequence>

</target>

</proxy>

Deploy and Run Stock Quote client

We need some background services to be available for testing purposes. For that, run an ANT task to deploy the “SimpleStockQuoteService” in the simple axis2server as follows.

Navigate to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService and run ‘ant’ to build the sample and deploy background services.
Next, run <ESB_HOME>/samples/axis2server/axis2Server.sh (on Linux) to start the Axis2 server.
Point your browser to http://127.0.0.1:9000/services/SimpleStockQuoteService?wsdl and verify that the service is running.
Go to <ESB_HOME>/samples/axis2client/ and execute below command in terminal.

ant stockquote -Dtrpurl=http://esb.wso2.com/services/StockQuoteProxy

It will show message is received to one of worker/manager nodes. Executing multiple times of this will result the message to receive in different nodes in cluster.

References:

[1] - http://ravindraranwala.blogspot.com/2015/09/wso2-esb-worker-manager-cluster-without.html

[2] - https://docs.wso2.com/display/CLUSTER420/SVN-Based+Deployment+Synchronizer+for+Carbon+4.3.0+and+4.4.0-Based+Products

[3] - http://www.thinkplexx.com/learn/howto/scm/svn/how-to-create-and-use-local-svn-subversion-repository-on-windows-or-linux-simple-and-fast-step-by-step

The life is a process of Learning!

Thursday, June 18, 2015

JMS Inbound Coordination with WSO2 ESB and WSO2 MB

JMS Inbound coordination is introduced with WSO2 ESB 4.9.0, which allows only one active Inbound Endpoint will poll messages from a JMS queue created in WSO2 Message Broker. The Inbound Endpoint should be created with protocol 'jms', then it will be started as polling inbound by default.

Sunday, June 7, 2015

Creating WSO2 ESB Cluster with Nginx LB

Ports Configuration Summary

Configure the Manager node

Configure the worker node

Setup Nginx load balancer

Create Proxy

Deploy and Run Stock Quote client

References:

[1] - http://ravindraranwala.blogspot.com/2015/09/wso2-esb-worker-manager-cluster-without.html

[2] - https://docs.wso2.com/display/CLUSTER420/SVN-Based+Deployment+Synchronizer+for+Carbon+4.3.0+and+4.4.0-Based+Products