Saturday, November 14, 2015

Indexing and Searching through Lucene

Why Lucene in WSO2 Data Analytics Server ?

A common use-case for using Lucene indexing in Data Analytics Server (DAS) is to perform a full-text search on one or more persisted event stream data. DAS provides interactive data analysis (means it is used where a stored dataset can be queried in an ad-hoc manner in finding useful information more quickly and more accurately) for allowing you to search for persisted events using the Data Explorer .

What is Lucene ?

Lucene is an extremely rich and powerful full-text search (information retrieval) library which is written in Java. You can use Lucene to provide full-text indexing across both database objects and documents in various formats. Lucene provides search over documents. A document is essentially a collection of fields, where a field supplies a field name and value (name-value pair).

The primitive concept behind the Lucene is to take dataset and place it in fields to either be stored, indexed, or both indexed and stored. Indexed means you can search against that field, stored means you cannot search against the field but you can retrieve it’s contents. There are also non-stored and non-indexed fields but they are primarily used for the storage of metadata.

You can retrieve the dataset stored in the database, put it into fields (as name-value pair), put those fields into a "document", and then add the document to the indexing process. The index is a set of files on disk or in memory. There are multiple files contained in an an index and the files are platform independent.

Searching and Indexing through Lucene

Lucene is able to retrieve informations fast and efficiently because, instead of searching the text directly, it searches an index instead. This would be the equivalent of retrieving pages in a book related to a keyword by searching the index at the back of a book, as opposed to searching the words in each page of the book.

What actually gets indexed is a set of terms. A term (eg:- title:"Modern") combines a field name with a token that may be used for search. For instance, a title field like Modern Operating Systems, 2nd Edition might yield the tokens modern, operat, 2, and edition after case normalization, stemming and stoplisting. The index structure provides the reverse mapping from terms, consisting of field names and tokens, back to documents. This type of index is called an inverted index, because it inverts a page-centric data structure (page -> words) to a keyword-centric data structure (word -> pages). 

The following diagram shows how the indexing process happens in Lucene.
In WSO2 DAS, published events by data agents through event receivers can be persisted in RDBMS such as MySql and denormalizing the tables (RDBMS) into Lucene Documents when performing the lucene indexing.

The pseudo code will look something like this:

//The sql query to be performed
String sql = "SELECT DISTINCT processInstanceId, duration FROM PROCESS_USAGE_SUMMARY";
//ResultSet to hold the data retreived from the database  
ResultSet rs = stmt.executeQuery(sql);
while (rs.next()) {
    Document doc = new Document();
    doc.add(new Field("processInstanceId", rs,getString("processInstanceId"), Field.Store.YES, Field.Index.TOKENIZED));
    doc.add(new Field("duration", rs,getLong("duration"), Field.Store.YES, Field.Index.UN_TOKENIZED));
    // ... repeat for each column in result set
    writer.addDocument(doc);
}

When you perform a Search operation, it involves creating a Query (usually via a QueryParser) and handing this Query to an IndexSearcher, which returns a list of Hits. Actually this returns a set of documents according to the query you provided and from that extract the information in the documents and finally display the results. You can build the query string as the format provided in the WSO2 DAS (as a JSON string) and then pass it to its REST API  to return the result in the JSON format.

The Lucene query language allows the user to specify which field or fields to search on, which fields to give more weight, the ability to perform boolean queries (AND, OR, NOT) and other functionality as well. For more about Lucene query parser syntax click here.

References



Saturday, October 10, 2015

SPARK SQL User Defined Functions (UDFs) for WSO2 Data Analytics Server

What are UDFs ?

Generally SPARK-Sequel having some built-in functions, we can use that built-in functions in the Spark script without adding any extra code or calculation. However some times user requirement is not satisfied by that built-in functions. At that time user can write some own custom functions called UDFs and they are operate on distributed data-frames and works row by row unless you're creating an user defined aggregation function. WSO2 DAS has an abstraction layer for generic Spark UDF which makes it convenient to introduce UDFs to the server.

Here I will describe how to write Spark-Sequel UDF Example in Java.

Simple UDF to convert the date into the given date format

Step 1: Create the POJO class

The following example shows the UDF POJO for converting the date in the format of eg:Thu Sep 24 09:35:56 IST 2015 to the date in the format of yyyy-MM-dd. The name of the Spark UDF should be the name of the method defined in the class (in this example it is dateStr). This will be used when invoking the UDF through Spark-SQL. Here dateStr("Thu Sep 24 09:35:56 IST 2015") returns the String “2015-09-24”. (POJO class name: AnalyticsUDF)
















 Step 2: Packaging the class as jar

The custom UDF class you created should be bundled as a jar and added to <DAS_HOME>/repository/components/lib directory.

Step 3: Update spark UDF configuration file

Add the newly created custom UDF to the <DAS_HOME>/repository/conf/analytics/spark/spark-udf-config.xml file as shown in the example below.








(Here org.wso2.carbon.pc.spark.udfs is the package name of the class).


Saturday, October 3, 2015

BPMN data publisher in WSO2 Business Process Server

The purpose of writing this blog post is to demonstrate how to publish completed BPMN process and task instances to the WSO2 Data Analytics Server (DAS). Before moving into the configuration part of Business Process Server (BPS) to publish data, first I will give you a brief introduction about this new feature.

The execution data on process instances has to be captured for analyzing the behavior of processes. In order to facilitate this, execution data of BPMN process instances has to be published to WSO2 DAS. Once published, such data can be analyzed and presented from DAS.

This data publishing component can run independent of the process runtime and access instance data using the REST API, Java API or directly from the database. Data streams, data items included in streams, polling frequency, publishing endpoints, etc have to configurable as much as possible.

Proposed solution mainly consisted of two parts. The first step is to access the completed process and task instances data using the activiti engine JAVA API. The next step is to publish instances which grabbed from the step one to WSO2 Data Analytics Server (DAS) through a data publisher.

WSO2 Business Process Server

 

Business Process Management (BPM) is a key technology and a systematic approach to increase productivity and re-energizing businesses, making an organization's work-flow more effective, more efficient and more capable of adapting to an ever-changing environment. WSO2 Business Process Server (BPS) enables developers to easily deploy, manage and monitor business processes and process instances written using the WS-BPEL and BPMN standards through a complete web-based graphical user interface. Here a business process is an activity or set of activities that will accomplish a specific organizational goal.
 

Business Process Model Notation (BPMN)

 

BPMN is a graphical notation which describes the logic of steps in a business process. This notation has been especially designed to coordinate the sequence of processes and messages that flow between participants in different activities. The BPMN provides a common language which allows all the parties involved to communicate processes clearly, completely and efficiently. In this way, BPMN defines the notation and semantics of a Business Process Diagram (BPD).

BPD is a diagram based on the flowchart technique, designed to present a graphical sequence of all the activities that take place during a process. It also includes all relative information for making an analysis. BPD is a diagram designed for the use of process analysts who design, control and manage processes. In a BPD diagram there are a series of graphical elements that are grouped into categories.

Access instances using activiti engine JAVA API


The primary component that have to deal with when designing and developing BPMN processes with Activiti is the Activiti Engine. Here the activiti engine is the entry point to deploy new process definitions, starting new process instances, querying for process instances, user tasks which are running as well as already completed and so on. In the first part, need to capture all the completed process and task instances and for that use a in-built interface called HistoryService which provided in the JAVA API to grab the completed process instances and task instances.

The HistoryService provides an interface to query for completed process and task instances to gain the information about them such as process definition key, process instance id, start time, end time, assignee of the task, owner, claim time and etc. This service exposes mainly query capabilities to access this historic data. Before dive into the configuration part of the server let's try to identify how the historic data about process and task instances is stored inside the Activiti Engine database.

The historic process instances store in the database table called ACT_HI_PROCINST and the historic tasks store in ACT_HI_ACTINST table. A historic process instance is stored in the ACT_HI_PROCINST table when a new process instance is started. So when try to query on historic process instances will also grant results of all created process instances that are still running which don't have an end time yet. When the process instance enters its first user task state a record in the historic activity table is made. Whenever the task is finished, the record is updated with the end time of the task. When the user task is completed, the end time of the task instance is updated with the time at completion.


Data Agent

DAS data agent is used to collect data from WSO2 service-hosting products (such as WSO2 Business Process Server (BPS), WSO2 ESB, WSO2 Application Server (AS), WSO2 Data Services Server (DSS) and etc.) and send to the WSO2 DAS server. DAS server receives data events via its Thrift API. This protocol uses a binary protocol and enables fast data transmission between the service-publishing products and DAS server. Data Agent can be configured early through stream definition so that the data to be extracted from the data-publishing product can be per-defined.
 

Data Publisher


Data publisher generally allow to send data to a predefined set of data fields, to the DAS only specified in the stream definitions. This is a set of fixed data fields, which send through a data bridge. And can also send custom key-value pairs with data events. Here in this project use the thrift Data publisher and it will be sending events to the Data Bridge via thrift using the tcp transport. When starting the server the Data Receiver of the Data Bridge is exposing two thrift ports. The “Thrift SSL port” should be used for secure data transmission and the “Thrift port” for non secure data transfer. To access the secure port and send data need connect to their server with the following url format.

ssl://<Ip address for the server>:<Thrift SSL post>

E.g. ssl://10.100.7.75:7711

Data Streams

 

In the stream concept the data sender has to agree on a set of data types it wish to send in an each data event. When the message is sent to the Data Receiver, the set of types of it wish to send, is sent with the message defining the Stream. This is known as the Stream Definition. Each Stream Definition is uniquely identified using the pair of Stream Name and its Version. Data Stream can be defined in two ways. One is using a Java code, and the other way is defining a Stream as a Java string object with the format of a JSON object. In this project use the JAVA based stream definition and you can see it in the implementation phase in this report.

(Like creating tables before sending any data to store in a database, first define the streams before sending events. Here streams are description of how the sending data look like (like database Schema))

Before sending events I need to define the stream definition to send data. Data Bridge can use various Stream Definition Stores and according to the stream definition store the frequency of defining the streams varies. Below two generic streams can be provided.
    process.instance.stream:
    {
      "name": "BPMN_Process_Instance_Data_Publish",
      "version": "1.0.0",
      "nickName": "",
      "description": "BPMN process instances data",
      "payloadData": [
        {
          "name": "processDefinitionId",
          "type": "STRING"
        },
        {
          "name": "processInstanceId",
          "type": "STRING"
        },
        {
          "name": "startActivityId",
          "type": "STRING"
        },
        {
          "name": "startUserId",
          "type": "STRING"
        },
        {
          "name": "startTime",
          "type": "STRING"
        },
        {
          "name": "endTime",
          "type": "STRING"
        },
        {
          "name": "duration",
          "type": "LONG"
        },
        {
          "name": "tenantId",
          "type": "STRING"
        }
      ]
    }

    task.instance.stream:
    {
      "name": "BPMN_Task_Instance_Data_Publish",
      "version": "1.0.0",
      "nickName": "",
      "description": "BPMN user tasks data",
      "payloadData": [
        {
          "name": "taskDefinitionKey",
          "type": "STRING"
        },
        {
          "name": "taskInstanceId",
          "type": "STRING"
        },
        {
          "name": "processInstanceId",
          "type": "STRING"
        },
        {
          "name": "createTime",
          "type": "STRING"
        },
        {
          "name": "startTime",
          "type": "STRING"
        },
        {
          "name": "endTime",
          "type": "STRING"
        },
        {
          "name": "duration",
          "type": "LONG"
        },
        {
          "name": "assignee",
          "type": "STRING"
        }
      ]
    }
     


      Configuring BPMN Data publisher to publish data to the WSO2 DAS

       Step 1

      In the BPS UI console, first select the Configure tag and then go to BPMN Data Publisher. Now you can see the configuration UI for the publisher as below. Then fill the Thrift API configuration part according to the correct DAS (Data Analytics Server) instance server username password and the thrift url.




      Step 2

      To enable the data publisher for publishing events to the DAS you should have to change the value of the property called dataPublishingEnabled to true. (This configuration includes in the activiti.xml file. You can find it from the <BPS_HOME>/repository/conf folder.)

      Step 3

      After doing the first two steps restart the BPS server instance. (To avoid the port conflict with DAS, change the port offset of the BPS inside of carbon.xml file). Now in the DAS side you should have to persist the stream definitions for both task and process instances separately. You can see the fallowing output when clicking the streams menu item in DAS UI console. For enabling to store the incoming data from the BPS server, need to create two data receivers to capture process and task instances data.  




      How to configure the event receivers ??? Click Here  

      Step 4

      You can see the event flow between event receivers and the persist event store by clicking the menu item called Flow in DAS UI console as below.
        
      Step 5

      Finally you can see the sample published process and task instances through the BPMN data publisher to the data Analytics Server in the following two figures respectively.

      Sunday, June 14, 2015

      Implement a WSO2 Carbon Component using eclipse IDE

      Introduction

      This tutorial mainly focus on how to implement a WSO2 carbon component from scratch and help you to understand the structure of the project that needs to be followed  when implementing a WSO2 carbon component. I assume that you have an overall understanding about WSO2 carbon platform and how it works.

      First of all I give you a brief introduction about the award-winning WSO2 carbon platform. It is a component-based, service oriented platform for the enterprise-grade WSO2 middleware products stack. It is 100% open source and delivered under Apache License 2.0. The WSO2 Carbon platform is lean, high-performant and consists of a collection of OSGi bundles.  

      The WSO2 Carbon core platform hosts a rich set of middleware components encompassing capabilities such as security, clustering, logging, statistics, management and more. These are basic features required by all WSO2 products that are developed on top of the base platform.

      All WSO2 products are a collection of Carbon components. They have been developed simply by plugging various Carbon components that provide different features. The WSO2 Carbon component manager provides the ability to extend the Carbon base platform, by selecting the components that address your unique requirements and installing them with point-and-click simplicity. As a result, by provisioning this innovative base platform, you can develop your own, lean middleware product that has remarkable flexibility to change as business requirements change.

      Once you have the basic knowledge on how the architecture works in carbon, you can start implementing a Carbon component. Before move on to any coding stuff first look at the prerequisites that we need to implement the carbon component using eclipse IDE.

       Prerequisites
      • Java
      • Maven
      • Any WSO2 carbon product (Here I use WSO2 Application Server)
      • Eclipse (or you can use IdeaJ as well)
      Scenario

      Suppose we have a simple object called OrderBean for storing order details in the back-end component and let’s try to display those information at the front-end UI.

      Creating the Project Structure

      Now I will explain about the project structure to implement the carbon component. Here I'm going to create an Order Process carbon component using ecpilse. This will consists of two parts called back-end runtime and front-end console UI. First look at how to implement back-end runtime.

      As a first step I will create a maven project. (Before that, you should have install maven plugin to the eclipse)

      File -> New -> Other -> Maven Project (Inside of the Maven folder)


      Then click Next and you will see the fallowing UI.


      Click Next and then select the appropriate archetype to create the project structure. Here I will use default project structure. And again click Next.


      Now I will have to specify Archetype parameters for my maven project. See the fallowing figure to setup those parameters (Please change the version to 1.0.0-SNAPSHOT). And then click Finish.


      Makesure that packaging type is bundle in the pom.xml file. (Because both backend and frontend must package as OSGi bundle in carbon). I'm using maven-bundle-plugin to do that.

      <groupId>org.wso2.carbon</groupId>
      <artifactId>org.wso2.carbon.example.OrderProcess</artifactId>
      <version>1.0.0-SNAPSHOT</version>
      <packaging>bundle</packaging>
       
      This will be an OSGI bundle. So, I have to configure the Apache Felix plugin to set up the configurations.

             <build>
        <plugins>
         <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>1.4.0</version>
          <extensions>true</extensions>
          <configuration>
           <instructions>
            <Bundle-SymbolicName>${pom.artifactId}</Bundle-SymbolicName>
            <Bundle-Name>${pom.artifactId}</Bundle-Name>
            <Export-Package>
             org.wso2.carbon.example.OrderProcess.*
            </Export-Package>
           </instructions>
          </configuration>
         </plugin>
        </plugins>
       </build>

      Since I'm using Carbon registry to store the items of the OrderBean, following dependencies should be added to the back-end project. (Remember to use byte arrays when you are storing the values in the Carbon registry)

      <dependencies>  
           <dependency>  
             <groupId>org.wso2.carbon</groupId>  
             <artifactId>org.wso2.carbon.registry.core</artifactId>  
             <version>4.2.0</version>  
           </dependency>  
           <dependency>  
             <groupId>org.wso2.carbon</groupId>  
             <artifactId>org.wso2.carbon.registry.api</artifactId>  
             <version>4.2.0</version>  
           </dependency>  
       </dependencies>  

      After adding the dependencies and the plugins, pom.xml file of the back-end will be similar to following pom. (If your project have an error then you should have to update the project such that right click the project then select Maven -> Update Project)

      (You should have to change the value of the Export-Package element in your pom.xml file according to the package structure)

      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
      
       <groupId>org.wso2.carbon</groupId>
       <artifactId>org.wso2.carbon.example.OrderProcess</artifactId>
       <version>1.0.0-SNAPSHOT</version>
       <packaging>bundle</packaging>
      
       <name>WSO2 Carbon - Order Process</name>
       <url>http://maven.apache.org</url>
      
       <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       </properties>
      
       <!-- <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> 
        <version>3.8.1</version> <scope>test</scope> </dependency> </dependencies> -->
      
       <build>
        <plugins>
         <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>1.4.0</version>
          <extensions>true</extensions>
          <configuration>
           <instructions>
            <Bundle-SymbolicName>${pom.artifactId}</Bundle-SymbolicName>
            <Bundle-Name>${pom.artifactId}</Bundle-Name>
            <Export-Package>
             org.wso2.carbon.example.OrderProcess.*
            </Export-Package>
           </instructions>
          </configuration>
         </plugin>
        </plugins>
       </build>
      
       <dependencies>
        <dependency>
         <groupId>org.wso2.carbon</groupId>
         <artifactId>org.wso2.carbon.registry.core</artifactId>
         <version>4.2.0</version>
        </dependency>
        <dependency>
         <groupId>org.wso2.carbon</groupId>
         <artifactId>org.wso2.carbon.registry.api</artifactId>
         <version>4.2.0</version>
        </dependency>
       </dependencies>
      
       <repositories>
        <repository>
         <id>wso2-nexus</id>
         <name>WSO2 internal Repository</name>
         <url>http://maven.wso2.org/nexus/content/groups/wso2-public/</url>
         <releases>
          <enabled>true</enabled>
          <updatePolicy>daily</updatePolicy>
          <checksumPolicy>ignore</checksumPolicy>
         </releases>
        </repository>
       </repositories>
      
       <pluginRepositories>
        <pluginRepository>
         <id>wso2-maven2-repository</id>
         <url>http://dist.wso2.org/maven2</url>
        </pluginRepository>
        <pluginRepository>
         <id>wso2-maven2-snapshot-repository</id>
         <url>http://dist.wso2.org/snapshots/maven2</url>
        </pluginRepository>
       </pluginRepositories>
      
      </project>
      

      Create the back-end service

      I already created a service class called ProcessOrderService inside the package org.wso2.carbon.example.OrderProcess. This service consists of two methods. One for processing the order and the other is for canceling the order.

      Before creating the service class I already created a package called org.wso2.carbon.example.OrderProcess.data to hold my data objects called OrderBean, Item, Address, Customer.

      Now I will show my OrderBean class implementation below and you will see that it implements the Serializable interface, since because I'm going to use Carbon registry to store the OrderBean objects in the carbon registry.

      package org.wso2.carbon.example.OrderProcess.data;
      
      import java.io.Serializable;
      
      public class OrderBean implements Serializable{
       private Customer customer;
       private Address shippingAddress;
       private Item[] orderItems;
       private String orderID;
       private double totalPrice;
      
       /**
        * @return customer
        */
       public Customer getCustomer() {
        return customer;
       }
      
       public void setCustomer(Customer customer) {
        this.customer = customer;
       }
      
       public Address getShippingAddress() {
        return shippingAddress;
       }
      
       public void setShippingAddress(Address shippingAddress) {
        this.shippingAddress = shippingAddress;
       }
      
       public Item[] getOrderItems() {
        return orderItems;
       }
      
       public void setOrderItems(Item[] orderItems) {
        this.orderItems = orderItems;
       }
      
       public String getOrderID() {
        return orderID;
       }
      
       public void setOrderID(String orderID) {
        this.orderID = orderID;
       }
      
       public double getPrice() {
        return totalPrice;
       }
      
       public void setPrice(double price) {
        this.totalPrice = price;
       }
      
      }

      package org.wso2.carbon.example.OrderProcess.data;
      
      import java.io.Serializable;
      
      public class Customer implements Serializable{
       private String custID;
       private String firstName;
       private String lastName;
      
       public String getCustID() {
        return custID;
       }
      
       public void setCustID(String custID) {
        this.custID = custID;
       }
      
       public String getFirstName() {
        return firstName;
       }
      
       public void setFirstName(String firstName) {
        this.firstName = firstName;
       }
      
       public String getLastName() {
        return lastName;
       }
      
       public void setLastName(String lastName) {
        this.lastName = lastName;
       }
      
      }

      package org.wso2.carbon.example.OrderProcess.data;
      
      import java.io.Serializable;
      
      public class Address implements Serializable{
       private String streetName;
       private String cityName;
       private String stateCode;
       private String country;
       private String zipCode;
      
       public String getStreetName() {
        return streetName;
       }
      
       public void setStreetName(String streetName) {
        this.streetName = streetName;
       }
      
       public String getCityName() {
        return cityName;
       }
      
       public void setCityName(String cityName) {
        this.cityName = cityName;
       }
      
       public String getStateCode() {
        return stateCode;
       }
      
       public void setStateCode(String stateCode) {
        this.stateCode = stateCode;
       }
      
       public String getCountry() {
        return country;
       }
      
       public void setCountry(String country) {
        this.country = country;
       }
      
       public String getZipCode() {
        return zipCode;
       }
      
       public void setZipCode(String zipCode) {
        this.zipCode = zipCode;
       }
      
      }

      package org.wso2.carbon.example.OrderProcess.data;
      
      import java.io.Serializable;
      
      public class Item implements Serializable{
       private String itemName;
       private String itemID;
       private double unitPrice;
       private int quantity;
      
       public String getItemName() {
        return itemName;
       }
      
       public void setItemName(String itemName) {
        this.itemName = itemName;
       }
      
       public String getItemID() {
        return itemID;
       }
      
       public void setItemID(String itemID) {
        this.itemID = itemID;
       }
      
       public int getQuantity() {
        return quantity;
       }
      
       public void setQuantity(int quantity) {
        this.quantity = quantity;
       }
      
       public double getUnitPrice() {
           return unitPrice;
          }
      
       public void setUnitPrice(double unitPrice) {
           this.unitPrice = unitPrice;
          }
      
      }

      Now you can see my service class implementation below.

      package org.wso2.carbon.example.OrderProcess;
      
      import java.io.ByteArrayInputStream;
      import java.io.ByteArrayOutputStream;
      import java.io.IOException;
      import java.io.ObjectInputStream;
      import java.io.ObjectOutputStream;
      import java.util.ArrayList;
      import java.util.List;
      import java.util.logging.Logger;
      
      import org.wso2.carbon.context.CarbonContext;
      import org.wso2.carbon.context.RegistryType;
      import org.wso2.carbon.example.OrderProcess.data.Item;
      import org.wso2.carbon.example.OrderProcess.data.OrderBean;
      import org.wso2.carbon.registry.api.Registry;
      import org.wso2.carbon.registry.api.RegistryException;
      import org.wso2.carbon.registry.api.Resource;
      
      
      public class ProcessOrderService {
       private final static Logger LOGGER = Logger.getLogger(ProcessOrderService.class.getName());
      
       private List<OrderBean> orderList = new ArrayList<OrderBean>();
       private int orderCounter = 0;
       private double totalAmount = 0;
       private Registry registry = null;
       private static final String ORDER_PATH = "order_location";
       
       public ProcessOrderService(){
        registry = CarbonContext.getThreadLocalCarbonContext().getRegistry(RegistryType.valueOf(RegistryType.LOCAL_REPOSITORY.toString()));
       }
      
       /**
        * Acquire the order
        * 
        * @param orderBean
        * @return OrderBean object
        */
       public OrderBean processOrder(OrderBean orderBean) {
      
        // Number of items ordered
        if (orderBean.getOrderItems() != null) {
         // Set the order ID.
         orderBean.setOrderID("ABC-" + (orderCounter++));
         try {
                   Resource orderRes = registry.newResource();
                   orderRes.setContent(serialize(orderBean.getOrderItems()));  
                   registry.put(ORDER_PATH, orderRes);
                   
                   Resource getItemsRes = registry.get(ORDER_PATH);
                   Item[] items = (Item[]) deserialize((byte[]) getItemsRes.getContent());
                   
                   for (Item item : items) {
           double totalItemCost = item.getUnitPrice() * item.getQuantity();
           totalAmount += totalItemCost;
          }
                   
                   // set the total price
          orderBean.setPrice(totalAmount);
          orderList.add(orderBean);
          
                   return orderBean;
                  } catch (RegistryException e) {
                   e.printStackTrace();
                  } catch (IOException e) {
                   e.printStackTrace(); 
                  } catch (ClassNotFoundException e) {
                   e.printStackTrace();
                  }
       
        }
      
        return new OrderBean();
       }
      
       /**
        * Delete the given order
        * 
        * @param orderID
        * @return boolean to check weather order is deleted or not
        */
       public boolean cancelOrder(String orderID) {
        LOGGER.info("cancelOrder method starting");
      
        for (OrderBean orderBean : orderList) {
      
         if (orderBean.getOrderID().equals(orderID)) {
          LOGGER.info("canceling OrderBean Processing");
          orderList.remove(orderBean);
          return true;
         }
        }
      
        LOGGER.info("cancelProcssing over");
        return false;
       }
       
       private static byte[] serialize(Object obj) throws IOException {
              ByteArrayOutputStream b = new ByteArrayOutputStream();
              ObjectOutputStream o = new ObjectOutputStream(b);
              o.writeObject(obj);
              return b.toByteArray();
          }
      
          private static Object deserialize(byte[] bytes) throws IOException, ClassNotFoundException {
              ByteArrayInputStream b = new ByteArrayInputStream(bytes);
              ObjectInputStream o = new ObjectInputStream(b);
              return o.readObject();
          }
      }

      (If you have App.java class inside your service package please remove it. )

      Now I should have to write the service configuration (services.xml) for my service implementation. For that first create a folder called resources inside the src/main/. Then create a folder called META-INF inside the resources folder. Inside the META-INF folder create services.xml file with following content. Change the service and service class names according to your project.

      <serviceGroup>
          <service name="ProcessOrderService" scope="transportsession">
              <transports>
                  <transport>https</transport>
              </transports>
              <parameter name="ServiceClass">org.wso2.carbon.example.OrderProcess.ProcessOrderService</parameter>
          </service>
          
          <parameter name="adminService" locked="true">true</parameter>
          <parameter name="hiddenService" locked="true">true</parameter>
          <parameter name="AuthorizationAction" locked="true">/permission/admin/protected</parameter>
      </serviceGroup>

      Now go to the pom.xml file location of the back-end project using command line interface and type mvn clean install to build the project. If the build get success you will get a jar file like org.wso2.carbon.example.OrderProcess-1.0.0-SNAPSHOT.jar inside the target directory. Then copy the created jar file to repository/components/dropins directory in the WSO2 Application server. 

      We can't see the WSDL file of the created service directly accessing the url (http://192.168.1.2:9765/services/ProcessOrderService?wsdl) after running the application server. That is because I have added this as a admin service and by default admin services WSDLs are hidden. In order to view the WSDL file open the carbon.xml file in the repository/conf and set the value of HideAdminServiceWSDLs as false.

      <HideAdminServiceWSDLs>false</HideAdminServiceWSDLs>  

      Now start the WSO2 Application Server and put the above URL in the browser (last part should be the Service name that you provide in the services.xml). Save the WSDL file in your computer to use it for front-end project.

       Create the front-end console UI

       Now I will create the front-end project like above (maven project) and edit the pom.xml file as below. Inside of this pom file you can see that I've used the previously saved WSDL file. Do the necessary modifications to the pom file according to the your project.
      • org.wso2.carbon.example.OrderProcess.ui
        • artifactId - org.wso2.carbon.example.OrderProcess.ui
        • packaging - bundle
        • name - WSO2 Carbon - Order Process
        • plugin - maven-bundle-plugin
      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
      
       <groupId>org.wso2.carbon</groupId>
       <artifactId>org.wso2.carbon.example.OrderProcess.ui</artifactId>
       <version>1.0.0-SNAPSHOT</version>
       <packaging>bundle</packaging>
      
       <name>WSO2 Carbon - Order Process</name>
       <url>http://maven.apache.org</url>
      
       <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       </properties>
      
       <!-- <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> 
        <version>3.8.1</version> <scope>test</scope> </dependency> </dependencies> -->
      
       <dependencies>
        <dependency>
         <groupId>org.apache.axis2.wso2</groupId>
         <artifactId>axis2</artifactId>
         <version>1.6.1.wso2v4</version>
        </dependency>
        <dependency>
         <groupId>org.apache.stratos</groupId>
         <artifactId>org.wso2.carbon.ui</artifactId>
         <version>4.2.0-stratos</version>
        </dependency>
       </dependencies>
      
       <build>
      
        <plugins>
         <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <configuration>
           <source>1.5</source>
           <target>1.5</target>
          </configuration>
         </plugin>
         <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-antrun-plugin</artifactId>
          <version>1.1</version>
          <executions>
           <execution>
            <id>source-code-generation</id>
            <phase>process-resources</phase>
            <goals>
             <goal>run</goal>
            </goals>
            <configuration>
             <tasks>
              <java classname="org.apache.axis2.wsdl.WSDL2Java" fork="true">
               <arg
                line="-uri src/main/resources/OrderProcess.wsdl -u -uw -o target/generated-code
                                          -p org.wso2.carbon.example.OrderProcess.ui
                                          -ns2p http://org.apache.axis2/xsd=org.wso2.carbon.example.OrderProcess.ui.types.axis2,http://OrderProcess.example.carbon.wso2.org=org.wso2.carbon.example.OrderProcess.ui,http://data.OrderProcess.example.carbon.wso2.org/xsd=org.wso2.carbon.example.OrderProcess.ui.types.data" />
               <classpath refid="maven.dependency.classpath" />
               <classpath refid="maven.compile.classpath" />
               <classpath refid="maven.runtime.classpath" />
              </java>
             </tasks>
            </configuration>
           </execution>
          </executions>
         </plugin>
         <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>build-helper-maven-plugin</artifactId>
          <executions>
           <execution>
            <id>add-source</id>
            <phase>generate-sources</phase>
            <goals>
             <goal>add-source</goal>
            </goals>
            <configuration>
             <sources>
              <source>target/generated-code/src</source>
             </sources>
            </configuration>
           </execution>
          </executions>
         </plugin>
         <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>1.4.0</version>
          <extensions>true</extensions>
          <configuration>
           <instructions>
            <Bundle-SymbolicName>${pom.artifactId}</Bundle-SymbolicName>
            <Export-Package>
             org.wso2.carbon.example.OrderProcess.ui.*
            </Export-Package>
            <Import-Package>
             !javax.xml.namespace,
             javax.xml.namespace;version="0.0.0",
             *;resolution:=optional,
            </Import-Package>
            <Carbon-Component>UIBundle</Carbon-Component>
           </instructions>
          </configuration>
         </plugin>
        </plugins>
      
       </build>
      
       <repositories>
        <repository>
         <id>wso2-nexus</id>
         <name>WSO2 internal Repository</name>
         <url>http://maven.wso2.org/nexus/content/groups/wso2-public/</url>
         <releases>
          <enabled>true</enabled>
          <updatePolicy>daily</updatePolicy>
          <checksumPolicy>ignore</checksumPolicy>
         </releases>
        </repository>
       </repositories>
       
       <pluginRepositories>
        <pluginRepository>
         <id>wso2-maven2-repository</id>
         <url>http://dist.wso2.org/maven2</url>
        </pluginRepository>
        <pluginRepository>
         <id>wso2-maven2-snapshot-repository</id>
         <url>http://dist.wso2.org/snapshots/maven2</url>
        </pluginRepository>
       </pluginRepositories>
       
      </project>

      Now go to the pom.xml file location of the front-end project using command line interface and type mvn compile to compile the project. (It will download the necessary dependencies and then compile the classes as well)

      As the next step I will create the Client called OrderProcessClient inside the org.wso2.carbon.example.OrderProcess.ui package, which will use the generated stub to access the back-end service which I created above.

      package org.wso2.carbon.example.OrderProcess.ui;
      
      import java.rmi.RemoteException;
      
      import org.apache.axis2.client.Options;
      import org.apache.axis2.client.ServiceClient;
      import org.apache.axis2.context.ConfigurationContext;
      import org.wso2.carbon.example.OrderProcess.ui.ProcessOrderServiceStub;
      import org.wso2.carbon.example.OrderProcess.ui.types.data.OrderBean;
      
      public class OrderProcessClient {
      
       private ProcessOrderServiceStub stub;
      
       public OrderProcessClient(ConfigurationContext configCtx, String backendServerURL,
                                   String cookie) throws Exception {
        String serviceURL = backendServerURL + "ProcessOrderService";
        stub = new ProcessOrderServiceStub(configCtx, serviceURL);
        ServiceClient client = stub._getServiceClient();
        Options options = client.getOptions();
        options.setManageSession(true);
        options.setProperty(org.apache.axis2.transport.http.HTTPConstants.COOKIE_STRING, cookie);
       }
      
       public OrderBean processOrder(OrderBean orderBean) throws Exception {
        try {
         return stub.processOrder(orderBean);
        } catch (RemoteException e) {
         String msg = "Cannot process the order" + " . Backend service may be unvailable";
         throw new Exception(msg, e);
        }
       }
      
       public boolean cancelOrder(String orderID) throws Exception {
        try {
         return stub.cancelOrder(orderID);
        } catch (RemoteException e) {
         String msg = "Cannot cancel the order" + " . Backend service may be unvailable";
         throw new Exception(msg, e);
        }
       }
      }

      Like I mentioned above in back-end project you will need to create resouces folder inside of the <folder-name>/src/main/ folder of your front-end  project. After that create a folder called web inside of the resource folder. Inside this web folder, create another directory and named it as orderprocess-mgt.

      Create a .jsp file called orderprocessmanager.jsp inside of the orderprocess-mgt directory. This is the jsp page that consist of the UI part. I will have a table of existing orders.

      <%@ page import="org.apache.axis2.context.ConfigurationContext" %>
      <%@ page import="org.wso2.carbon.CarbonConstants" %>
      <%@ page import="org.wso2.carbon.ui.CarbonUIUtil" %>
      <%@ page import="org.wso2.carbon.utils.ServerConstants" %>
      <%@ page import="org.wso2.carbon.ui.CarbonUIMessage" %>
      <%@ page import="org.wso2.carbon.example.OrderProcess.ui.OrderProcessClient" %>
      <%@ page import="org.wso2.carbon.example.OrderProcess.ui.types.data.OrderBean" %>
      <%@ page import="org.wso2.carbon.example.OrderProcess.ui.types.data.Customer" %>
      <%@ page import="org.wso2.carbon.example.OrderProcess.ui.types.data.Address" %>
      <%@ page import="org.wso2.carbon.example.OrderProcess.ui.types.data.Item" %>
      <%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %>
      <%@ taglib uri="http://wso2.org/projects/carbon/taglibs/carbontags.jar" prefix="carbon" %>
      <%
              String serverURL = CarbonUIUtil.getServerURL(config.getServletContext(), session);
              ConfigurationContext configContext =
                      (ConfigurationContext) config.getServletContext().getAttribute(CarbonConstants.CONFIGURATION_CONTEXT);
              String cookie = (String) session.getAttribute(ServerConstants.ADMIN_SERVICE_COOKIE);
      
              OrderProcessClient client;
              OrderBean order;
              OrderBean orderBean = new OrderBean();
      
        Customer customer = new Customer();
        customer.setCustID("A123");
        customer.setFirstName("Isuru");
        customer.setLastName("Wijesinghe");
        orderBean.setCustomer(customer);
      
        Address address = new Address();
        address.setCityName("Colombo");
        address.setCountry("Sri Lanka");
        address.setStateCode("04");
        address.setStreetName("Armer Street");
        address.setZipCode("02");
        orderBean.setShippingAddress(address);
      
        Item item1 = new Item();
        item1.setItemID("11");
        item1.setItemName("MACBook");
        item1.setQuantity(12);
        item1.setUnitPrice(100);
      
        Item item2 = new Item();
        item2.setItemID("10");
        item2.setItemName("UltrasBook");
        item2.setQuantity(10);
        item2.setUnitPrice(30);
      
        Item[] orderItems = { item1, item2 };
      
        orderBean.setOrderItems(orderItems);
      
              try {
                  client = new OrderProcessClient(configContext, serverURL, cookie);
                  order = client.processOrder(orderBean);
              } catch (Exception e) {
                  CarbonUIMessage.sendCarbonUIMessage(e.getMessage(), CarbonUIMessage.ERROR, request, e);
      %>
                  <script type="text/javascript">
                         location.href = "../admin/error.jsp";
                  </script>
      <%
                  return;
          }
      %>
      
      <div id="middle">
       <h2>Order Process Management</h2>
      
          <div id="workArea">
        <table class="styledLeft" id="moduleTable">
                      <thead>
                      <tr>
                          <th width="20%">Customer ID</th>
                          <th width="20%">First Name</th>
                          <th width="20%">Last Name</th>
                          <th width="20%">Order Price</th>
                          <th width="20%">Number Of Items</th>
                      </tr>
                      </thead>
                      <tbody>
                 <%
                   
                 %>
                   <tr>
                       <td><%=order.getCustomer().getCustID()%></td>
                       <td><%=order.getCustomer().getFirstName()%></td>
                       <td><%=order.getCustomer().getLastName()%></td> 
                       <td><%=order.getPrice()%></td> 
                       <td><%=order.getOrderItems().length%></td>              
                      </tr>
                 <%
                   
                 %>
                      </tbody>
               </table>
          </div>
      </div

      Here you can see that I've used some style classes and IDs. Those are predefined classes and IDs in the Carbon. Don't forget to import the carbon tag library as well.

      Now I will have to add the UI component to the menu bar as a menu item of the application server. For that you must create the component.xml file. Befeore creating it first you should have to create META-INF inside the resources folder in front-end project and then create the component.xml file inside of it as below.

      <component xmlns="http://products.wso2.org/carbon">
          <menus>
              <menu>
                  <id>orderprocess_menu</id>
                  <i18n-key>orderprocess.menu</i18n-key>
                  <i18n-bundle>org.wso2.carbon.example.OrderProcess.ui.i18n.Resources</i18n-bundle>
                  <parent-menu>manage_menu</parent-menu>
                  <link>../orderprocess-mgt/orderprocessmanager.jsp</link>
                  <region>region1</region>
                  <order>50</order>
                  <style-class>manage</style-class>
                  <!--  --><icon>../log-admin/images/log.gif</icon>-->
                  <require-permission>/permission/protected/manage</require-permission>
              </menu>
          </menus>
      </component>

      Here i18n-bundle value depend on the package that the created Client resides. Create folder structure according to the package name inside the web folder. As an example I created the package called org.wso2.carbon.example.OrderProcess.ui to hold the client code. Therefore I must have to create a directory structure similar to the package name of the client code and inside of it create another directory called i18n. Then inside of it create a resource bundle called Resources.properties (create a empty file and named it as Resources.properties) inside the above created folder. Then update the file contend as below.

      orderprocess.menu=Order Process

      (This is similar to the i18n-key value inside of the component.xml and assign it a name and it is the menu item name in the menu bar of the application server that you can see after deploying it. Here I mentioned it as Order Process)

      Now go to the pom.xml file location of the front-end project and type maven clean install in the command line interface.

      Deploying the component


      Now you copy the generated jar files inside the target folder in both back-end  front-end projects in to the dropins folder that I mentioned previously and restart the WSO2 Application Server. Then under the Services (in the main menu tab) you can see your menu item name called Order Process. Once you click it you can see the fallowing output.


      Saturday, May 2, 2015

      Calabash an automation test framework for mobile development


      Calabash uses to write and execute automated acceptance tests for mobile applications. It helps for both android and IOS applications. Calabash contains set of libraries that enable set of test code to programmatically interact with applications. Each action can be one of the fallows. 
      • Gestures (Touches or gestures. As an example tap, swipe and rotate)
      • Assertions (As an example there should be a "Login" button or the web view should contain an "<h1>" element with the any text)
      • Screenshots (screen dump the current view on the current device model)

      Calabash provides set of APIs that support for apps running on touch screen devices. Calabash consists of two libraries called calabash-android and calabash-ios. Calabash-android is the automation and testing library for Android, and similarly calabash-ios is for ios. These two libraries are the low level libraries that allow the Cucumber tool to run automated functional tests on mobile devices as well as simulators. Using these two libraries test can be written in natural language. For an example below showing you a part of tests that written with calabash.
      Feature: Rating a stand
        Scenario: Find and rate a stand from the list
          Given I am on the List
          Then I should see a "rating" button
          And I should not see “Sandwich & Soup”
          And take picture
          Then I touch the "rating" button
          And I should see “Sandwich & Soup”
          And take picture
          When I touch "Sandwich & Soup"
          Then I should see details for “Sandwich & Soup”
          When I touch the "click" button
          Then I should see the rating panel
          Then I touch "star4
          And I touch "button"
          And take picture


      This example is called a feature file in Calabash. It describes the intended behavior of the application. In this example as you can see each line after Scenario corresponds to a step. In calabash a step does one of three things: makes a user action, makes an assertion or takes a screenshot.
       

      Thursday, October 16, 2014

      Evaluating HCI Aspects of a website


      Introduction

      Usability is a fundamental concept for Interaction Design research and practice, since the dawn of Human-Computer Interaction as an inter-disciplinary endeavor. It satisfies users’ needs in an efficient and effective way.

      The prominence of HCI in the current and future of website development is not to be taken lightly. It has been shown that a large percentage of the design and programming strength of projects go into the actual website design. The interface is a fundamental part of making the site more successful, safe, useful, and functional and in the long run, more agreeable for the user.

      The tools and techniques that have been developed have contributed vastly towards reducing costs of design and development and increasing productivity. Savings have been created through decreased task time, fewer user errors, greatly reduced user disruption, reduced load on support teams or staffs, the elimination of training, and avoidance of changes in maintenance and redesign costs. However the iterative design has been proven now to be more efficient and cost effective, and this is why software usability comes into the picture form the early phases of design; it is also checked throughout the different production processes. This comes in accordance with the user-centered design, and as pointed by, software products including websites only meet the goals of their producers once they have met those of the users; hence businesses should align the two and ensure that their software products suites the key scenarios of use.

      Our report is structured as follows. Next section we will begin with a brief on website usability guidelines and then we will describe the different evaluation methods. At there we will give brief description about two evaluation methods namely heuristic evaluation and task based evaluation. After that we will then have a case study for an existing website. At there we will break down it into two parts. In part one we will describe a brief introduction on the website that we selected for our evaluation. And the next part we will present our results for the website we selected for evaluation using the two evaluation methods (Heuristic and Task based evaluation). Finally to overcome issues and problems (to reduce issues in some certain level) that we identified from evaluation we will present our alternative design in the next section.

      Website usability guidelines

      The research who was done by Neilson and Hackos clearly emphasized that for software to be usable, it has to comply with the following five attributes namely memorability, efficiency, learnability, user’s satisfaction and few errors. Our key objective here is to study usability of web content; we better focus on how those attributes were further explained by Nielsen, after adapting them for web navigation and other main functionalities in the selected website (Venosc). According to Neilson and Hackos research they summarized above attributes as fallows. Users should be able to learn the basic navigation options of a web page and to find their desired content easily. They also should be able to reach such content in an efficient and quick way, and be able to remember the navigation options if they happen to return to the website after a while. Users should also be guided to follow the right links until they reach their preferred content, and if they happen to make mistakes during this process, they should be able to recover by going back to their previous page or content effortlessly.

      But when we see beyond there are divergent factors that make website navigation problematic. The content and the structure of the websites are typically built after the inner structure of the content providers, rather than the users' needs. Moreover the content itself often needs to be upgraded in the web, specially when it is originally intended to be provided in a printed form. Finally, web pages are sometimes not subject to the same quality measures for printed materials, due to the lower cost of creating the former. To motivate users for efficient access to the contents large websites also we can provide shortcut keys in parallel to the navigation procedure for experienced users.

      Evaluation Guidelines

      Usability evaluation can broadly be categorized into inspection and testing methods. In the former, experts study and report usability issues, while in the latter, issues are discovered by observing the users during their interaction with the interface. Four narrower evaluation categories were listed by Nielsen in 1994: formal, informal, empirical and automatic. We will focus on the informal and empirical evaluations here, as the first one has been abandoned in favour of the formal methods and/or grouped into other methods such as cognitive walk, which is beyond the scope of our evaluation.

      Heuristic Evaluation (Informal)

      Heuristic evaluation is a good method of identifying both major and minor problems with an interface. In heuristic evaluation method a small group of experts are asked to evaluate the interface in accordance with a set of usability criterions or heuristics. They added that each of evaluator should conduct the evaluation independently of the others, and they should only be allowed to communicate after completing their evaluation. Under our heuristic evaluation we followed by Nielsen’s ten heuristic guidelines presented as follows.
        • Visibility of System Status
        • Match between the system and the real world
        • User control and freedom
        • Consistency and Standards
        • Error Prevention
        • Recognition rather than recall
        • Flexibility and efficiency of use
        • Aesthetic and minimalist design
        • User can recognize and recover from error
        • Documentation and help

          Normally we experts or evaluators begin heuristic evaluation by visiting the system to get a general feel of its flow and then they should revisit it at least one more time to evaluate the specific components of its interface and their functionalities. As an example you can see that as below.
            

          Problem Found
          Severity
          Heuristic Number
          Violated Heuristic
          Translation for languages doesn't work properly
          3
          #5
          Error Prevention

          Here rating usability problems according to their severity facilitate the allocation of resources to fix the most serious problems. Severity ratings are a combination of frequency, impact, and persistence. Using heuristic evaluation it is easy and cheap and can help in finding many usability issues quickly. But however some issues can be missed, especially the business-specific or domain-specific ones.
          Task based evaluation (Empirical)

          Task based evaluation is part of a formative evaluation to redesign the interface, or the whole application, based on users' responses, experiences and problems. Users carry out some specific tasks which are given by experts or evaluators and it involves the major system functionalities. While the users are doing their tasks, evaluators / observers should document the results such as the number of tasks accomplished, time taken and/or number of pages navigated by each user. And also observers can give help to users to accomplish the given tasks, but however they should take notice with the incidents where help was needed. They also suggested that discussions taking place between the participants afterwards are helpful in collecting their reactions and suggestions for improvements. Also, the observers may ask the participants to be more verbose and explain what they are thinking about at the moment, the actions they are trying to take, and why such kind of actions takes.

          Usually involve five participants for this kind of evaluation. The number of participants is not the only decisive factor here but also the design of the tasks, their goals, the diversity of the participants and their skills play a role in the number of their usability findings. Below we listed the five requirements for a good task based usability test.
            • Setting the test goals
            • Setting the characteristics of the participating sample
            • Setting the scenarios and tasks to be done
            • Setting the measurement criteria
            • Setting the testing environment and needed materials