Oracle SOA: Sending delayed JMS messages

Sometimes you might want to put something on a JMS queue and make it available after a certain period has passed to consumers. How can you achieve this using Oracle SOA Suite?

Dramatically reduce SOA Suite 11g startup time by cleaning the MDS

SOA Suite can sometimes be a bit slow to start. This is especially the case when there are a lot of composites to load. Customers using different versions of composites can benefit from undeploying non-default revisions of processes which do not have any running instances (see for example here). Undeployment in most cases is an asynchronous process which does not give feedback. It can partially fail without you noticing (apparently not an atomic transaction). This sometimes leaves composite remains; parts of the composite which are still loaded at startup but are not visible from the Enterprise Manager. Removing these can dramatically reduce server startup time. Especially in an environment which has been used for some time and environments with many versions of composites. Reducing the time required to get the soa-infra application fully up and running is of course mostly relevant for 11g SOA installations and less for 12.1.3 (which does some lazy loading) and 12.2.1 (which supports parallel deployments, also during server start-up).

In this article I'll demonstrate how these left-over composite parts can be identified and removed on an 11.1.1.7 SOA environment. First try this procedure on a development or test environment before executing it in production! This method is not supported by Oracle (or me) in any way and using it is entirely at your own risk. If something breaks, tell me so I can update this article. Thanks!

Please mind that these actions, although they help with the start time and memory usage of your SOA environment, have less impact on run-time performance than for example purging of instances and reducing the amount of deployed composites (or tweaking datasources, soa-infra database, JVM, etc).

SOA Suite can be up quickly!

A first look at Splunk. Monitor Oracle SOA Suite service response times

Measuring performance of services can be done in various ways. In this blog I will describe a method of measuring Oracle SOA service response times with Splunk a popular monitoring tool. In order to monitor service response times with Splunk, Splunk needs to obtain its data from somewhere. In this example I'll use the HTTP access log which I expand with a time-taken field. Disclaimer; my experience with Splunk is something like 2 hours. This might also be an indication of what can quickly be achieved with Splunk with little knowledge.

A first look at Oracle Business Transaction Management (BTM)

Monitoring and debugging Oracle SOA Suite environments is often a topic not paid much attention to. The people who write the software are often not much involved in the maintenance of the running software and the maintenance people do not have much application knowledge.

Usually Oracle SOA applications are composed of several components which interact with each other. These components are often of different technologies. Personally I tend to use databases, BPEL processes and Java Webservices a lot. For the maintenance people it is difficult to understand all application call chains and debug these in case of problems.

I was curious if the Oracle BTM product would provide a solution for this; http://www.oracle.com/technetwork/oem/btm-496775.html. BTM is part of the SOA Management Pack Enterprise Edition. It used to be a product from AmberPoint before Oracle bought them in 2010. I've used SOA Suite 11.1.1.7 with BTM 12.1.0.4.1 on an XE 11r2 Db for this tryout. I did not read much documentation about this product or followed any courses, so it would also be a test of intuitiveness.

Receiving JSON requests in Oracle BPEL by using the SocketAdapter

Some frameworks prefer the use of JSON, such as webinterface frameworks, GIS frameworks or mobile app frameworks. In order to integrate and provide for example service orchestration, Oracle SOA Suite can be used. Oracle SOA Suite however does not have a JSONAdapter in the current release.

Using the HTTPAdapter is not an option to provide JSON support. The Oracle HTTPAdapter does not support receiving and sending JSON (only XML). It can however be used to send a HTTP GET request with parameters which is how REST services are often called. See for example; http://shrikworld.blogspot.nl/2011/04/http-adapter-in-soa-11g.html

The SocketAdapter however does not have the XML limitations of the HTTPAdapter. In this post I describe how the SocketAdapter can be used to receive a JSON message from an HTTP Post request.

Implementation

SocketAdapter

To use the  SocketAdapter for this usecase requires some parsing of the HTTP communication which is usually abstracted. It is possible to use XSLT or a custom Java class to parse the request received from the adapter and transform it to XML (http://docs.oracle.com/cd/E21764_01/integration.1111/e10231/adptr_sock.htm#BABHECAG). I decided to try the XSLT option.

I downloaded an example from; http://java.net/projects/oraclesoasuite11g/pages/SocketAdapters

In this example I needed to change the following to get it to work;
- update adf-config.xml with the correct path of the jdeveloper/integration folder
- make sure bpm-infra.jar is in the classpath of the server

Based on the configuration of the SocketAdapter connection factory I tried connecting to port 12110 with telnet.

In the logfile I saw the following;

- Socket Adapter ClientProcessor:run() Error occured in processing client request
- Socket Stylesheet Parsing Error.
Error while trying to parse the stylesheet.
Please ensure that that the stylesheet exists and is configured properly. Please cross check the extension functions. Contact Oracle support if error is not fixable. 

This was as expected since I didn't provide the correct test message.

When I opened in my browser; http://localhost:12110 it did work however. I noticed the input was;

0.9,*/*

I wondered where this came from. I created a custom Java socket listener based on http://docs.oracle.com/javase/tutorial/networking/sockets/readingWriting.html and http://zerioh.tripod.com/ressources/sockets.html and did the request on the listener. The result was;

client>GET / HTTP/1.1
client>Host: localhost:2004
client>User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
client>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
client>Accept-Language: en-US,en;q=0.5
client>Accept-Encoding: gzip, deflate
client>Connection: keep-alive

The request.xsl from the example was as followed;

<?xml version="1.0" encoding="windows-1252" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ora="http://www.oracle.com/XSL/Transform/java/"
                xmlns:socket="http://www.oracle.com/XSL/Transform/java/oracle.tip.adapter.socket.ProtocolTranslator"
                xmlns="http://xmlns.oracle.com/HelloWorld">
 <!-- Root template -->
 <xsl:template match="//input">
  <!-- Input coming from the browser would look like
  GET /input=sagar; HTTP/1.1.
  dummy variable to read the input= text -->
  <xsl:variable name="variable1"
                select="socket:socketRead('terminated', 'terminatedBy==')"/>
               
  <!-- Read the input string terminated by ;and build the input xml -->
  <HelloWorldProcessRequest>
   <input>
    <xsl:value-of select="socket:socketRead('terminated', 'terminatedBy=;')"/>
   </input>
  </HelloWorldProcessRequest>
 </xsl:template>
</xsl:stylesheet>


The SocketAdapter took the part from the first = sign to the first ; after that. This was confirmed by looking at the log set to trace level.

When trying again with telnet with the input;
input=Maarten;

It worked and I got an instance of my process. I was however not yet able to receive JSON from a HTTP POST request. First I needed an example request;

POST /request HTTP/1.1
Accept: application/jsonrequest
Content-Length: 123
Content-Type: application/jsonrequest
Host: localhost

{"user":"test","var":7,"t":"something","stuff":123}

I needed to extract everything after the first empty line (I'm not sure yet if this is always the case in HTTP messages!) to the end of the message. This way if the JSON message was multiline, it would also work. To do that I needed to understand the XPath function socket:socketRead used in the XSLT transformation used by the SocketAdapter when receiving the message

I found the following (http://docs.oracle.com/cd/E21764_01/integration.1111/e10231/adptr_sock.htm#BABHECAG);

"By using StyleReader, which is exposed by the NXSD framework, to read and write from the socket stream using the following methods: socketRead(nxsdStyle:String, nxsdStyleAttributes:String):String"

I remembered from the Native Format Builder (which also uses the NXSD framework) some codes I could use in the terminatedBy clause. The resulting XSLT for my usecase was;

<?xml version="1.0" encoding="windows-1252" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:ora="http://www.oracle.com/XSL/Transform/java/"
                xmlns:socket="http://www.oracle.com/XSL/Transform/java/oracle.tip.adapter.socket.ProtocolTranslator"
                xmlns="http://xmlns.oracle.com/HelloWorld">
 <!-- Root template -->
 <xsl:template match="//input">
  <!-- Input coming from the browser would look like
  GET /input=sagar; HTTP/1.1.
  dummy variable to read the input= text -->
  <xsl:variable name="variable1"
                select="socket:socketRead('terminated', 'terminatedBy=${eol}${eol}')"/>
              
  <!-- Read the input string terminated by ;and build the input xml -->
  <HelloWorldProcessRequest>
   <input>
    <xsl:value-of select="socket:socketRead('terminated', 'terminatedBy=${eof}')"/>
   </input>
  </HelloWorldProcessRequest>
 </xsl:template>
</xsl:stylesheet>

With this XSLT to parse the request I could obtain the JSON String from a HTTP POST request. This is the first step in creating a BPEL JSON HelloWorld service.

JsonPath

The parsing or querying of a JSON String can be done in several ways such as using the JsonPath (http://code.google.com/p/json-path/). Below is code for an example webservice which can be used.

package ms.testapp.soa.utils;

import com.jayway.jsonpath.JsonPath;

import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;

@WebService
public class JsonPathUtils {
   
    @WebResult(name = "result")
    public String ExecuteJsonPath(@WebParam(name = "jsonstring") String jsonstring, @WebParam(name = "jsonpath")String jsonpath) {
        String result = JsonPath.read(jsonstring, jsonpath).toString();
        return result;
    }
   
    public JsonPathUtils() {
        super();
    }
/*   
    public static void main(String[] args) {
        JsonPathUtils myPath = new JsonPathUtils();
        System.out.println(myPath.ExecuteJsonPath("{ \"store\": {\n" +
        "    \"book\": [ \n" +
        "      { \"category\": \"reference\",\n" +
        "        \"author\": \"Nigel Rees\",\n" +
        "        \"title\": \"Sayings of the Century\",\n" +
        "        \"price\": 8.95\n" +
        "      },\n" +
        "      { \"category\": \"fiction\",\n" +
        "        \"author\": \"Evelyn Waugh\",\n" +
        "        \"title\": \"Sword of Honour\",\n" +
        "        \"price\": 12.99,\n" +
        "        \"isbn\": \"0-553-21311-3\"\n" +
        "      }\n" +
        "    ],\n" +
        "    \"bicycle\": {\n" +
        "      \"color\": \"red\",\n" +
        "      \"price\": 19.95\n" +
        "    }\n" +
        "  }\n" +
        "}", "$.store.book[1].author"));
    }
*/
}

Conclusion

I haven't finished the HelloWorld BPEL JSON service implementation yet. What I've found however is that it requires some work to get relatively simple functionality.

Not only does it require some work, but to make the code reusuable is also a challenge. You will need routing mechanisms which sounds a lot like where the OSB is really good at.
JSON is often used when performance is important. JSON messages can be smaller then XML messages and a strict message or interface definition (such as for XML XSD/WSDL) is not a requirement. Also the (Java) frameworks required to work with JSON are often relatively easy to implement making quick development possible. When using the above solution, several components are used which cause additional overhead (such as the SocketAdapter and XSLT transformations). This is expensive when considering there can be a relatively light and easy implementation by using the OSB or custom Java code per integration.

File processing using a Spring component

Introduction

The Oracle SOA Suite 11g FileAdapter provides (among other things) functionality to read a variety of text file formats such as CSV and transform them into XML. This makes it easier to use them in for example BPEL. Usually the FileAdapter provides enough functionality to correctly parse files. Sometimes however, this is not possible. An example of such a file;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

h1 to h4 are header fields. The second and third line are the records and the last line indicates how many records are present in the file.

The last line cannot be excluded from parsing by the FileAdapter and will lead to errors. As a workaround and to provide some flexibility in file parsing, the following solution can be implemented. Keep in mind that this solution will cause problems for large files since the file will in memory be converted to XML.

Read the file as opaque using the FileAdapter. Create a Java class which has a public method with a byte array as input parameter and some parsing parameters. Wrap the Java class as a Spring component. Then you can use the component in a BPEL process as JDeveloper will generate XSD's/WSDL's for the Spring component for you.

Implementation

FileAdapter

In BPEL define a file adapter and select not to use a schema;

Java class parsefile

A Java class does the parsing. You can of course expand this example. Currently empty lines are also processed as entries, which might not be what you want. This class is also specific for the example given in the introduction (but is flexible enough for related file formats). I have chosen to return an ArrayList of ArrayLists. The first (outer) ArrayList contains the records of the file and the inner ArrayList contains the items in the record separated on a separator regular expression. The header lines can be ignored and you can specify to skip lines starting with a specific string. If you want to be able to skip multiple strings (in order to make a selection in the file of lines to process), you can change the type of the last parameter of the parse function to ArrayList and alter the code to loop over that.

Create the following Java class;

package nl.smeets.myfilereader;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;

import java.util.ArrayList;

public class parsefile {
    public parsefile() {
        super();
    }

    public ArrayList<ArrayList<String>> parse(byte[] bytes_in,
                                              String separator,
                                              Integer ignorefirstlines,
                                              String ignoreStartsWith) {
        ArrayList<ArrayList<String>> l_retval =
            new ArrayList<ArrayList<String>>();
        ByteArrayInputStream ba_in = new ByteArrayInputStream(bytes_in);
        BufferedReader br_in =
            new BufferedReader(new InputStreamReader(ba_in));
        Integer linecounter = 0;
        String line = null;
        try {
            while ((line = br_in.readLine()) != null) {
                if (!line.startsWith(ignoreStartsWith) &&
                    linecounter >= ignorefirstlines) {
                    l_retval.add(parseLine(line, separator));
                }
                linecounter++;
            }
        } catch (FileNotFoundException ex) {
            ex.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        } finally {
            //Close the BufferedReader
            try {
                if (br_in != null)
                    br_in.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
        return l_retval;
    }

    ArrayList<String> parseLine(String input, String separator) {
        ArrayList<String> retval = new ArrayList<String>();
        String[] splitVals = input.split(separator);
        for (String addVal : splitVals) {
            retval.add(addVal);
        }
        return retval;
    }

    public static void main(String[] args) {
        //test; skips first line and all lines starting with Total. uses ; as separator
        //parsefile parsefile = new parsefile();
        //System.out.println(parsefile.parse("h1;h2;h3;h4\ne1;f1;g1;h1\ne2;f2;g2;h2\nTotal: 1".getBytes(), ";", 1, "Total"));
    }
}

Extract interface

Right-click the Java file, Refactor, Extract interface

Spring bean

Use the wizard to create a Spring bean and define it as follows;

<?xml version="1.0" encoding="windows-1252" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:jee="http://www.springframework.org/schema/jee"
       xmlns:lang="http://www.springframework.org/schema/lang"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:sca="http://xmlns.oracle.com/weblogic/weblogic-sca"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.5.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.5.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd http://www.springframework.org/schema/tool http://www.springframework.org/schema/tool/spring-tool-2.5.xsd http://xmlns.oracle.com/weblogic/weblogic-sca META-INF/weblogic-sca.xsd">
 
<bean name="ParseFileBean" class="nl.smeets.myfilereader.parsefile"/>
<sca:service name="ParseFileBeanService" target="ParseFileBean" type="nl.smeets.myfilereader.Iparsefile"/>
</beans>

First make sure the Java interface and class are compiled to avoid errors such as the one below;

Now you can use the composite editor to expose the Spring bean or use it in a BPEL process.

BPEL

Use BPEL to bring it all together. In BPEL you can assign the parameters of the parsing Java method. In this example I've hardcoded them in the process but of course you can also use BPEL preferences (described in a previous post; http://javaoraclesoa.blogspot.com/2012/02/changing-properties-of-bpel-process-at.html) which can be defined in the composite.xml file and modified at runtime so if the format of the file changes, you don't have to redeploy the BPEL process. Don't forget to update the configuration plan supplied to reflect your local environment. *.txt files from (logical path) READ_FILE_DIR are read and xml files are written as output to logical path WRITE_FILE_DIR.


Input;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

Output;

<?xml version="1.0" encoding="UTF-8" ?>
<parseResponse xmlns="http://myfilereader.smeets.nl/">
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>A</item>
      <item>B</item>
      <item>C</item>
      <item>D</item>
   </return>
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>E</item>
      <item>F</item>
      <item>G</item>
      <item>H</item>
   </return>
</parseResponse>



You can download the entire example here; https://www.dropbox.com/s/ytgwjz7zjlvscmj/DemoReadFile.zip?dl=0