Seamless source "migration" from SOA Suite 12.1.3 to 12.2.1 using WLST and XSLT

When you migrate sources from SOA Suite 12.1.3 to SOA Suite 12.2.1, the only change I've seen JDeveloper do to the (SCA and Service Bus) code is updating versions in the pom.xml files from 12.1.3 to 12.2.1 (and some changes to jws and jpr files). Service Bus 12.2.1 has some build difficulties when using Maven. See Oracle Support: "OSB 12.2.1 Maven plugin error, 'Could not find artifact com.oracle.servicebus:sbar-project-common:pom' (Doc ID 2100799.1)". Oracle suggests updating the pom.xml of the project, changing the packaging type from sbar to jar and removing the reference to the parent project. This however will not help you because the created jar file does not have the structure required of Service Bus resources to be imported. To deploy Service Bus with Maven I've used the 12.1.3 plugin to create the sbar and a custom WLST file to do the actual deployment of this sbar to a 12.2.1 environment. A similar solution is described here.

Updates to the pom files can easily be automated as part of a build pipeline. This allows you to develop 12.1.3 code and automate the migration to 12.2.1. This can be useful if you want to avoid keeping separate 12.1.3 and 12.2.1 versions of your sources during a gradual migration. You can do bug fixes on the 12.1.3 sources and compile/deploy to production (usually production is the last environment to be upgraded) and use the same pipeline to compile and deploy the same sources (using altered pom files) to a 12.2.1 environment.

Standalone Buildtool for deployment of Oracle SOA Suite 11.1.1.6 composites

Introduction

SOA Suite 11g allows deployment to environments from JDeveloper. The people installing software often don't have JDeveloper available or configured correctly. Often there is the requirement to provide a standalone package for installation which can be executed without too much configuration by someone with relatively little knowledge about what he/she is installing.

There have already been numerous posts about build automation for Oracle SOA 11g applications, such as the excellent post from Edwin Biemond; http://biemond.blogspot.nl/2009/09/deploy-soa-suite-11g-composite.html and http://redstack.wordpress.com/2011/03/15/getting-started-with-continuous-integration-for-soa-projects/

In this post I've created a standalone Buildtool for SOA Suite 11.1.1.6. This Buildtool allows compilation and deployment of various SOA artifacts. It is heavily based on http://biemond.blogspot.nl/2009/09/deploy-soa-suite-11g-composite.html however provides some added value such as;

- it's fully based on 11.1.1.6 libraries
- can be used standalone without the need to install JDeveloper or Weblogic Server
- allows applications to be packaged/configured separately and independently
- provides a script to recreate most of the Buildtool from scratch (hopefully making upgrades to the buildtool more easy)
- this buildtool compiles and installs composites. compilation is thus not dependent on the client providing a jar file; the sources can be used as source for the build.

Implementation

Install JDeveloper

Download and install JDeveloper 11.1.1.6. Install the following plugins from http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/156082.xml

-          Oracle BPM Studio 11g

-          Oracle SOA Composite Editor

This will be used as the start-point for the Buildtool.

Copy sources

The below DOS batch script copies the required libraries to the TARGETDIR (build tool base directory). The SOURCEDIR is a locally installed JDeveloper 11.1.1.6.

set SOURCEDIR=D:\dev\Middleware11116
set TARGETDIR=D:\dev\buildtool11116


rem buildtool generator 11.1.1.6
rem Maarten Smeets, 2012-08-05


mkdir %TARGETDIR%\logs
mkdir %TARGETDIR%\junit
mkdir %TARGETDIR%\builds

rem wlserver_10.3/server/lib
mkdir %TARGETDIR%\Middleware\wlserver_10.3\server\lib
xcopy /i %SOURCEDIR%\wlserver_10.3\server\lib\*.jar %TARGETDIR%\Middleware\wlserver_10.3\server\lib


rem jdeveloper\ant
mkdir %TARGETDIR%\Middleware\jdeveloper\ant
xcopy /s %SOURCEDIR%\jdeveloper\ant %TARGETDIR%\Middleware\jdeveloper\ant


rem jdeveloper\bin
mkdir %TARGETDIR%\Middleware\jdeveloper\bin
xcopy /s %SOURCEDIR%\jdeveloper\bin %TARGETDIR%\Middleware\jdeveloper\bin


rem jdeveloper\soa\modules
mkdir %TARGETDIR%\Middleware\jdeveloper\soa\modules
xcopy /s %SOURCEDIR%\jdeveloper\soa\modules %TARGETDIR%\Middleware\jdeveloper\soa\modules


rem jdeveloper\uddi
mkdir %TARGETDIR%\Middleware\jdeveloper\uddi
xcopy /s %SOURCEDIR%\jdeveloper\uddi %TARGETDIR%\Middleware\jdeveloper\uddi


rem jdk160_24
rem 11.1.1.4: jdk160_21
mkdir %TARGETDIR%\Middleware\jdk160_24
xcopy /s %SOURCEDIR%\jdk160_24 %TARGETDIR%\Middleware\jdk160_24


rem ant-contrib.jar
mkdir %TARGETDIR%\lib
xcopy %SOURCEDIR%\modules\net.sf.antcontrib_1.1.0.0_1-0b2\lib\* %TARGETDIR%\lib


rem oracle_common\webservices
mkdir %TARGETDIR%\Middleware\oracle_common\webservices
xcopy /s %SOURCEDIR%\oracle_common\webservices %TARGETDIR%\Middleware\oracle_common\webservices


rem oracle_common/modules
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.model_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.share.ca_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.share_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.bali.share_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.dms_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.fabriccommon_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.javacache_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.jmx_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.jps_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.jrf_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.mds_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.nlsrtl_11.1.0
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.odl_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.webservices_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.wsm.common_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.xdk_11.1.0
mkdir %TARGETDIR%\Middleware\oracle_common\modules\oracle.xmlef_11.1.1
mkdir %TARGETDIR%\Middleware\oracle_common\modules\datadirect_4.1


xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.adf.model_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.model_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.adf.share.ca_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.share.ca_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.adf.share_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.adf.share_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.bali.share_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.bali.share_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.dms_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.dms_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.fabriccommon_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.fabriccommon_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.javacache_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.javacache_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.jmx_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.jmx_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.jps_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.jps_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.jrf_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.jrf_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.mds_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.mds_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.nlsrtl_11.1.0 %TARGETDIR%\Middleware\oracle_common\modules\oracle.nlsrtl_11.1.0
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.odl_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.odl_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.webservices_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.webservices_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.wsm.common_11.1.1 %TARGETDIR%\Middleware\oracle_common\modules\oracle.wsm.common_11.1.1
xcopy /s %SOURCEDIR%\oracle_common\modules\oracle.xdk_11.1.0 %TARGETDIR%\Middleware\oracle_common\modules\oracle.xdk_11.1.0
xcopy /s %SOURCEDIR%\oracle_common\modules\datadirect_4.1 %TARGETDIR%\Middleware\oracle_common\modules\datadirect_4.1


xcopy /i %SOURCEDIR%\oracle_common\modules\oracle.ucp_11.1.0.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\org.apache.commons.beanutils_1.6.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\org.apache.commons.digester_1.8.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\org.apache.commons.logging_1.0.4.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\org.springframework_2.5.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\oracle.classloader_11.1.1.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\oracle.http_client_11.1.1.jar %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\oracle.logging-utils_11.1.1.jar  %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\commonj.sdo_2.1.0.jar  %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\com.bea.core.xml.xmlbeans_2.1.0.0_2-5-1.jar  %TARGETDIR%\Middleware\oracle_common\modules
xcopy /i %SOURCEDIR%\oracle_common\modules\com.bea.core.weblogic.security.identity_1.1.2.1.jar  %TARGETDIR%\Middleware\oracle_common\modules

rem modules


mkdir %TARGETDIR%\Middleware\modules


xcopy /i %SOURCEDIR%\modules\com.bea.core.apache.commons.collections_3.2.0.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\com.bea.core.apache.commons.lang_2.1.0.jar %TARGETDIR%\Middleware\modules
rem 11.1.1.4
rem xcopy /i %SOURCEDIR%\modules\com.bea.core.weblogic.workmanager_1.9.0.0.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\com.bea.core.weblogic.workmanager_1.10.0.0.jar %TARGETDIR%\Middleware\modules
rem 11.1.1.4
rem xcopy /i %SOURCEDIR%\modules\com.bea.core.xml.beaxmlbeans_2.2.0.0_2-5-1.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\com.bea.core.xml.beaxmlbeans_2.3.0.0_2-5-1.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\commonj.sdo_1.0.0.0_1-0.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\glassfish.jaxb_1.0.0.0_2-1-12.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\javax.management.j2ee_1.0.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\javax.xml.rpc_1.2.1.jar %TARGETDIR%\Middleware\modules
rem 11.1.1.4
rem xcopy /i %SOURCEDIR%\modules\org.eclipse.persistence_1.0.0.0_2-1.jar %TARGETDIR%\Middleware\modules
xcopy /i %SOURCEDIR%\modules\org.eclipse.persistence_1.1.0.0_2-1.jar %TARGETDIR%\Middleware\modules

Some script files were taken from https://github.com/biemond/soa_tools/tree/master/ant and updated. The updated scriptfiles can be downloaded here; https://dl.dropbox.com/u/6693935/blog/buildtool11116scriptfiles.zip

Usage

In my case, the buildtool base dir is; D:\dev\buildtool11116. Here you can create a directory build-area (as specified in the build.properties file). In this directory you can create a structure like for example; https://dl.dropbox.com/u/6693935/blog/build-area.zip

The structure contains an example project, a deployment profile and configuration files. This is what the developer has to provide. The person installing the software has to update the following files;
- update deployAll.bat to reflect the correct locations (once)
- update build.properties to reflect the environment which is used (it is recommended the installer keeps per environment a separate build.properties file)

When deployAll.bat is executed, log files are generated in the logs directory and the deployed applications are put in the builds directory.

The build.xml is capable of calling unit tests, but since I got the following error which I couldn't fix quickly enough;

  [scatest] java.lang.NoClassDefFoundError: weblogic/security/subject/AbstractSubject

and

D:\dev\buildtool11116\Middleware\jdeveloper\bin\ant-sca-test.xml:111: Problem executing test cases
weblogic/security/subject/AbstractSubject

I've decided to put this functionality in comments and maybe look at it later. wlclient.jar contains this class and it should be in the classpath.

The example build-area provides a simple deployment of a SCA composite. The supplied build.xml provides functionality for MDS deployments and partitions (Edwin Biemond has described this functionality on his blog site). Some minor changes to the script might be needed to make sure deployment of the MDS is from the build-area folder so developers can more easily provide the required files. You can download the entire 11.1.1.6 buildtool here; https://dl.dropbox.com/u/6693935/blog/buildtool11116.zip.

File processing using a Spring component

Introduction

The Oracle SOA Suite 11g FileAdapter provides (among other things) functionality to read a variety of text file formats such as CSV and transform them into XML. This makes it easier to use them in for example BPEL. Usually the FileAdapter provides enough functionality to correctly parse files. Sometimes however, this is not possible. An example of such a file;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

h1 to h4 are header fields. The second and third line are the records and the last line indicates how many records are present in the file.

The last line cannot be excluded from parsing by the FileAdapter and will lead to errors. As a workaround and to provide some flexibility in file parsing, the following solution can be implemented. Keep in mind that this solution will cause problems for large files since the file will in memory be converted to XML.

Read the file as opaque using the FileAdapter. Create a Java class which has a public method with a byte array as input parameter and some parsing parameters. Wrap the Java class as a Spring component. Then you can use the component in a BPEL process as JDeveloper will generate XSD's/WSDL's for the Spring component for you.

Implementation

FileAdapter

In BPEL define a file adapter and select not to use a schema;

Java class parsefile

A Java class does the parsing. You can of course expand this example. Currently empty lines are also processed as entries, which might not be what you want. This class is also specific for the example given in the introduction (but is flexible enough for related file formats). I have chosen to return an ArrayList of ArrayLists. The first (outer) ArrayList contains the records of the file and the inner ArrayList contains the items in the record separated on a separator regular expression. The header lines can be ignored and you can specify to skip lines starting with a specific string. If you want to be able to skip multiple strings (in order to make a selection in the file of lines to process), you can change the type of the last parameter of the parse function to ArrayList and alter the code to loop over that.

Create the following Java class;

package nl.smeets.myfilereader;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;

import java.util.ArrayList;

public class parsefile {
    public parsefile() {
        super();
    }

    public ArrayList<ArrayList<String>> parse(byte[] bytes_in,
                                              String separator,
                                              Integer ignorefirstlines,
                                              String ignoreStartsWith) {
        ArrayList<ArrayList<String>> l_retval =
            new ArrayList<ArrayList<String>>();
        ByteArrayInputStream ba_in = new ByteArrayInputStream(bytes_in);
        BufferedReader br_in =
            new BufferedReader(new InputStreamReader(ba_in));
        Integer linecounter = 0;
        String line = null;
        try {
            while ((line = br_in.readLine()) != null) {
                if (!line.startsWith(ignoreStartsWith) &&
                    linecounter >= ignorefirstlines) {
                    l_retval.add(parseLine(line, separator));
                }
                linecounter++;
            }
        } catch (FileNotFoundException ex) {
            ex.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        } finally {
            //Close the BufferedReader
            try {
                if (br_in != null)
                    br_in.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
        return l_retval;
    }

    ArrayList<String> parseLine(String input, String separator) {
        ArrayList<String> retval = new ArrayList<String>();
        String[] splitVals = input.split(separator);
        for (String addVal : splitVals) {
            retval.add(addVal);
        }
        return retval;
    }

    public static void main(String[] args) {
        //test; skips first line and all lines starting with Total. uses ; as separator
        //parsefile parsefile = new parsefile();
        //System.out.println(parsefile.parse("h1;h2;h3;h4\ne1;f1;g1;h1\ne2;f2;g2;h2\nTotal: 1".getBytes(), ";", 1, "Total"));
    }
}

Extract interface

Right-click the Java file, Refactor, Extract interface

Spring bean

Use the wizard to create a Spring bean and define it as follows;

<?xml version="1.0" encoding="windows-1252" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:jee="http://www.springframework.org/schema/jee"
       xmlns:lang="http://www.springframework.org/schema/lang"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:sca="http://xmlns.oracle.com/weblogic/weblogic-sca"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.5.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.5.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd http://www.springframework.org/schema/tool http://www.springframework.org/schema/tool/spring-tool-2.5.xsd http://xmlns.oracle.com/weblogic/weblogic-sca META-INF/weblogic-sca.xsd">
 
<bean name="ParseFileBean" class="nl.smeets.myfilereader.parsefile"/>
<sca:service name="ParseFileBeanService" target="ParseFileBean" type="nl.smeets.myfilereader.Iparsefile"/>
</beans>

First make sure the Java interface and class are compiled to avoid errors such as the one below;

Now you can use the composite editor to expose the Spring bean or use it in a BPEL process.

BPEL

Use BPEL to bring it all together. In BPEL you can assign the parameters of the parsing Java method. In this example I've hardcoded them in the process but of course you can also use BPEL preferences (described in a previous post; http://javaoraclesoa.blogspot.com/2012/02/changing-properties-of-bpel-process-at.html) which can be defined in the composite.xml file and modified at runtime so if the format of the file changes, you don't have to redeploy the BPEL process. Don't forget to update the configuration plan supplied to reflect your local environment. *.txt files from (logical path) READ_FILE_DIR are read and xml files are written as output to logical path WRITE_FILE_DIR.


Input;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

Output;

<?xml version="1.0" encoding="UTF-8" ?>
<parseResponse xmlns="http://myfilereader.smeets.nl/">
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>A</item>
      <item>B</item>
      <item>C</item>
      <item>D</item>
   </return>
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>E</item>
      <item>F</item>
      <item>G</item>
      <item>H</item>
   </return>
</parseResponse>



You can download the entire example here; https://www.dropbox.com/s/ytgwjz7zjlvscmj/DemoReadFile.zip?dl=0