Oracle Service Bus: Pipeline alerts in Splunk using SNMP traps

Oracle Service Bus provides a reporting activity called Alert. The OSB pipeline alerts use a persistent store. This store is file based. Changing the persistent store to JDBC based, does not cause pipeline alerts to be stored in a database instead of on disk. When the persistent store on disk becomes large, opening pipeline alerts in the Enterprise Manager (12c) or Service Bus console (11g) can suffer from poor performance. If you put an archive setting on pipeline alerts (see here), the space from the persistent store on disk is not reduced when alerts get deleted. You can compact the store to reduce space (see here), but this requires the store to be offline and this might require shutting down the Service Bus. This can be cumbersome to do often and is not good for your availability.

If you do not want to use the EM / SB console or have the issues with the filestore, there is an alternative. Pipeline alerts can produce SNMP traps. SNMP traps can be forwarded by a WebLogic SNMP Agent to an SNMP Manager. This manager can store the SNMP traps in a file and Splunk can monitor the file. Splunk makes searching alerts and visualizing them easy. In this blog I will describe the steps needed to get a minimal setup with SNMP traps going and how to see the pipeline alerts in Splunk.

WebLogic Server: Logging the SOAP action in the access.log

WebLogic Server allows you to customize your access.log. This can be very powerful if you want to monitor for example service response times in a tool like Splunk (see here). When working with SOAP services though, especially those with many operations, it can be insufficient to monitor services to the level of the individual endpoint. You want to also know with which intent the endpoint is called. In this blog I will show how this can be achieved.

WebLogic Server: Automate obtaining performance metrics from DMS

Oracle provides the Dynamic Monitoring Service (DMS) as part of WebLogic Server which is extremely useful if you want to obtain aggregated data of an environment in case of for example a performance test. The data which can be obtained from DMS is extensive. This varies from average duration of service calls to JVM garbage collects to datasource statistics. DMS can be queried with WLST.  See for example here. On example script based on this can be found here. You can also directly go to a web-interface such as: http://<host>:<port>/dms/Spy. The DMS Spy servlet is by default only enabled on development environments but can be deployed on production environments (see here).

Obtaining data from DMS in an automated fashion, even with the WLST support, can be a challenge. In this blog I provide a Python 2.7 script which allows you to get information from the DMS and dump it in a CSV file for further processing. The script first logs and uses the obtained session information to download information from a specific table in XML. This XML is converted to CSV. The code does not require an Oracle Home (it is not WLST based). The purpose here is to provide an easy to use starting point which can be expanded to suit specific use-cases. The script works against WebLogic 11g and 12c environments (has been tested against 11.1.1.7 and 12.2.1). Do mind that the example URL given in the script obtains performance data on webservice operations. This works great on composites but not on Service Bus or JAX-WS services. You can download a general script here (which requires minimal changes to use) and a (more specific) script with examples of how to preprocess data in the script here.

Oracle Service Bus: Obtaining a list of exposed SOAP HTTP endpoints

The Oracle Service Bus is often used for service virtualization. Endpoints are exposed on the Service Bus which proxy other services. Using such an abstraction layer can provide benefits such as (among many other things) monitoring/logging, dealing with different versions of services, throttling/error handling and result caching.

In this blog I will provide a small (Java) script, which works for SOA Suite 11g and 12c, which determines exposed endpoints on the Service Bus.

Oracle SOA Suite 11g and 12c: Determining composite dependencies to the level of individual operations

In large companies, often there are many services and dependencies between services. It is important to track service dependencies in order to for example estimate the impact of changes. Design documents or architecture views can be used for this but as everybody knows, there is often a gap between theory and practice (design and implementation).

In this blog post I provide code to determine dependencies between composites to the level of operation calls. In order to achieve this, I'll parse the composite.xml files, JCA files (used by adapters) and also the BPEL and BPMN files in order to determine the operations. The script can be used for SOA Suite 11g and 12c composites.

The above picture shows different parts of which a composite is composed and how they are linked. The script first determines references. The references specify which external services are called. Then by using wires, the relevant components are determined. Based on the component type, specific logic is used to extract the operation. Not shown in this picture is how database dependencies can also be determined by the script by parsing the JCA files specified in the reference. If you're in a hurry, you can go to the 'Executing the script' part directly and skip the explanation.

Generic configuration plan for SOA Suite 11g composite deployments

In order to adapt deployments of SOA Suite 11g composites to specific environments, configuration plans can be used to rewrite the composite.xml file. These configuration plans can be used during deployment by the Oracle supplied ANT scripts. See for example; http://biemond.blogspot.nl/2009/09/deploy-soa-suite-11g-composite.html on how these ANT scripts can be used to deploy composites.

A configuration plan can be generated from the composite.xml file. See figure 41-1 on http://docs.oracle.com/cd/E14571_01/integration.1111/e10224/sca_lifecycle.htm.

When however using the JDeveloper default generated configuration plan per process, adding new references, WSDL files, XSD files, JCA files, imports, properties, etc requires updating the configuration plan. Maintaining the configuration plans can become quite cumbersome when there are a lot of composites, endpoints and regular changes in environments.

As stated in http://www.oracle.com/technetwork/articles/soa/start-small-luttikhuizen-1502916.html; 'Another best practice is to avoid creating a separate configuration plan per composite per environment.'

This led me to want to create such a common configuration plan file which would be composite independent and thus reusable.

Implementation

Usually development is done against a development environment. The processes which are under version control have endpoints and properties specific to the development environment. To make a process specific for other environments, the development references are replaced by for example test references.

I generated a standard configuration plan for a process and made it generally usable. I reduced the standard generated configuration plan to the following (of course the http://dev and http://tst should be replaced with your specific environment);

<?xml version="1.0" encoding="UTF-8"?>
<SOAConfigPlan xmlns:jca="http://platform.integration.oracle/blocks/adapter/fw/metadata"
               xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
               xmlns:orawsp="http://schemas.oracle.com/ws/2006/01/policy"
               xmlns:edl="http://schemas.oracle.com/events/edl"
               xmlns="http://schemas.oracle.com/soa/configplan">
    <composite name="*">
        <import>
            <searchReplace>
                <search>http://dev</search>
                <replace>http://tst</replace>
            </searchReplace>
        </import>
        <reference name="*">
            <binding type="ws">
                <property name="endpointURI">
                    <searchReplace>
                        <search>http://dev</search>
                        <replace>http://tst</replace>
                    </searchReplace>
                </property>
            </binding>
        </reference>
        <reference name="*">
            <binding type="ws">
                <attribute name="location">
                    <searchReplace>
                        <search>http://dev</search>
                        <replace>http://tst</replace>
                    </searchReplace>
                </attribute>
            </binding>
        </reference>
    </composite>
    <wsdlAndSchema name="*">
        <searchReplace>
            <search>http://dev</search>
            <replace>http://tst</replace>
        </searchReplace>
    </wsdlAndSchema>
</SOAConfigPlan>

This configuration plan works composite independent and replaces all imports, references to endpoints and location references and environment references in all WSDL, XSD and JCA files.

Conclusion

Configuration plans are the recommended option to use for making composites environment specific (http://www.oracle.com/technetwork/articles/soa/start-small-luttikhuizen-1502916.html). Efficient use of configuration plans can greatly reduce the work required per process.

The configuration plan as displayed above does not replace BPEL properties which might also be environment specific. To achieve replacing of BPEL properties, the properties should be consistent in name and value among processes. In order to add such replacement code to the configuration plan, you can add properties to your BPEL process, generate a configuration plan, adapt the generated code and add the relevant portion to the generic configuration plan.

You should check if using this configuration plan does what is required by validating it in JDeveloper and checking if all required references are replaced. The danger of not having all references replaced is that messages meant for for example the test environment end up on development or even worse; production messages end up in development. Also message definitions can change among environments. If an XSD from a wrong environment is imported, conflicts can arise.

An alternative for using configuration plans is deploying against localhost. Localhost is a reference to the current machine. When developing for example on a standalone machine, localhost refers to that machine. When the process is deployed on a different server, localhost refers to that different machine. This method has several drawbacks however and is thus not recommended. Questions arise when using such a method;
- How to deal with external services? (especially if they differ amongst environments). Using an abstraction to the actual service endpoints (such as a service registry) can help, but you will undoubtedly encounter other similar issues in this category.
- How to deal with load balancing in a clustered environment? Localhost is the current machine so a process initiated on one machine does not initiate processes on the other machines in the cluster and thus load balancing is limited.

File processing using a Spring component

Introduction

The Oracle SOA Suite 11g FileAdapter provides (among other things) functionality to read a variety of text file formats such as CSV and transform them into XML. This makes it easier to use them in for example BPEL. Usually the FileAdapter provides enough functionality to correctly parse files. Sometimes however, this is not possible. An example of such a file;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

h1 to h4 are header fields. The second and third line are the records and the last line indicates how many records are present in the file.

The last line cannot be excluded from parsing by the FileAdapter and will lead to errors. As a workaround and to provide some flexibility in file parsing, the following solution can be implemented. Keep in mind that this solution will cause problems for large files since the file will in memory be converted to XML.

Read the file as opaque using the FileAdapter. Create a Java class which has a public method with a byte array as input parameter and some parsing parameters. Wrap the Java class as a Spring component. Then you can use the component in a BPEL process as JDeveloper will generate XSD's/WSDL's for the Spring component for you.

Implementation

FileAdapter

In BPEL define a file adapter and select not to use a schema;

Java class parsefile

A Java class does the parsing. You can of course expand this example. Currently empty lines are also processed as entries, which might not be what you want. This class is also specific for the example given in the introduction (but is flexible enough for related file formats). I have chosen to return an ArrayList of ArrayLists. The first (outer) ArrayList contains the records of the file and the inner ArrayList contains the items in the record separated on a separator regular expression. The header lines can be ignored and you can specify to skip lines starting with a specific string. If you want to be able to skip multiple strings (in order to make a selection in the file of lines to process), you can change the type of the last parameter of the parse function to ArrayList and alter the code to loop over that.

Create the following Java class;

package nl.smeets.myfilereader;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;

import java.util.ArrayList;

public class parsefile {
    public parsefile() {
        super();
    }

    public ArrayList<ArrayList<String>> parse(byte[] bytes_in,
                                              String separator,
                                              Integer ignorefirstlines,
                                              String ignoreStartsWith) {
        ArrayList<ArrayList<String>> l_retval =
            new ArrayList<ArrayList<String>>();
        ByteArrayInputStream ba_in = new ByteArrayInputStream(bytes_in);
        BufferedReader br_in =
            new BufferedReader(new InputStreamReader(ba_in));
        Integer linecounter = 0;
        String line = null;
        try {
            while ((line = br_in.readLine()) != null) {
                if (!line.startsWith(ignoreStartsWith) &&
                    linecounter >= ignorefirstlines) {
                    l_retval.add(parseLine(line, separator));
                }
                linecounter++;
            }
        } catch (FileNotFoundException ex) {
            ex.printStackTrace();
        } catch (IOException ex) {
            ex.printStackTrace();
        } finally {
            //Close the BufferedReader
            try {
                if (br_in != null)
                    br_in.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
        return l_retval;
    }

    ArrayList<String> parseLine(String input, String separator) {
        ArrayList<String> retval = new ArrayList<String>();
        String[] splitVals = input.split(separator);
        for (String addVal : splitVals) {
            retval.add(addVal);
        }
        return retval;
    }

    public static void main(String[] args) {
        //test; skips first line and all lines starting with Total. uses ; as separator
        //parsefile parsefile = new parsefile();
        //System.out.println(parsefile.parse("h1;h2;h3;h4\ne1;f1;g1;h1\ne2;f2;g2;h2\nTotal: 1".getBytes(), ";", 1, "Total"));
    }
}

Extract interface

Right-click the Java file, Refactor, Extract interface

Spring bean

Use the wizard to create a Spring bean and define it as follows;

<?xml version="1.0" encoding="windows-1252" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:jee="http://www.springframework.org/schema/jee"
       xmlns:lang="http://www.springframework.org/schema/lang"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:sca="http://xmlns.oracle.com/weblogic/weblogic-sca"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.5.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.5.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd http://www.springframework.org/schema/tool http://www.springframework.org/schema/tool/spring-tool-2.5.xsd http://xmlns.oracle.com/weblogic/weblogic-sca META-INF/weblogic-sca.xsd">
 
<bean name="ParseFileBean" class="nl.smeets.myfilereader.parsefile"/>
<sca:service name="ParseFileBeanService" target="ParseFileBean" type="nl.smeets.myfilereader.Iparsefile"/>
</beans>

First make sure the Java interface and class are compiled to avoid errors such as the one below;

Now you can use the composite editor to expose the Spring bean or use it in a BPEL process.

BPEL

Use BPEL to bring it all together. In BPEL you can assign the parameters of the parsing Java method. In this example I've hardcoded them in the process but of course you can also use BPEL preferences (described in a previous post; http://javaoraclesoa.blogspot.com/2012/02/changing-properties-of-bpel-process-at.html) which can be defined in the composite.xml file and modified at runtime so if the format of the file changes, you don't have to redeploy the BPEL process. Don't forget to update the configuration plan supplied to reflect your local environment. *.txt files from (logical path) READ_FILE_DIR are read and xml files are written as output to logical path WRITE_FILE_DIR.


Input;

h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2

Output;

<?xml version="1.0" encoding="UTF-8" ?>
<parseResponse xmlns="http://myfilereader.smeets.nl/">
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>A</item>
      <item>B</item>
      <item>C</item>
      <item>D</item>
   </return>
   <return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
      <item>E</item>
      <item>F</item>
      <item>G</item>
      <item>H</item>
   </return>
</parseResponse>



You can download the entire example here; https://www.dropbox.com/s/ytgwjz7zjlvscmj/DemoReadFile.zip?dl=0

Oracle SOA Suite Cluster part 2

Introduction

This is the second part in which I describe the configuration of a Weblogic Cluster. Quick and easy. This does not conform to the Oracle Enterprise Deployment Guide (http://docs.oracle.com/cd/E16764_01/core.1111/e12036/toc.htm) and should thus not be used for enterprise installations. I for example don't install Oracle HTTP server.

This part consists of installation of
- Load balancer
- RCU (Repository creation utility)
- Weblogic server and SOA Suite
- Configure the Nodemanager
- Transport the managed domain to the other machine
- Start the cluster

Load balancer

By using the load balancer URL when referring to other processes, I can do process call distribution. The idea of this setup is illustrated in part 1.

For the load balancer I used pound which can be downloaded at; http://pkgs.repoforge.org/pound/

I downloaded it and installed the rpm using (as root); rpm -i pound-2.4.3-1.el5.rf.x86_64.rpm

I used the below configuration. The host running the load balancer (and database) is 192.168.1.160. The hosts which will run the Weblogic servers are 192.168.1.161 and 192.168.1.162.

User        "nobody"
Group       "nobody"

## Main listening ports
ListenHTTP
    Address 192.168.1.160
    Port    8001
End
Service
        BackEnd
                Address 192.168.1.161
                Port    8001
        End
        BackEnd
                Address 192.168.1.162
                Port    8001
        End
        Session
                Type Cookie
                ID   "JSESSIONID"
                TTL  300
        End
End

RCU

As shown in the image below, I installed the required schema's. No errors this time with the Oracle Enterprise Scheduler schema such as when I used the Database XE 11r2 installation thus it is a good idea to use the Enterprise Edition database.

Weblogic server and SOA Suite

The below is based on the Oracle documentation; http://docs.oracle.com/cd/E23943_01/web.1111/e13709/setup.htm#CLUST431

I choose 2 different virtual machines to install the Weblogic servers on. I did this to be able to identify the challenges which need to be overcome when the server instances are running on different machines (and with different IP addresses). It is of course also possible to install the Weblogic cluster on a single machine, but that would not be sufficient to identify points of attention when developing against a production clustered environment at customers.

First a few basic things;

• start the installation of the Weblogic server as followed;  java -d64 -jar wlsXXX_generic.jar. Then install the SOA Suite
• set the server to listen on all local addresses. References to localhost will cause problems
• Use the configuration wizard (/u01/app/fmw/wlserver_10.3/common/bin/config.sh in my case; http://docs.oracle.com/cd/E23943_01/web.1111/e14140/configuration_screens.htm#WLDCW112 to create an Administration domain, a Managed domain and a cluster. See also  http://biemond.blogspot.com/2008/11/build-your-own-weblogic-103-cluster.html for some screenshots on how to configure the cluster.

If by accident, you run the installer of the Weblogic server or SOA Suite as a different user, remove the installation, reinstall and get the below exception;  you can do chmod -R 777 /tmp to fix it (https://forums.oracle.com/forums/thread.jspa?threadID=1085777)

weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider. Enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01538: The default policy provider was not found. at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1394)

Configure the nodemanager

To make it possible to use the nodemanager, you need to configure SSL for the server, keystore, truststore etc. Look at; http://wls4mscratch.wordpress.com/2010/05/31/configure-node-manager-over-ssl-custom-certifcates/.

I started as followed;

Host 1;

keytool -genkey -alias server01 -keyalg RSA -keysize 1024 -dname "CN=Puneeth, OU=Oracle, O=BEA, L=Denver, ST=Colorado, C=US" -keypass password -keystore identity.jks -storepass password
keytool -selfcert -v -alias server01 -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -export -v -alias server01 -file rootCA01.der -keystore identity.jks -storepass password
keytool -import -v -trustcacerts -alias server01 -file rootCA01.der -keystore trust.jks -storepass password

Host 2;

keytool -genkey -alias server02 -keyalg RSA -keysize 1024 -dname "CN=Puneeth, OU=Oracle, O=BEA, L=Denver, ST=Colorado, C=US" -keypass password -keystore identity.jks -storepass password
keytool -selfcert -v -alias server02 -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -export -v -alias server02 -file rootCA02.der -keystore identity.jks -storepass password
keytool -import -v -trustcacerts -alias server02 -file rootCA02.der -keystore trust.jks -storepass password

Copy rootCA01.der and rootCA02.der to both servers. Then on server 1 do;
keytool -import -v -trustcacerts -alias server02 -file rootCA02.der -keystore trust.jks -storepass password
and on server 2 do;
keytool -import -v -trustcacerts -alias server01 -file rootCA01.der -keystore trust.jks -storepass password

I also did the changes to nodemanager.properties, setDomainEnv, startNodemanager and the Weblogic server SSL certificate configuration as described at the previously mentioned location. This SSL configuration will prevent the following error from occurring when the nodemanager is started and you try to access it from the Weblogic console.

<Mar 25, 2012 1:31:34 PM CEST> <Warning> <Security> <BEA-090482> <BAD_CERTIFICATE alert was received from localhost.localdomain - 127.0.0.1. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>
<Mar 25, 2012 1:31:34 PM> <WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from localhost.localdomain - 127.0.0.1. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>

Transport the domain

Use pack to create a template and transport it to the other machine; http://docs.oracle.com/cd/E23943_01/web.1111/e14144/tasks.htm#WLDPU136

pack.sh is present in several locations;
/u01/app/fmw/Oracle_SOA1/common/bin/pack.sh
/u01/app/fmw/oracle_common/common/bin/pack.sh
/u01/app/fmw/wlserver_10.3/common/bin/pack.sh

I've used the one in the OracleSOA1 folder

On the machine on which you created the domain;
/u01/app/fmw/Oracle_SOA1/common/bin/pack.sh -managed=true -domain=/u01/app/fmw/user_projects/domains/soa_domain -template=misakimw1.jar -template_name="misakimw1"

On the other machine which will be running the managed domain;
/u01/app/fmw/Oracle_SOA1/common/bin/unpack.sh -domain=/u01/app/fmw/user_projects/domains/soa_domain -template=/home/oracle/misakimw1.jar

Starting the cluster

First start the database machine. Pound (the load balancer) is optional at this point and becomes important when starting with experiments on process call load balancing. Then start the nodemanager on both machines. Then start the administration server. From the Weblogic console you can then start both managed domains. This is how it looks in the enterprise manager when it is all started;

When you deploy, you deploy to the admin server. The admin server makes sure the different managed servers get the processes.

I will start experimenting on this setup shortly and compare it to production environments.