Dramatically reduce SOA Suite 11g startup time by cleaning the MDS

SOA Suite can sometimes be a bit slow to start. This is especially the case when there are a lot of composites to load. Customers using different versions of composites can benefit from undeploying non-default revisions of processes which do not have any running instances (see for example here). Undeployment in most cases is an asynchronous process which does not give feedback. It can partially fail without you noticing (apparently not an atomic transaction). This sometimes leaves composite remains; parts of the composite which are still loaded at startup but are not visible from the Enterprise Manager. Removing these can dramatically reduce server startup time. Especially in an environment which has been used for some time and environments with many versions of composites. Reducing the time required to get the soa-infra application fully up and running is of course mostly relevant for 11g SOA installations and less for 12.1.3 (which does some lazy loading) and 12.2.1 (which supports parallel deployments, also during server start-up).

In this article I'll demonstrate how these left-over composite parts can be identified and removed on an 11.1.1.7 SOA environment. First try this procedure on a development or test environment before executing it in production! This method is not supported by Oracle (or me) in any way and using it is entirely at your own risk. If something breaks, tell me so I can update this article. Thanks!

Please mind that these actions, although they help with the start time and memory usage of your SOA environment, have less impact on run-time performance than for example purging of instances and reducing the amount of deployed composites (or tweaking datasources, soa-infra database, JVM, etc).

SOA Suite can be up quickly!

A first look at Splunk. Monitor Oracle SOA Suite service response times

Measuring performance of services can be done in various ways. In this blog I will describe a method of measuring Oracle SOA service response times with Splunk a popular monitoring tool. In order to monitor service response times with Splunk, Splunk needs to obtain its data from somewhere. In this example I'll use the HTTP access log which I expand with a time-taken field. Disclaimer; my experience with Splunk is something like 2 hours. This might also be an indication of what can quickly be achieved with Splunk with little knowledge.

WebLogic Server: Analyzing stuck threads

A while ago I was confronted with a WebLogic server which did not want to start. The log files did not give an indication of what was wrong. I tried the usual suspects such as clearing the tmp and removing lok files but they did solve the issue. How to proceed? In this blog article I'll provide an example of how such an issue can be debugged.

A stuck thread is a thread which takes longer than the configured stuck thread max wait time (600 seconds by default). To simulate a stuck thread issue I created a simple servlet which does a Thread.sleep upon a request. I'll show how to use various tools to detect the servlet causing the problem and even identity the specific request which causes the code to malfunction. A similar mechanism can be used to debug for example JDBC issues and can help in the identification of which connection causes an issue. The tools used do not require the server to be fully started.

SOA Suite 12.2.1: A first look at end-to-end JSON support in SOA Composites

SOA Suite 12.2.1 introduces end-to-end JSON support in composites, support for JavaScript in BPEL and a JavaScript embedding activity. The REST-binding (which can be used by Service Bus, BPEL, BPM) can receive and send untyped JSON without the need to translate it to XML. In BPEL, JavaScript can be used as expression language in various activities and there is a JavaScript embedding activity available.

In this article I'll show some examples on what you can do with this end-to-end JSON support and give some examples on how to use JavaScript in your BPEL process.

Quick overview of SOA Suite 12.2.1 new features

Oracle has just released SOA Suite 12.2.1 which contains several exciting new features. The below entries have shamelessly copied from the developers guide in order to provide a quick overview of highlights for this release of the SOA Suite. Also at the end of the article some links for new features of WebLogic Server 12.2.1 which has also been released.

Patching running instances

See Patching Running Instances of a SOA Composite.

Oracle SOA Suite 12c (12.2.1) supports Composite Instance Patching, which enables you to patch running instances of a composite and recover faulted instances after patching the runtime. You can only include those fixes in the patch that are compatible with Composite Instance Patching. Use the SOA Patch Developer role in Oracle JDeveloper to make the fixes and create the patch.

Composite Instance Patching enables you to deliver urgent composite fixes that can be picked up by long running instances. You can make compatible/allowed changes without aborting in-flight instances. If a patched running instance comes across a business process that has been fixed by the patch, say a BPEL transformation, then it picks up the fixes applied to the business process.

When designing the patch, the SOA Patch Developer mode in JDeveloper automatically disables changes that cannot be made to the patch. Some of the compatible changes that you can make include:

Non-schema related XSLT changes, changes to fault policy, sensor data, and analytics data.
Compatible BPEL changes such as transformation activity, assign operations, etc.
JCA Adapter configuration properties.

In-Memory SOA

See Using In-Memory SOA to Improve System Performance.

You can leverage the Coherence cache associated with WebLogic Server to run your non-transactional business processes in memory. This improves performance and scalability for these business processes, as read and write operations are performed out of the cache. Database performance and management also improves, as the costs associated with continuous disk reads and writes are significantly reduced.

In-memory SOA enables short-running processes to live in memory. The process state gets written to the database only when faulted, or at regular, deferred intervals using a write-behind thread. The BPEL state information is dehydrated and rehydrated to/from the Coherence cache.

Support for debugging XSLT maps

See Debugging the XSLT Map.

Starting in 12.2.1, you can debug your XSLT maps using the SOA Debugger. You can add breakpoints at strategic locations in the XSLT map. When debugging, the debugger halts execution at the breakpoints, enabling you to verify the data and output.

XSLT maps can be complex, making them difficult to debug. For example, you may have a Java function, or other functionality, that is best tested in the application server. Also, you might find it easier to debug in the application environment, as the XSLT may be invoked from many different applications in the server. The SOA debugger provides remote debugging capability for XSLT maps that have been deployed in the application server.

You can also use the debugger with your Oracle Service Bus projects.

Support for End-to-End JSON and JavaScript

See Integrating REST Operations in SOA Composite Applications.

Starting in 12.2.1, your SOA composites can use end-to-end JSON. This means that the REST service can receive the REST request and route it to the BPEL engine without translating it to XML. The BPEL component can use the JavaScript action, and also use JavaScript in conditional and iterative constructs, to work on JSON objects directly. The REST reference can receive the REST message from the BPEL engine and route it to an external REST endpoint without translation.

The REST interfaces and BPEL component support end-to-end JSON. However, if you are using other service components, like the Mediator, you need to use the 12.1.3–style composite that internally maps REST resources and verbs to WSDL operations and XML schemas, and translates the incoming payload into XML.

Running on WebLogic Server 12.2.1

Of course SOA Suite 12.2.1 runs on WebLogic Server 12.2.1 which also has several Interesting new features. See here.

• Multi-tenancy support
• Java EE 7 Support
• JDK 8 Certification
• Docker Certification

Among several other interesting things. Interesting to read the WebLogic full client is being deprecated.

How to get most out of your PaaS solution

Many companies are starting to implement PaaS (platform as a service) solutions. There are obvious benefits such as easy patching, scaling, pay per use, etc. There are also challenges when implementing a PaaS solution. In this post I will describe some of the challenges and provide some suggestions to allow you to enjoy your PaaS solution to the fullest.

The post is based on my presentation of a customer case on the Oracle Cloud day the 6th of October in the Netherlands this year.

https://www.oracle.com/cloudday/index.html

This customer had implemented a private PaaS solution and had faced several challenges. In this post I would like to present some of my findings (mainly based on interviews) in order to help customers who are considering a PaaS solution to make a good start with their implementation. The challenges / benefits are valid for private PaaS and public PaaS solutions.

How to use WLST as a Jython 2.7 module

WebLogic Scripting Tool (WLST) in WebLogic Server 12.1.3 uses Jython version 2.2.1 (based on Python 2.2.1). This can be an important limitation when using WLST. Many modules are not available for 2.2.1 or are difficult to install. See here for an example. WLST however can be used as a module in Jython 2.7. This allows you to use all kinds of nice Jython 2.7 goodness while still having all the great WLST functionality available.

To just name some nice Jython 2.7 features:

• pip and easy_install can be used to easily add new modules
• useful new API’s are available such as xml.etree.ElementTree to allow XML processing, the multiprocessing module to use multiple threads and the argparse module to make parsing of script arguments easy.

In this article I’ll describe how you can use WLST as a Jython 2.7 module in order to allow you to combine the best of both worlds in your scripts.

Create a release of artifacts. Automate adding Maven dependencies

"Continuous Delivery (CD) is a software engineering approach in which teams keep producing valuable software in short cycles and ensure that the software can be reliably released at any time." (from here)

Software artefacts are developed using a build pipeline. This pipeline consists of several steps to provide quick feedback on software quality by means of code quality checks, automated tests, test coverage checks, etc. When the software is done (adhering to a Definition of Done (DoD)), it is released. This release as a whole is then tested and promoted until it reaches a production environment. In the meantime, work on a next release has already started. This process is shown in the below image. This is a minimal example (especially on the test part). In the below image, you can see there are 3 releases in progress. This is only in order to illustrate the process. You should of course try to limit the number of releases which are not in production to reduce the overhead of fixes on those releases.

BPM Suite 12c: Oracle Adaptive Case Management: Monitoring Case Events

Oracle Adaptive Case Management (ACM) is an interesting addition to Oracle BPM Suite which has been introduced in 11.1.1.7. Adaptive Case Management is suitable to model complex work-flows in which there is no set order of activities taking place. This allows more control to the end user on what to do when.

When a case is started, it is a running process in the SOA infrastructure. The main component is Oracle Business Rules which governs (among other things) the availability of activities and when certain process milestones are achieved. The case API allows you to query the case events and milestones (how you can expose the API as a service is described here and here by Roger Goossens).

Sometimes people want to obtain information about cases such as;

• in how many cases has a certain activity been executed?
• in which cases has a certain milestone been reached?

Cases can crash, be restarted, migrated, aborted, purged, etc. Sometimes you might not want to depend on the running case being there to provide the information you want. Also using the API every time you want certain information might put a serious strain on your system. Using sensors or BAM might help but they require an investment to implement and are still manual implementations with no guarantee you can obtain information in the future you did not think you would need in the present/past.

SOA Suite 12c: Best practices for project structure and deployment

Efficient usage of version control has specific requirements to allow identification of versions and synchronous development on different branches. Design time you will want to have your Service Bus projects in a single application in order to allow usage of shared objects. At deploy-time or when creating a release, you want to group SCA composites together with Service Bus projects. How do you combine these different requirements?

In this article I'll describe several practices and considerations which can help you structuring your version control and artifact repository. The main challenge is finding a workable balance between the amount/complexity your deployment scripts and developer productivity / focus on business value. A lot of scripts (large investment) can make it easy for developers on the short term, however those scripts can easily become a burden.

If you are just looking for some good practices to structure your version control and artifact repository, look at the list below. If however you want to know why I think certain things are good and bad practice, read on.

SOA Suite 12c: Collect & Deploy SCA composites & Service Bus artifacts using Maven

An artifact repository has many benefits for collaboration and governance of artifacts. In this blog post I will illustrate how you can fetch SCA composites and Service Bus artifacts from an artifact repository and deploy them. The purpose of this exercise is to show that you do not need loads of custom scripts to do these simple tasks. Why re-invent a wheel when Oracle already provides it?

This example has been created for SOA Suite 12.1.3. This will not work as-is for 11g and earlier since they lack Maven support for SOA Suite artifacts. In order to start using Maven to do command-line deployments, you need to have some Oracle artifacts in your repository. See http://biemond.blogspot.nl/2014/06/maven-support-for-1213-service-bus-soa.html on how to put them there. I have used two test projects which were already in the repository. A SCA composite called HelloWorld_1.0 and a Service Bus project also called HelloWorld_1.0. In my example, the SCA composite is in the GroupId nl.amis.smeetsm.composite and the Service Bus project is in the GroupId nl.amis.smeetsm.servicebus.

Overview of WebLogic RESTful Management Services

Inspired by a presentation given by Shukie Ganguly on the free Oracle Virtual Technology Summit in July (see here); "New APIs and Tools for Application Development in WebLogic 12c", I decided to take a look at an interesting new feature in WebLogic Server 12c: the RESTful Management Services. You can see here how to enable them. In this post I will provide an overview of my short study on the topic.

RESTful management services consist of two sets of resources. tenant-monitoring resources and 'wls' resources. The first is more flexible in response format (JSON, XML, HTML) and more suitable for monitoring. With the latter you can for example update datasource properties and create entire servers. It however only supports JSON as return format. The 'wls' resources also provide links so you can automagically traverse the resource tree which is very useful. I've provided a Python script to do just that at the end of this post.

Sonatype Nexus: Delete artifacts based on a selection

Sonatype Nexus provides several mechanisms to remove artifacts from the repository. You can schedule a job to keep only specified number of the latest releases (see here). You can also specifically remove a single artifact or an entire group using the API (see here). Suppose you want to make a selection though. I only want to delete artifacts from before a certain date with a specified groupid. In this article I have provided a Python 2.7 script which allows you to do just that.

The script has been created for my specific sample situation. Yours might differ. For example, I have only used the Releases repository and no snapshot versions. First check if the artifacts are the ones you expect to be selected based on your criteria before actually performing the artifact deletion. If they differ, it is easy to alter the script to suit your particular needs.

You can download the NetBeans 8.0.2 project containing the code of the script here. I've used the NetBeans Python plugin you can find here. Also I have not used any third party Python libraries so a default installation should suffice.

Continuous delivery culture. Why do we do the things we do the way we do them?

Usually at first there is a problem to be solved. A solution is conjured and implemented. After a while, the solution is re-used and re-used again. It changes depending on the person implementing it and his/hers background, ideas, motives, likes and dislikes. People start implementing the solution because other people do it or someone orders you to do it. The solution becomes part of a culture. This can happen to such extents that the solution causes increasing amounts of side effects, other new problems which require new solutions.

WebLogic Server and OpenLDAP. Using Dynamic groups

Dynamic groups in an LDAP are groups which contain a query to specify its members instead of specifying every member separately. Efficient usage of dynamic groups makes user maintenance a lot easier. Dynamic groups are implemented differently in different LDAP server implementations. Weblogic Server can be configured to use dynamic groups in order to fetch users for a specific group. In this blog I will describe how dynamic groups can be created in OpenLDAP and used in Weblogic Server.

In this example I use two users. smeetsm the developer and doej the operator. As shown in the image below, there are many servers which follow a similar access pattern for operators and developers. We are considering a case here where users do not use a shared account (e.g. weblogic) to login to different systems. This is for trace-ability and security purposes a better practice than when everyone uses the same shared user. See http://otechmag.com/magazine/2015/spring/maarten-smeets.html for a more thorough explanation on why you would want this.

A small note though. I'm a developer and this is not my main area of expertise. I have not implemented this specific pattern in any large scale organization.

Authentication using OpenLDAP. Weblogic Console and BPM Worklist

In this blog I will illustrate how you can configure Weblogic Server to use OpenLDAP as authentication provider and to allow OpenLDAP users to login to the Oracle BPM Worklist application. In a previous blog I have already shown how to do Weblogic Authentication with ApacheDS (http://javaoraclesoa.blogspot.nl/2014/08/ldap-and-weblogic-using-apacheds-as.html). In this blog I will use OpenLDAP to also do BPM Worklist authentication.

Unleash the power of Java API's on your WLST scripts!

Oracle SOA Suite and many other Oracle products have extensive Java API's to expose their functionality. WLST can often be used for relatively course grained actions.  WLST (the version supplied in Weblogic 12.1.3) uses Jython 2.2.1. Jython is the Python scripting language implemented on the Java Virtual Machine. Jython allows easy integration with Java. In this article I describe how you can unleash the power of these Java API's on your WLST scripts!

Searching Oracle Service Bus Pipeline Alert contents

There are several ways monitor messages passing through the Service Bus. Using pipeline alerts is one of them. Pipeline alerts can be searched in the Enterprise Manager based on several parameters such as summary or when they have occurred. Usually an important part of the message payload is saved in the content of the alert. This content can not be searched from the Enterprise Manager. In this post I will provide an example for logging Service Bus request and response messages using pipeline alerts and a means to search alert contents for a specific occurrence. The example provided has been created in SOA Suite 12.1.3 but the script also works in SOA Suite 11.1.1.6.

Combine version control (SVN) and issue management (JIRA) to improve traceability

Version control and bug tracking systems are found in almost every software development project. Both contain information on release content. In version control, it is usual (and a best practice) to supply an issue number when code is checked in. Also it allows identification of code which is in a release (by looking at release branches). Issue management allows providing metadata to issues such as the fix release and test status. This is usually what release management thinks is in a release.

In this article I will provide a simple example on how you can quickly add value to your software project by improving traceability. This is done by combining the information from version control (SVN) and issue management (JIRA) to generate release notes and enforcing some version control rules.

To allow this to work, certain rules need to be adhered to.

• code is committed using a commit message or tag which allows linking of code to issue or change
• it should be possible to identify the code which is part of a release from version control
• the bug tracking system should allow a selection of issues per release

Exposing JMS queues and topics with a JAX-WS webservice

Everyone can do HTTP calls and thus call most webservices. Interfacing with JMS queues or topics though is a bit more difficult (when not using Oracle SOA Suite). An alternative is using custom code. This usually requires libraries, JNDI lookups, opening connections and such. Because I wanted to make it easy for myself to put stuff on queues and topics, I created a simple JAX-WS wrapper service. By using this service, JMS suddenly becomes a whole lot easier.

Deploying SOA Suite 12c artifacts from Nexus

SOA Suite 12c introduces Maven support to build and deploy artifacts. Oracle has provided extensive documentation on this. Also there already are plenty of blog posts describing how to do this. I will not repeat those posts (only shortly describe the steps). What I couldn't find quickly enough though was how to deploy artifacts from an artifact repository to an environment. This is a task often done by provisioning software such as Puppet or Jenkins. Sometimes though you want to do this from a command-line. In this post I'll briefly describe steps required to get your Continuous Delivery efforts going and how to deploy an artifact from the Nexus repository to a SOA Suite runtime environment.

Oracle introduces API Manager!

Oracle has introduced a new product; API Manager (you can find the official documentation here). API Manager is an important addition to the already impressive Oracle SOA stack. In this article I'll explain what this new product does and how it helps in managing your API's. I will focus on the features and benefits you can have of this product and will also elaborate a little about my current experiences with it.

Sonatype Nexus: Retrieving artifacts using the REST API or Apache Ivy

Sonatype Nexus is an often used artifact repository. In a previous blog post I have shown an example how Maven can be used to assemble and release artifacts to Nexus. In this blog post I will describe two ways how artifacts can be fetched from the Nexus repository; by using the REST API and by using Apache Ivy.

Oracle SOA Suite 12c: SOA instance purging with the Java API in multiple threads

Let's tackle a single topic at a time in this blog post ;) SOA Suite 12c provides a Java API to perform operations on the SOA infrastructure and things running there such as composites and instances. There are several blog posts available on how to do this in SOA Suite 10g and 11g. However since 12c is relatively new, I decided to try if I could easily access the Java API in 12c and if the API was still similar to 11g or had undergone major changes. As a usecase I decided I wanted to purge instances. This is usually done by database scripts on the SOA infra database. If however you do not have direct access to the database, this can be used as an alternative. I could of course have done the same by using WLST. I found that purging instances by using the Java API was slow so I decided I would like to try this using multiple threads working in parallel. The resulting code is provided in this post.

Oracle SOA Suite 12c: The Coherence Adapter

Oracle SOA Suite 12c ships with a new technology adapter, the Oracle CoherenceAdapter. This adapter allows easy integration with a Coherence Data Grid (a distributed cache). The excellent post by Antony Reynolds provides the information required to setup your Coherence cache and get started with the adapter. In this blog post I will describe a pattern which can be used to use Coherence Data Grid as a cache for a database table (or view, or...). Also I did some performance measures on the Oracle quickstart JDeveloper installation with Integrated Weblogic server with surprising results!