Oracle SOA 11g BPEL transaction semantics and performance

In this post I'll provide a simple integration example and provide some suggestions to optimize the performance. Optimization suggestions are focused on transaction semantics. It's purpose is to indicate the importance to take into account various settings related to transaction management.

The base and inspiration for this post are the presentations and material from SOA Blackbelt training which was given by Oracle in Berlin this year from the 11th to the 14th of June. The training covered a lot of material in great depth. If you have the chance to follow it, I highly recommend it!

Test setup

First I enqueue 2000 messages on an Oracle AQ and take a timestamp which I write in a separate table in the same transaction. After a COMMIT, a BPEL process is triggered and picks up the messages (one instance per message). This process puts the message in a table. The moment of insertion is determined by having a default value on a field in the table. I then determine the time difference between the last message put in the table in the batch and the moment of insertion in the AQ. To avoid the overhead of audit logging, I turn this off for the specific process. The code used can be downloaded at the end of this post.

I will vary the bpel.config.oneWayDeliveryPolicy. I will try sync and async.persist (the default). async.persist will first put dequeued messages in the DLV_MESSAGE table before they are further processed in a separate transaction. sync will not do this and will invoke the BPEL process synchronously. I use three different datasource settings for this test. I will try both oneWayDeliveryPolicy settings with an XA datasource (which uses a 2-phase commit) and two non XA datasources. For the non XA datasource I will test with and without Global Transaction support. I will test this using different datasources and the same datasource. I will also test all combinations with the bpel.config.transaction setting to required and requiresNew.

Summarized; four different settings are varied in all combinations of the other settings. Two measures are taken with each combination.
- bpel.config.oneWayDeliveryPolicy (async.persist, sync)
- bpel.config.transaction (required, requiresNew)
- different datasource settings; XA, NonXA, NonXA no global transaction support
- using the same and different datasources with the same settings

I created 6 datasources with the three different settings;
testuserXa, testuserXa2
testuserNonXa, testuserNonXa2
testuserNonXaGlobal, testuserNonXaGlobal2

Next I created 6 connection factories for the DbAdapter and 6 connection factories for the AqAdapter. I varied the datasources in the JCA files in the BPEL process I created. For every test I primed the datasources/engine with 100 messages. Next I took 2 measures of 2000 messages.

Results

The * indicates the process failed with the following exception;
BINDING.JCA-11616
DBWriteInteractionSpec Execute Failed Exception.
insert failed. Descriptor name: [WriteToStore.TestStore].
Caused by java.sql.SQLException: Cannot call Connection.commit in distributed transaction.  Transaction Manager will commit the resource manager when the distributed transaction is committed..

As can be seen, using an XA datasource decreased performance. Also using the sync property increased performance. In this example, little effect was visible when changing the transaction property. There was little difference in using two different datasources instead of a single datasource for processing. The NonXa datasource with global transaction support executed an explicit commit (apparently) which conflicted with the distributed nature of the transaction. This happened in all cases when performing an insert action using this datasource. This also indicates the transaction was distributed in all cases (even when using a Non Xa datasource without global transaction support).

Conclusion

When using the oneWayDeliveryPolicy setting of sync, the entire process is processed in a single transaction. When using async.persist, 2 transactions are involved. One to write to the DLV_MESSAGE table and one to call the DB insert.

The performance impact of writing to DLV_MESSAGE and the extra transaction was measurable. When calling a process synchronously, the effect would have been greater since then it would have been 4 transactions when using the async.persist setting.

Using an XA datasource means a two-phase commit is used. This has a slight overhead which is measurable in this example. It also provides a difference in behavior, mainly in the event of a fault. See for example; http://soaranch.wordpress.com/2010/10/08/global-and-local-transactions-in-oracle-soa-11g-composites/.

Because of the process setup, I could not measure much effect on the transaction setting since the process initialization by the Aq adapter always starts a new transaction. When a process is called from another process, I would have expected to see a performance gain with the 'required' setting since then I woukld have expected less transactions.

I would have liked to see if the DbAdapter and AqAdapter behaved differently when different datasources on the same database schema were used to connect instead of the same datasource. The only difference found however was that when using async.persist, requiresNew and using the same NonXA datasource without global transaction support, errors occurred. These errors did not occur when using different datasources (also NonXA without global transaction support). Apparently with these settings, an explicit commit is executed when performing an insert by using the DbAdapter. Also the incoming message was read using the AqAdapter and the result was written using the DbAdapter. These adapters both have their own connectionpools. Using the same adapter might have caused different behavior. Using different datasources might make it possible to have more open incoming connections. This however in this case was most likely limited by other settings such as invoker threads.

Something to mind when considering the transaction related settings is it's impact on fault handling. This is however not the focus of this post. See for example; http://docs.oracle.com/cd/E15523_01/integration.1111/e10224/soa_transactions.htm and https://blogs.oracle.com/soabpm/entry/soa_suite_11g_-_transactions_b

Of course many other settings can be tuned to increase performance. The effects found when changing the settings might also differ with the nature of the process tested. In this case, invoker threads can be increased to allow messages to be picked up faster and the maximum number of connections allowed by the datasources can be increased. The process can also be made transient. The list of other possible options to make this specific process go faster are numerous. The purpose of this example is however to indicate the impact transactions can have on the performance of a process so the other factors are kept constant and the default settings from the Oracle SOA 11g PS5 image are used; http://www.oracle.com/technetwork/middleware/soasuite/learnmore/vmsoa-172279.html

You can download the used code and test results below;
https://dl.dropboxusercontent.com/u/6693935/blog/Transactions.zip

Suggestions to increase BPEL performance

The performance improvements suggested in this post are partially based on suggestions from the following book; https://beatechnologies.wordpress.com/2012/08/28/oracle-soa-suite-11g-administrators-handbook/

If you're interested in how you can improve SOA Suite performance (and of course several other aspects of SOA Suite administration), I suggest reading it.

I've used the Oracle VM which can be downloaded from http://www.oracle.com/technetwork/middleware/soasuite/learnmore/vmsoa-172279.html  .

Performance

I've written about some performance suggestions earlier; http://javaoraclesoa.blogspot.nl/2012/06/increasing-bpel-performance.html. Recently I've been doing several measures on different systems specifically concerning BPEL performance and have updated the above post to reflect those findings. In this post I will describe some new conclusions.

Test scenario's

I've created 5 test scenario's;
- a dummy scenarion (empty BPEL activity)
- a synchronous locally optimized webservice call
- an asynchronous locally optimized webservice call
- a synchronous not locally optimized webservice call
- a database call

Every scenario I've tested in a blocking invoke and nonblocking invoke setting. I've performed every test 200 times in a parallel executed for-each loop and measured the time by saving the start and end time in a database table.

When I'm talking about 'performance improved' or 'performance worsened, I'm talking about all scenarios. I've not found settings which improved the performance for one test scenario and decreased it for another.

I've found that testing just after a server restart caused performance measures to worsen. This is most likely due to the connection pools which need to create initial database connections (which is slow...). That's why I've done all tests at least 3 times. When testing on the Oracle VM I've purged the dehydration store after every run in order to get the same initial situation for the tests.

The settings which increased the performance on my systems can of course be system specific. If other people have different experiences concerning some of the settings described in this post, it will be interesting to know. Also keep in mind that the settings I've found to improve performance on my system, might worsen the performance if other test scenario's are used. You should always test the settings on your own system to make sure they have the desired result and to confirm system stability is not effected. I've not tested in a clustered environment.

I have results concerning the performance improvements for reducing audit and log levels. Reducing audit and other logging however and can make it harder to debug problems. Putting BPEL process audit logging on 'Development' however in a production environment is in my opinion overkill and should be avoided or only used for a short period in order to solve issues.

Results

Locally optimized webservice calls

Locally optimized calls are faster then not locally optimized calls (non locally optimized calls have among other things SOAP overhead). I've not yet looked at this behavior when working in a cluster with a load balancer. The following might be interesting and would suggest local optimizations does not work through a load balancer if the server URL differs from the load balancer URL; https://blogs.oracle.com/rammenon/entry/is_local_optimization_kicking.It would be interesting to test this and try and make it work in clustered environments since the performance increase for using locally optimized calls is (not statistically tested) significant (about 25%).

Asynchronous callbacks

I've seen in my measures that asynchronous callbacks are a lot slower (about 5 times) then synchronous interactions. This is of course to be expected since correlation data needs to be saved and incoming messages need to be matched to the correlation data. If performance is important and processes are short-running, I'd advice to avoid asynchronous constructions.

nonBlockingInvoke

Setting the nonBlockingInvoke property on a partnerlink caused a decrease in performance. This is something I had not expected based on; http://docs.oracle.com/cd/E23943_01/core.1111/e10108/bpel.htm#ASPER99890. In my tests however, the services were fast and did little. It could be that if slower services were used, an improvement could be measured when using the nonBlockingInvoke option. I have however not tested this.

Tuning datasources (SOAINFRA schema)

My measures concerning the performance changes achieved when increasing the connection pool size of the SOALocalTxDataSource and SOADataSource are not consistent. On the Oracle VM I found that increasing the connection pool size, performance was improved. When trying this setting on a customer system however, performance was worsened. On the Oracle VM, a dedicated XE database is installed which is setup to allow plenty of sessions. On the customer system, the database was not only installed on a (most likely physically) different server but also not dedicated; several SOA Suite instances used the same database for their dehydration store. At the customer, the database sessions parameter (see for example https://forums.oracle.com/forums/thread.jspa?threadID=898395) was found to be limiting in the past. An educated guess is that this caused the performance decrease on that system.

One can conclude from the above that only looking at JDBC datasource settings on the Weblogic server is not enough to achieve consistent improvement across environments. One should also look at the database settings.

JVM settings

The following settings increased performance on all systems;

In setSOADomainEnv.sh

Default
PORT_MEM_ARGS="-Xms768m -Xmx1536m"
New
PORT_MEM_ARGS="-Xms1536m -Xmx1536m -Xgcprio:throughput -XX:+HeapDumpOnOutOfMemoryError -XXtlasize:min=16k,preferred=128k,wasteLimit=8k"
The following settings decreased performance (from http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/bestpractices.html);
-XXaggressive
-XXcallProfiling

BPEL Engine tuning

Threads
I've tried and measured several settings related to the BPEL engine. There are 4 settings related to the number of threads used by the BPEL Engine;
DispatcherSystemThreads
DispatcherEngineThreads   
DispatcherInvokeThreads   
DispatcherNonBlockInvokeThreads

Increasing them did not lead to performance improvements

AuditStorePolicy
I've also tried playing with the AuditStorePolicy setting. This caused errors or worsened performance when combined with several of the other audit settings I've tried (for example in combination with deferred audit logging as described in the post referred to in the introduction). I got the best results leaving this set to syncSingleWrite.

Coherence
Setting the QualityOfService from DirectWrite to CacheEnabled led to worsened performance.

OneWayDeliveryPolicy
Setting this from async.persist to async.cache improved performance

Asynchronous audit logging
See http://javaoraclesoa.blogspot.nl/2012/06/increasing-bpel-performance.html. It improves performance but check the log files to see if the audit store does not encoutner unique key constraints.

I've not tried altering audit thresholds. This can also lead to performance improvements.

Process improvements

The process of using, monitoring and maintaining the SOA Suite installation is maybe even more important in increasing the performance then providing technical tweaks.

Based on the topics discussed in the Administrators handbook (https://beatechnologies.wordpress.com/2012/08/28/oracle-soa-suite-11g-administrators-handbook/) there are several activities which are considered to be part of the SOA Administrators responsibilities. This might be interesting for customers to consider. If for example a SOA Administrator has little background in application server maintenance or the SOA Suite in specific, there's a high probability project developers are more knowledgeable in some area's. This often leads to shift of responsibilities to developers.

Some of the tasks mentioned as part of the SOA Administrators job are listed below. These usually end up being done by developers.
- automating composite management and structuring composite deployments using ant scripts
- using and managing configuration plans
- administration of services, engines and MDS
- tuning SOA Suite 11g (operating system, application server, SOA infrastructure, database connections, dehydration store)
- monitoring (sometimes 'it's up and running!' is not enough...)

Other suggestions

The book contains several other suggestions for increasing performance. Interesting is also how the SOAINFRA database/schema can be tweaked. I've however not tested/measured this.

Since BPEL often uses services from different systems, tweaking those services can increase the performance of the process as a whole. Most noteworthy here are of course databases and JDBC settings.