Wednesday, March 27, 2013

Monitoring Oracle SOA Suite 11g composites using HTTP polling

This post is about efficient vendor neutral monitoring of Oracle SOA composites by providing a mechanism which allows HTTP polling for the state/mode on a per composite basis. First I'll describe what could be the reason to monitor composite process state/mode. Then I'll provide code which can be used to allow HTTP polling of the process state/mode in order to link it to various monitoring tools or custom dashboards.

What is composite state?

There are two properties related to the behavior of a composite in relation to accepting requests and processing already running instances; composite mode and composite state. The mode can be active or retired and the state can be on or off. If the mode is retired, running instances will finish but no new instances can be created. Active means the service is ready to accept requests. If the state is off, the composite is not loaded (shutdown) and as such will not process running requests or pickup new requests. If the state is on, the composite is loaded. The mode and state can be manually changed from Enterprise Manager Fusion Middleware Control. They can also be changed by accessing the API for example as part of an exception handling mechanism.

In the below image you can see the state/mode of all the composites in a partition from Fusion Middleware Control;

In the below image you can see the buttons in Fusion Middleware control to retire/shutdown a composite. Once this is done, buttons for activating/starting the composite become visible.


Why can composite state be important to monitor?

Starting the server

When an Oracle SOA Suite server starts, resources can be pulled in and loaded as part of a composite. Such can be the case when external webservices are called and remote XSD's and WSDL's are used (references). If resources can not be loaded this way, the loading of the entire server might stall. In the Weblogic console however, the server will have the state RUNNING. The SOA Server however will not allow much actions from the webinterface since it is still starting. How to check the SOA serverstate with a servlet is illustrated on http://javaoraclesoa.blogspot.nl/2012/11/soa-suite-cluster-deployments-and.html. If the server is starting, usually the composite which is being started will have state off. Composites which have not been started yet will have state unknown. If the server hangs during start, this can help identifying the problem. Sometimes a manual undeploy is required of a faulty process. This is described on; http://shrikworld.blogspot.nl/2011/04/how-to-undeploy-composite-manually.html. Monitoring the state/mode of individual composites might provide an indication of start-up issues in addition to monitoring the entire SOA Server state.

Exception handling

When using certain methods of exception handling in Oracle SOA Suite 11g, processes can get the retired state in order to stop processing messages after a fault situation. For example, when using the method described on; http://javaoraclesoa.blogspot.nl/2012/06/exception-handling-fault-management-and.html. Usually an error event is triggered linked to sending an e-mail to the person responsible for solving the problem. This person however can take some time before taking action or checking/responding to his mail. In the mean time, the number of requests which need to be processed can increase, requiring more time after activating the composite again before the system is capable of processing new requests. Sometimes this mechanism might be sufficient but a little redundancy in informing people doesn't hurt.

Common monitoring tools

Organizations (especially the larger ones) often use monitoring tooling to determine the state of their server park. If a critical system fails, these monitoring tools immediately inform the person responsible for maintaining server stability, even at night or in weekends. This avoids the problem which can arise when the mail indicating a problem is overlooked. Also usually these monitoring tools have dashboards at which more people look, so the chance the problem is solved quickly increases.

Integrating the raising of the error event from a composite with specific monitoring tooling causes a direct dependency (tight coupling) between the monitoring tooling and the composite error handling mechanism. Also it will often require different disciplines/departments to achieve. Because of this, it is often not advisable to do.

The monitoring tools are not always from the same software vendor as the system which need to be monitored and thus specific components and states/modes are usually not fully supported. Certain common vendor neutral monitoring mechanisms are often used to allow monitoring of diverse systems. Usually these tools allow HTTP polling mechanisms. For example HP SiteScope (http://www8.hp.com/us/en/software-solutions/software.html?compURI=1174244#.UVKxsleyJh4).

Monitoring composite state by using HTTP polling

To monitor composite state via a HTTP polling mechanism, a HttpServlet can be deployed on the Oracle SOA server. The servlet provided in this post accepts an HTTP POST or GET request with the following parameters; name, partition, revision.

It then selects the composite and returns it's state in the response. It's code is based on the previously mentioned sample on; http://javaoraclesoa.blogspot.nl/2012/11/soa-suite-cluster-deployments-and.html. There the code is also explained.

You can test the process after deployment like (where of course hostname needs to be replaced and you should refer to a  composite which is present in your environment);
http://soabpm-vm:7001/DetermineBPELProcessStatus/determinebpelprocessstatus?name=HelloWorld&revision=1.0&partition=default

The output in my case is State: on, Mode: active

To test it's function, you can retire the composite and confirm the servlet returns; State: on, Mode: retired. You can also check the presence of a composite. If you provide a name/revision/partition which is not a valid composite, the State and Mode field will remain empty.

You can download the code here; https://dl.dropbox.com/u/6693935/blog/DetermineCompositeStatus.zip