Graceful shutdown of forked workers in Python and JavaScript running in Docker containers

You might encounter a situation where you want to fork a script during execution. For example if the amount of forks is dependent on user input or another specific situation. I encountered such a situation in which I wanted to put load on a service using multiple concurrent processes. In addition, when running in a docker container, only the process with PID=1 receives a SIGTERM signal. If it has terminated, the worker processes receive a SIGKILL signal and are not allowed a graceful shutdown. In order to do a graceful shutdown of the worker processes, the main process needs to manage them and only exit after the worker processes have terminated gracefully. Why do you want processes to be terminated gracefully? In my case because I store performance data in memory (disk is too slow) and only write the data to disk when the test has completed.

This seems relatively straightforward, but there are some challenges. Also I implemented this in JavaScript running on Node and in Python. Python and JavaScript handle forking differently.

Oracle Mobile Cloud Service (MCS). Implementing custom APIs using JavaScript on Node.js.

Oracle Mobile Cloud Service is a mobile backend as a service. MCS does its magic by providing a lot of features to make implementing mobile services easy such as (among many other) authentication, logging/analytics, lookups and calling other services. There are also features available to make integration with mobile clients easy such as providing an easy way to implement push notifications.

Personally I think one of the most powerful features of MCS is the ability to write custom JavaScript code and use that as an API implementation. This custom code can (among the regular JavaScript features) call MCS connectors and platform services. This provides a lot of flexibility in defining API behavior.

In this blog I will show how you can use this custom Node.js code to create an end to end example. I will use a RAML file to define my interface. Next I will define a connector in MCS to call the OpenWeatherMap API. This API returns (amongst other things) the temperature at a location in Kelvin. I want to define my own custom result message (with the temperature in Celsius) which better matches the requirements of my mobile client. I will use a custom JavaScript implementation to call the connector which calls the OpenWeatherMap API and create a custom response message from the result.

The described example is not suitable for a production implementation and is based on limited experience (and watching some really nice YouTube presentations). It is provided to give an idea on how to get started easily with a simple working example.

Simple IoT security system using Raspberry Pi 2B + Razberry + Fibaro Motion Sensor (FGMS-001)

In this article I'll describe how I created a simple home-brew burglar detection system to send me a mail when someone enters my house (so I can call the police). First my choice for the components is explained. Next how these components combine to achieve the functionality wanted. Based on this article you should be able to avoid certain issues I encountered and have a nice suggestion for a simple relatively cheap burglar detection system.

My purpose was to create a simple security system based on a Raspberry Pi. A Raspberry Pi is a tiny computer which can run a Debian like Linux distribution called Rasbian. I wanted to avoid going to low-level into sensor configuration and programming. That's why I decided early on to use an extension board and not directly attach the sensors to the Raspberry Pi. I decided to go for the Razberry. I also looked at the GrovePi and Arduino. Both are still too low-level for my tastes though. The Razberry is an extension board for the Raspberry which provides a Z-Wave controller chip. Z-Wave is a wireless protocol popular in the area of home automation. This was an attractive option since if in the future I would want to use additional sensors or maybe even use a commercial home automation system, I could very well get compatibility out of the box. For the sensor, I decided on the Fibaro FGMS-001 Motion Sensor. This is a multi-sensor which allows detection of motion, temperature, luminiscence and vibrations. It can even detect tampering and earthquakes (which is relevant since I live in the Dutch city of Groningen).

Z-Wave.Me (the company providing the Razberry), provides software for the Razberry called Z-Way. There are several alternatives. One of the most popular seems to be Domoticz which is provided with OpenZWave. Domoticz allows quite extensive home automation but I was having difficulty getting the sensor to work with OpenZWave so I decided to go with Z-Way. Z-Way supported the sensor out of the box. With the Z-Way server however it was difficult to automate actions based on sensor values. How I solved this is also described in this article.

SOA Suite 12.2.1: A first look at end-to-end JSON support in SOA Composites

SOA Suite 12.2.1 introduces end-to-end JSON support in composites, support for JavaScript in BPEL and a JavaScript embedding activity. The REST-binding (which can be used by Service Bus, BPEL, BPM) can receive and send untyped JSON without the need to translate it to XML. In BPEL, JavaScript can be used as expression language in various activities and there is a JavaScript embedding activity available.

In this article I'll show some examples on what you can do with this end-to-end JSON support and give some examples on how to use JavaScript in your BPEL process.

Webinterface REST/JSON vs Middleware SOAP/XML

Introduction

JavaScript

Webdevelopers use a lot of JavaScript on the clientside. There are things which JavaScript is good at and there are things you're better of not doing in JavaScript. JSON and REST are often used by JavaScript developers and XML and SOAP are shunned.

JSON

JSON (see http://www.json.org/), the JavaScript Object Notation, is an easy lightweight notation which can be used to transfer JavaScript objects; objects can be transported as strings. JSON is a lot easier for JavaScript then for example XML. JSON has some query languages (such as XQuery / XPATH for XML) which are still work in progress but seem to work to some extend; http://stackoverflow.com/questions/777455/is-there-a-query-language-for-json. For Java there are libraries available to make working with JSON more easy (such as JSON-lib and Jackson JSON).

REST

Since webinterfaces nowadays often require asynchronous interaction with the server to for example validate values in forms without requiring a submit of the entire form, services on the server are often deployed.

For service interaction, JavaScript can easily do HTTP POST and GET requests but it is harder to create entire SOAP messages. An example on how asynchronous HTTP requests can be done from JavaScript; http://rest.elkstein.org/2008/02/using-rest-in-javascript.html

REST services (http://en.wikipedia.org/wiki/Representational_state_transfer) are an alternative to SOAP services. REST services often do not have an interface defined (WSDL and WADL are disputable for REST services); there is often less strict message exchange (if you want to validate messages or throw exceptions, you have to do it yourself, they are application specific and not inherent to the protocol). For the sake of quick (webinterface) development, less strict standards can be a choice.

Oracle SOA and webinterfaces

This is an Oracle SOA / Java blog, so where do those fit in? Well, Middleware often has to link in some way to the Frontend or Frontend oriented backend systems. The Oracle SOA Middleware is heavily XML/SOAP based and the Frontend likes JSON and REST services a lot better. This is a gap which needs to be bridged.

SOAP and REST services

Oracle supplies an HTTP binding adapter which can be used to send and receive HTTP requests. See http://docs.oracle.com/cd/E25054_01/dev.1111/e10224/sca_bindingcomps.htm#autoId3 for more information. This adapter allows sending/receiving HTTP GET and POST requests and is capable of receiving XML from REST services. The Spring component can also be used for REST service interaction and since it is custom Java code which is used, it can more easily be expanded with additional functionality; http://technology.amis.nl/2009/12/16/soa-suite-11g-using-spring-component-to-mimic-http-binding-and-integrate-restful-services/

JSON and XML

Java

When examining the differences between JSON and XML, one major problem arises; there can be no strict fixed conversion between the two. See for example the below article on a user forum of one of the better known/performing JSON libraries (Jackson JSON); http://jackson-users.ning.com/forum/topics/xml-to-json-conversion-using; JSON and XML information models are fundamentally incompatible. Jackson JSON is good at converting JSON to Java objects and the other way around. Converting to XML however will most likely not be implemented because of the difference in information models.

JSON-lib (http://json-lib.sourceforge.net/) however does allow a conversion of JSON to XML and the other way around (although not reversible) with relatively little code. Of course certain assumptions need to be made to allow the conversion to work. If you're interested, you can read more about this on; http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html

XSLT

Going from XML to JSON is specific and can be modeled using XSLT to be able to support different conversion patterns. See http://controlfreak.net/xml-to-json-in-xslt-a-toolkit/ for some useful examples.

Conversion JSON to XML and XML to JSON webservice

I wanted to be able to easily convert JSON from the webinterface and REST services (Java code) to XML in order to be more flexible in BPEL with for example XPATH and XSLT. When I have XML, I can use XPATH to create the relevant JSON again if the default transformation is insufficient. I created a webservice based on JSON-lib to do the conversion. I used JAX WS to make developing/exposing the methods as a webservice easy. The code required is the following;

package ms.testapp.jsonxml;

import java.io.InputStream;
import java.io.InputStreamReader;

import javax.jws.WebMethod;
import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;

import net.sf.json.JSON;
import net.sf.json.JSONArray;
import net.sf.json.JSONObject;
import net.sf.json.JSONSerializer;
import net.sf.json.xml.XMLSerializer;

import org.apache.commons.lang.StringEscapeUtils;

@WebService
public class JsonXml {
   
    @WebMethod
    @WebResult(name = "xml")
    public String JsonToXml(@WebParam(name = "namespace")String namespace, @WebParam(name = "rootelement")String rootelement,@WebParam(name = "jsonstring")String jsonStr) {
            XMLSerializer serializer = new XMLSerializer();
            serializer.setRootName(rootelement);
            serializer.addNamespace("", namespace);
            JSON json = JSONSerializer.toJSON(jsonStr);
            String xml = serializer.write(json);
           
            return xml;
    }

    @WebMethod
    @WebResult(name = "json")
    public String EscapedXmlToJson(@WebParam(name = "xmlstring")String xmlStr) {
        XMLSerializer xmlSerializer = new XMLSerializer();
        JSON json = xmlSerializer.read(StringEscapeUtils.unescapeXml(xmlStr)); 
        return( json.toString() ); 
    }
   
    @WebMethod(exclude=true)
    public String XmlToJson(@WebParam(name = "xmlstring")String xmlStr) {
        XMLSerializer xmlSerializer = new XMLSerializer(); 
        JSON json = xmlSerializer.read( xmlStr ); 
        return( json.toString() ); 
    }
/*
    public static void main(String[] args) {
        String Json = "{'foo':'bar',\n" +
        " 'coolness':2.0,\n" +
        " 'altitude':39000,\n" +
        " 'pilot':{'firstName':'Buzz',\n" +
        "          'lastName':'Aldrin'},\n" +
        " 'mission':'apollo 11'}";
        String Xml = new JsonXml().JsonToXml("http://www.google.com","root",Json);
        System.out.println(Json);
        System.out.println(Xml);
        Json = new JsonXml().XmlToJson(Xml);
        System.out.println(Json);
        String escapedXml = StringEscapeUtils.escapeXml(Xml);
        System.out.println(escapedXml);
        Json = new JsonXml().EscapedXmlToJson(escapedXml);
        System.out.println(Json);
       
    }
*/
}

The full code is available here; https://dl.dropbox.com/u/6693935/blog/JsonXmlApp.zip. There is also a JAR deployment profile (in addition to the WAR deployment profile) in the package and the dependencies are included. For Java developers, it is recommended (for performance reasons to avoid webservice overhead) to use the Jar deployment profile. From BPEL (in a service oriented landscape) it is more easy to have the functionality available as a webservice. The main method has been used as a sort of unittest.

Please mind, the conversion is not reversible and specific.I personally think the JSON to XML conversion is more useful then the XML to JSON conversion since that can be achieved more specifically with XSLT. I have not exposed the XmlToJson method since it's hard to call the method as a webservice with an XML parameter containing a non-escaped XML fragment.

The method of converting JSON to XML, has the benefit of not requiring an XSD schema describing the XML for the conversion and not requiring a Java object representation of the data. This has the benefit that one service/class is enough to do the conversions different messages (at least JSON to XML).

Further suggestions; JSON and BPEL

When you want to use the XML created by converting JSON in BPEL, you need to do the following;
- create a schema for the created XML message and include it in a used WSDL file
- create a variable using this schema
- use ora:parseEscapedXML to assign the output of the JsonToXml to the variable
- use the variable however you like

Because this can be cumbersome (creating the schema from the XML message, JDeveloper has a wizard but still...), it's advisable to create another webservice with two input variables; JSON string and JSON path expression returning the result of the JSON path expression on the JSON string. This can easily be accomplished by using for example http://code.google.com/p/json-path/. This way if you want to use a single variable from the JSON string, you can query for it directly and you don't need to create an XSD schema for the result of the JSON to XML conversion since it is a simple string. Use can use something like; String result = JsonPath.read(jsonstring, jsonpathstring).toString(). This is however not recommended if you need a lot of variables from the JSON string since calling a webservice is also overhead and might cost performance.

Packaging the above JSON samples in an XPATH library (see for example http://docs.oracle.com/cd/E23943_01/dev.1111/e10224/bp_appx_functs.htm) might be a better solution then wrapping them in webservices since it further increases their ease of use and performance in BPEL. If the services need to be re-used outside of BPEL by for example other middleware products, webservices might be the better solution.