Previously I wrote about generating random data which conforms to an AVRO schema (here). In a recent use-case, I encountered the situation where there were several separate schema files containing different AVRO types. The message used types from those different files. For the generation of random data, I first needed to merge the different files into a single schema. In addition, I wanted to generate Java classes for the complete message which required importing dependent types in the pom.xml. In this blog post I'll describe how I did that.
Articles containing tips, tricks and nice to knows related to IT stuff I find interesting. Also serves as online memory.
Sunday, February 6, 2022
Monday, March 1, 2021
Java Security: Open Source tools for use in CI/CD pipelines
It is often expected of a DevOps team to also take security into consideration when delivering software. Often however, this does not get the attention it deserves. In this blog post I'll describe some easy to use, CI/CD pipeline friendly, open source tools you can use to perform several checks during your Java software delivery process which will help you identify and fix issues with minimal effort.
You can view my sample project here which implements all these tools. There is also a docker-compose.yml file supplied. SonarQube en Jenkins however do not come preconfigured in this setup. You can look at the Jenkinsfile (pipeline definition) to see what Jenkins configuration is required (installing plugins and creating credentials).
This is provided as an overview and small example only. It is not meant as a walkthrough to setup a CI/CD environment or a manual how to use the different tools. Also the mentioned tools are not all the tools which are available but a set of tools which are freely available, popular and which I managed to get working without too much effort. Using them together allows you to improve the security of your Java applications during your software delivery process and provide quick feedback to developers. This can also help increase security awareness. When you have some experience with these tools you can implement more strict policies (which can let a build fail) and quality gates.
Wednesday, February 17, 2021
Java Agent: Rewrite Java code at runtime using Javassist
There are situations where you want to change Java code not at the source code level but at runtime. For example when you want to instrument code for logging purposes but do not want to change the source code (because you might not have access to it). This can be done by using a Java Agent. For example, Dynatrace uses a Java agent to collect data from inside the JVM. Another example is the GraalVM tracing agent (here) which helps you create configuration for the generation of native images. Logging is one use-case but you can also more dramatically alter runtime code to obtain a completely different runtime behavior.
This blog post is not a step by step introduction for creating Java Agents. For that please take a look at the following. In this blog post I have created a Java agent which rewrites synchronized methods to use a ReentrantLock instead (see here). The use-case for this is to allow applications to use Project Loom's Virtual Threads more efficiently.
You can find the code of the agent here.
Saturday, October 26, 2019
Oracle Database: Write arbitrary log messages to the syslog from PL/SQL
In *NIX systems syslog messages often end up in /var/log/messages. You can configure these messages to be forwarded to remote syslog daemons. Also a pattern which often seen is that the local log files are monitored and processed by an agent.
Oracle database audit information can be send to the syslog daemon. See for example the audit functionality. If you however want to use a custom format in the syslog or write an entry to the syslog which is not related to an audit action, this functionality will not suffice. How to achieve this without depending on the audit functionality is described in this blog post. PL/SQL calls database hosted Java code. This code executes an UDP call to the local syslog. You can find the code here.
Friday, August 9, 2019
Apache Camel and Spring Boot: Calling multiple services in parallel and merging results
Thursday, August 30, 2018
Oracle SOA: Sending delayed JMS messages
Monday, August 6, 2018
Running Spring Tool Suite and other GUI applications from Docker containers
Tuesday, May 5, 2015
Unleash the power of Java API's on your WLST scripts!
Friday, February 7, 2014
JAXB and XML generation; missing elements
If you generate Java classes from an XML schema definition, create an instance of the generated class and unmarshall it to XML, the XML will not in all cases be compliant to the schema the classes were generated from. This can cause problems if correctness is assumed based on usage of the framework. Validation can help detect problems and there are several methods to circumvent them.
Wednesday, June 12, 2013
Oracle BPEL and Java. A comparison of different interaction options
In the examples provided in this post, I'm using the JsonPath library (https://code.google.com/p/json-path/) inside a BPEL process. A usecase for this could be that a webclient calls the BPEL process with a JSON message and BPEL needs to extract fields from that message.
The Java code to execute, is the following;
package ms.testapp;
import com.jayway.jsonpath.JsonPath;
public class JsonPathUtils {
public String ExecuteJsonPath(String jsonstring, String jsonpath) {
String result = JsonPath.read(jsonstring, jsonpath).toString();
return result;
}
public JsonPathUtils() {
super();
}
}
Of course small changes were necessary for the specific integration methods. I provided code samples at the end of this post for every method.
Integration methods
Embedding
Oracle has provided an extension activity for BPEL which allows Java embedding. By using this activity, Java code can be used directly from BPEL. The JsonPath libraries to use in the embedding activity can be put in different locations such as the domain library directory or be deployed as part of the BPEL process. Different classloaders will be involved. To check whether this matters I've tried both locations.
Performance
The Java call happens within the same component engine. Below are measures from when using JSON libraries deployed as part of the BPEL process (in SCA-INF/lib).
Below are measures from when putting the libraries in the domain library folder.
As you can see from the measures, the performance is very comparable. The location where the BPEL process gets it's classes from has no clear measurable consequences for the performance.
Re-use potential
When the libraries are placed in the domain lib folder, they can be reused by almost everything deployed on the applications server. This should be considered. When deploying as part of the composite, there is no re-use potential outside the composite except possibly indirectly by calling the composite.
Maintenance considerations
Embedded Java code is difficult to maintain and debug. When deployed as part of a BPEL process, changes to the library require redeployment of the process. When libraries are put in the domain library directory, changes to it, impact all applications using it and might require a restart.
XPath
XPath extension functions can be created and used in BPEL (+ other components) and JDeveloper. This is nicely described on; https://blogs.oracle.com/reynolds/entry/building_your_own_path.
Performance
The custom XPath library is included as part of the SOA infrastructure and does not leave this context. As can be seen, the performance is comparable to the Java embedding method.
Re-use potential
The reuse potential is high. The custom XPath library can be used in different locations/engines, dependent on the descriptor file.
Maintenance considerations
Reuse by different developers in JDeveloper requires minimal local configuration, but this allows GUI support of the custom library. There are no specific changes to BPEL code thus low maintenance overhead. Changing the library on the application server requires a restart of the server.
Spring component
The Java code can be called as a Spring component inside a composite. Here another component within the composite is called. The Java code is executed outside the BPEL engine.
Performance
Re-use potential
The following blog posts links to options with the Spring Framework; https://blogs.oracle.com/rammenon/entry/spring_framework_samples. When deployed inside a composite, reuse is limited to the composite. It is possible to define global Spring beans however, increasing re-use. The code can be created/debugged outside an embedding activity.
Maintenance considerations
The Spring component is relatively new to Oracle SOA Suite so some developers might not know how to work with this option yet. It's maintenance options are better then for the BPEL embedding activity. It is however still deployed as part of a composite.
External webservice
Java code can be externalized completely by for example providing it as a JAX-WS webservice or an EJB.
Performance
Performance is poor compared to the solutions described above. This is most likely due to the overhead of leaving soa-infra and the layers the message needs to pass to be able to be called from BPEL.
Re-use potential
Re-use potential is highest for this option. Even external processes can call the webservice so re-use is not limited to the same application server.
Maintenance considerations
Since the code is completely externalized, this option provides the best maintenance options. It can be developed separately by Java developers and provided to be used by an Oracle SOA developer. Also it can be replaced without requiring a server restart or composite redeploy.
Conclusion
The technical effort required to implement the different methods is comparable. Depending on the usecase/requirements, different options might be relevant. If performance is very important, embedding and XPath expressions might be your best choice. If maintenance and reuse are important, then externalizing the Java code in for example an external webservice might be the better option.
Summary of results
Monday, September 3, 2012
JVM does not start on Windows Server 2008
A customer is using Windows Server 2008 64 bit and the Oracle JDK version 1.6.33 64 bit. Recently, the following error occurred when a (any) Java application was started;
Even just starting java.exe -version caused this exception. De-installing all present JDK's/JRE's and installing different (32 bit and 64 bit) versions, did not solve this problem. When we downgraded to a JDK 1.5 version however, the problem did not occur anymore but using a version 1.5 JDK was not a viable option.
The Exception Code: c0000094 is an INT_DIVIDE_BY_ZERO exception, so not much help there in clarifying the problem.
What we tried
There was no traceable change leading to this error.
We tried a clean install of the entire server (Windows reinstallation) and the JVM worked. Also after completely updating Windows, it still worked. We tried this locally with Virtualbox in a Virtual Machine instead of using the image/configuration of the machine on which the problem originally occurred. Multiple environments experienced the same problem. Also based on the below, our estimate is that this error is related to local (customer specific) policies (otherwise undoubtedly Google would have more information on this problem).
We encountered several errors when using Process Monitor; http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
What we saw was that in jvm.dll, different behavior started to occur. We could however not pinpoint the cause of this change of behavior. Decompiling the dll and running it with breakpoints on the server with Visual Studio, was not an option.
When we installed JRockit JDK, the problem was solved. To make the new JDK the default JDK used, several environment settings needed updating. We also use a Tomcat service which needed to be updated to use the new JDK.
Description
Install JRockit
First download the JRockit JDK;
Don't install Demo's, samples and sources (unless of course you will use them)
Install the public JRE (you might need the JRE some day and not the complete JDK)
Choose a directory to install the JRE
Update environment variables
Right mouse click the Computer icon, Properties (as you can see, I need a new laptop...).
Advanced system properties
Environment variables
Make sure no JAVA_HOME is present in the User variables. If JAVA_HOME is present under user variables, remove it.
If present, update and if not present, create the JAVA_HOME environment variable under System variables. Set it to be equal to the JDK installation directory. In this case; C:\Program Files\Java\jrockit-jdk1.6.0_33-R28.2.4-4.1.0.
Friday, March 30, 2012
Base64Encode and Base64Decode in BPEL
Introduction
Implementation
Make the required classes available in the classpath of the BPEL project.
Import the classes in BPEL
Create the Java embedding activity
Base64Encode
Base64Decode
Tuesday, March 27, 2012
File processing using a Spring component
The Oracle SOA Suite 11g FileAdapter provides (among other things) functionality to read a variety of text file formats such as CSV and transform them into XML. This makes it easier to use them in for example BPEL. Usually the FileAdapter provides enough functionality to correctly parse files. Sometimes however, this is not possible. An example of such a file;
h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2
h1 to h4 are header fields. The second and third line are the records and the last line indicates how many records are present in the file.
The last line cannot be excluded from parsing by the FileAdapter and will lead to errors. As a workaround and to provide some flexibility in file parsing, the following solution can be implemented. Keep in mind that this solution will cause problems for large files since the file will in memory be converted to XML.
Read the file as opaque using the FileAdapter. Create a Java class which has a public method with a byte array as input parameter and some parsing parameters. Wrap the Java class as a Spring component. Then you can use the component in a BPEL process as JDeveloper will generate XSD's/WSDL's for the Spring component for you.
Implementation
FileAdapter
In BPEL define a file adapter and select not to use a schema;
Java class parsefile
A Java class does the parsing. You can of course expand this example. Currently empty lines are also processed as entries, which might not be what you want. This class is also specific for the example given in the introduction (but is flexible enough for related file formats). I have chosen to return an ArrayList of ArrayLists. The first (outer) ArrayList contains the records of the file and the inner ArrayList contains the items in the record separated on a separator regular expression. The header lines can be ignored and you can specify to skip lines starting with a specific string. If you want to be able to skip multiple strings (in order to make a selection in the file of lines to process), you can change the type of the last parameter of the parse function to ArrayList and alter the code to loop over that.
Create the following Java class;
package nl.smeets.myfilereader;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
public class parsefile {
public parsefile() {
super();
}
public ArrayList<ArrayList<String>> parse(byte[] bytes_in,
String separator,
Integer ignorefirstlines,
String ignoreStartsWith) {
ArrayList<ArrayList<String>> l_retval =
new ArrayList<ArrayList<String>>();
ByteArrayInputStream ba_in = new ByteArrayInputStream(bytes_in);
BufferedReader br_in =
new BufferedReader(new InputStreamReader(ba_in));
Integer linecounter = 0;
String line = null;
try {
while ((line = br_in.readLine()) != null) {
if (!line.startsWith(ignoreStartsWith) &&
linecounter >= ignorefirstlines) {
l_retval.add(parseLine(line, separator));
}
linecounter++;
}
} catch (FileNotFoundException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
} finally {
//Close the BufferedReader
try {
if (br_in != null)
br_in.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
return l_retval;
}
ArrayList<String> parseLine(String input, String separator) {
ArrayList<String> retval = new ArrayList<String>();
String[] splitVals = input.split(separator);
for (String addVal : splitVals) {
retval.add(addVal);
}
return retval;
}
public static void main(String[] args) {
//test; skips first line and all lines starting with Total. uses ; as separator
//parsefile parsefile = new parsefile();
//System.out.println(parsefile.parse("h1;h2;h3;h4\ne1;f1;g1;h1\ne2;f2;g2;h2\nTotal: 1".getBytes(), ";", 1, "Total"));
}
}
Extract interface
Right-click the Java file, Refactor, Extract interface
Spring bean
Use the wizard to create a Spring bean and define it as follows;
<?xml version="1.0" encoding="windows-1252" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:jee="http://www.springframework.org/schema/jee"
xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:sca="http://xmlns.oracle.com/weblogic/weblogic-sca"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.5.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.5.xsd http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.5.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd http://www.springframework.org/schema/tool http://www.springframework.org/schema/tool/spring-tool-2.5.xsd http://xmlns.oracle.com/weblogic/weblogic-sca META-INF/weblogic-sca.xsd">
<bean name="ParseFileBean" class="nl.smeets.myfilereader.parsefile"/>
<sca:service name="ParseFileBeanService" target="ParseFileBean" type="nl.smeets.myfilereader.Iparsefile"/>
</beans>
First make sure the Java interface and class are compiled to avoid errors such as the one below;
Now you can use the composite editor to expose the Spring bean or use it in a BPEL process.
BPEL
Use BPEL to bring it all together. In BPEL you can assign the parameters of the parsing Java method. In this example I've hardcoded them in the process but of course you can also use BPEL preferences (described in a previous post; http://javaoraclesoa.blogspot.com/2012/02/changing-properties-of-bpel-process-at.html) which can be defined in the composite.xml file and modified at runtime so if the format of the file changes, you don't have to redeploy the BPEL process. Don't forget to update the configuration plan supplied to reflect your local environment. *.txt files from (logical path) READ_FILE_DIR are read and xml files are written as output to logical path WRITE_FILE_DIR.
Input;
h1;h2;h3;h4
A;B;C;D
E;F;G;H
Total: 2
Output;
<?xml version="1.0" encoding="UTF-8" ?>
<parseResponse xmlns="http://myfilereader.smeets.nl/">
<return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
<item>A</item>
<item>B</item>
<item>C</item>
<item>D</item>
</return>
<return xmlns:ns0="http://myfilereader.smeets.nl/types" xmlns="">
<item>E</item>
<item>F</item>
<item>G</item>
<item>H</item>
</return>
</parseResponse>
You can download the entire example here; https://www.dropbox.com/s/ytgwjz7zjlvscmj/DemoReadFile.zip?dl=0