When using AI models like Stable Diffusion, sometimes input images need to be of a specific size. In case of Stable Diffusion, multiples of 64 are required. Stable Diffusion (at least 1.5) works best with images of 512 pixels in width or height. If you for example take an image of 599 x 205 pixels and you resize it to 1496 x 512 (maintaining the aspect ratio), you end up with 1496, which is not a multiple of 64. In order to obtain a usable image, it needs to be padded to a size of 1536 x 512 to allow processing. In order to batch resize and pad images I created a Python script using OpenCV. Why create a script? Doing this for a large amount of images quickly becomes a chore and writing a script for this is fun. Why OpenCV? It is powerful, easy to use and popular.
Articles containing tips, tricks and nice to knows related to IT stuff I find interesting. Also serves as online memory.
Thursday, December 29, 2022
Friday, April 15, 2022
Apache NiFi: Automating tasks using NiPyAPI
Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps make them more reproducible. It allows you to incorporate tasks in for example a CI/CD system without requiring human intervention. A NiFi feature to help you automate tasks is its powerful API. In order to more easily use this API from Python, NiPyAPI is available. In this blog post I'll describe some things you can do with NiPyAPI, some challenges I encountered and how I fixed them. You can find my sample code here.
Sunday, January 24, 2021
Python: A Google Translate service using Playwright
There are a lot of use-cases in which you might want to automate a web-browser. For example to automate tedious repetitive tasks or to perform automated tests of front-end applications. There are also several tools available to do this such as Selenium, Cypress and Puppeteer. Several blog posts and presentations by Lucas Jellema picked my interest in Playwright so I decided to give it a try. I'm not a great fan of JavaScript so I decided to go with Python for this one. I also did some tests with wrk, a simple yet powerful HTTP bench-marking tool, to get an indication about how Playwright would handle concurrency and was not disappointed.
Saturday, February 1, 2020
HTTP benchmarking using wrk. Parsing output to CSV or JSON using Python
Parsing the wrk output is a challenge. It would be nice to have a feature to output the results in the same units as CSV or JSON file. More people asked this question and the answer was: do some LuaJIT scripting to achieve that. Since I'm no Lua expert and to be honest, I don't have any people in my vicinity that are, I decided to parse the output using Python (my favorite language for data processing and visualization) and provide you with the code so you don't have to repeat this exercise.
You can see example Python code of this here.
Thursday, June 6, 2019
Graceful shutdown of forked workers in Python and JavaScript running in Docker containers
This seems relatively straightforward, but there are some challenges. Also I implemented this in JavaScript running on Node and in Python. Python and JavaScript handle forking differently.
Saturday, March 16, 2019
Using Python to performancetest an Oracle DB
There are various ways to look at performance. You can look at user experience, generate load similar to what application usage produces or you can do more basic things like query performance. What will you be looking at? Resource consumption and throughput are the usual suspects.
I'll look at a simple example in this blog post. I'll change database parameters and look at throughput of various actions which are regularly performed on databases. This takes away the complexity of distributed systems. I used a single Python script for this which can be downloaded here.
Friday, January 26, 2018
Automate calls to SOAP and REST webservices using simple Python scripts
Monday, December 19, 2016
WebLogic Server: Automate obtaining performance metrics from DMS
Obtaining data from DMS in an automated fashion, even with the WLST support, can be a challenge. In this blog I provide a Python 2.7 script which allows you to get information from the DMS and dump it in a CSV file for further processing. The script first logs and uses the obtained session information to download information from a specific table in XML. This XML is converted to CSV. The code does not require an Oracle Home (it is not WLST based). The purpose here is to provide an easy to use starting point which can be expanded to suit specific use-cases. The script works against WebLogic 11g and 12c environments (has been tested against 11.1.1.7 and 12.2.1). Do mind that the example URL given in the script obtains performance data on webservice operations. This works great on composites but not on Service Bus or JAX-WS services. You can download a general script here (which requires minimal changes to use) and a (more specific) script with examples of how to preprocess data in the script here.
Friday, October 28, 2016
Sonatype Nexus 2.x: Using the REST API to clean-up your repository
Monday, January 4, 2016
Simple IoT security system using Raspberry Pi 2B + Razberry + Fibaro Motion Sensor (FGMS-001)
My purpose was to create a simple security system based on a Raspberry Pi. A Raspberry Pi is a tiny computer which can run a Debian like Linux distribution called Rasbian. I wanted to avoid going to low-level into sensor configuration and programming. That's why I decided early on to use an extension board and not directly attach the sensors to the Raspberry Pi. I decided to go for the Razberry. I also looked at the GrovePi and Arduino. Both are still too low-level for my tastes though. The Razberry is an extension board for the Raspberry which provides a Z-Wave controller chip. Z-Wave is a wireless protocol popular in the area of home automation. This was an attractive option since if in the future I would want to use additional sensors or maybe even use a commercial home automation system, I could very well get compatibility out of the box. For the sensor, I decided on the Fibaro FGMS-001 Motion Sensor. This is a multi-sensor which allows detection of motion, temperature, luminiscence and vibrations. It can even detect tampering and earthquakes (which is relevant since I live in the Dutch city of Groningen).
Z-Wave.Me (the company providing the Razberry), provides software for the Razberry called Z-Way. There are several alternatives. One of the most popular seems to be Domoticz which is provided with OpenZWave. Domoticz allows quite extensive home automation but I was having difficulty getting the sensor to work with OpenZWave so I decided to go with Z-Way. Z-Way supported the sensor out of the box. With the Z-Way server however it was difficult to automate actions based on sensor values. How I solved this is also described in this article.
Tuesday, September 29, 2015
How to use WLST as a Jython 2.7 module
To just name some nice Jython 2.7 features:
- pip and easy_install can be used to easily add new modules
- useful new API’s are available such as xml.etree.ElementTree to allow XML processing, the multiprocessing module to use multiple threads and the argparse module to make parsing of script arguments easy.
In this article I’ll describe how you can use WLST as a Jython 2.7 module in order to allow you to combine the best of both worlds in your scripts.
Saturday, September 12, 2015
Create a release of artifacts. Automate adding Maven dependencies
Software artefacts are developed using a build pipeline. This pipeline consists of several steps to provide quick feedback on software quality by means of code quality checks, automated tests, test coverage checks, etc. When the software is done (adhering to a Definition of Done (DoD)), it is released. This release as a whole is then tested and promoted until it reaches a production environment. In the meantime, work on a next release has already started. This process is shown in the below image. This is a minimal example (especially on the test part). In the below image, you can see there are 3 releases in progress. This is only in order to illustrate the process. You should of course try to limit the number of releases which are not in production to reduce the overhead of fixes on those releases.
Wednesday, May 1, 2013
Cleaning up unused namespaces in Oracle SOA 11g BPEL processes by using a Python script
I took this issue as a nice opportunity to learn myself a bit of Python. Python is a popular scripting language (http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html) and used by several software vendors such as Oracle for Weblogic server; WLST (http://docs.oracle.com/cd/E14571_01/web.1111/e13813/quick_ref.htm) and ESRI (http://esripress.esri.com/display/index.cfm?fuseaction=display&websiteID=224&moduleID=0) for GIS related programming. There is an official Python tutorial available on http://docs.python.org/3.3/tutorial/ and I've also used http://www.vogella.com/articles/Python/article.html to learn some basics.
First impression of the Python language
- I like the usage of indentation compared to the use of brackets or end statements
- code completion is far from perfect with the PyDev plug-in for Eclipse when using Python 3.3 (I needed to Google a lot for API documentation)
- even without background in Python, you can quickly get something working after reading some tutorials (although I have some experiences with other scripting languages like PERL, PHP, JavaScript. I usually use PERL for my regular scripting needs).
Implementation
Purpose
I wanted to create a script which would cleanup unused namespaces in XML files. I started with a BPEL file as an example (the same example as used in; http://javaoraclesoa.blogspot.nl/2013/04/soa-suite-ps6-11117-service-loose.html. It can be downloaded here; https://dl.dropboxusercontent.com/u/6693935/blog/HelloWorldTokens.zip). The BPEL file had the following contents;
<?xml version = "1.0" encoding = "UTF-8" ?>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Oracle JDeveloper BPEL Designer
Created: Thu Apr 11 12:23:10 CEST 2013
Author: Maarten
Type: BPEL 2.0 Process
Purpose: Synchronous BPEL Process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<process name="CallHelloWorld"
targetNamespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld"
xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable"
xmlns:client="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld"
xmlns:ora="http://schemas.oracle.com/xpath/extension"
xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
xmlns:bpel="http://docs.oasis-open.org/wsbpel/2.0/process/executable"
xmlns:ns1="http://xmlns.oracle.com/HelloWorld/HelloWorld/HelloWorld">
<import namespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld" location="CallHelloWorld.wsdl" importType="http://schemas.xmlsoap.org/wsdl/"/>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
PARTNERLINKS
List of services participating in this BPEL process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<partnerLinks>
<!--
The 'client' role represents the requester of this service. It is
used for callback. The location and correlation information associated
with the client role are automatically set using WS-Addressing.
-->
<partnerLink name="callhelloworld_client" partnerLinkType="client:CallHelloWorld" myRole="CallHelloWorldProvider"/>
<partnerLink name="HelloWorld" partnerLinkType="ns1:HelloWorld"
partnerRole="HelloWorldProvider"/>
</partnerLinks>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
VARIABLES
List of messages and XML documents used within this BPEL process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<variables>
<!-- Reference to the message passed as input during initiation -->
<variable name="inputVariable" messageType="client:CallHelloWorldRequestMessage"/>
<!-- Reference to the message that will be returned to the requester-->
<variable name="outputVariable" messageType="client:CallHelloWorldResponseMessage"/>
<variable name="InvokeHelloWorld_process_InputVariable"
messageType="ns1:HelloWorldRequestMessage"/>
<variable name="InvokeHelloWorld_process_OutputVariable"
messageType="ns1:HelloWorldResponseMessage"/>
</variables>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
ORCHESTRATION LOGIC
Set of activities coordinating the flow of messages across the
services integrated within this business process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<sequence name="main">
<!-- Receive input from requestor. (Note: This maps to operation defined in CallHelloWorld.wsdl) -->
<receive name="receiveInput" partnerLink="callhelloworld_client" portType="client:CallHelloWorld" operation="process" variable="inputVariable" createInstance="yes"/>
<assign name="Assign1">
<copy>
<from>$inputVariable.payload/client:input</from>
<to>$InvokeHelloWorld_process_InputVariable.payload/ns1:input</to>
</copy>
</assign>
<invoke name="InvokeHelloWorld"
partnerLink="HelloWorld" portType="ns1:HelloWorld"
operation="process"
inputVariable="InvokeHelloWorld_process_InputVariable"
outputVariable="InvokeHelloWorld_process_OutputVariable"
bpelx:invokeAsDetail="no"/>
<assign name="Assign2">
<copy>
<from>$InvokeHelloWorld_process_OutputVariable.payload/ns1:result</from>
<to>$outputVariable.payload/client:result</to>
</copy>
</assign>
<!-- Generate reply to synchronous request -->
<reply name="replyOutput" partnerLink="callhelloworld_client" portType="client:CallHelloWorld" operation="process" variable="outputVariable"/>
</sequence>
</process>
The ora namespace was not used in this process but it is specified in the process tag.
lxml
Based on the following I started with lxml; http://lxml.de/api/lxml.etree-module.html#cleanup_namespaces since it had a function to easily clean unused namespaces and on StackOverflow.com there were many posts on lxml. To install lxml for Windows, I had to first download and install it from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml
cleanup_namespaces
My first try was the following;
import lxml.etree as et
filename_in='D:\\dev\\HelloWorld\\CallHelloWorld\\CallHelloWorld.bpel'
filename_out='D:\\dev\\HelloWorld\\CallHelloWorld\\CallHelloWorld.bpel.out'
tree = et.parse(filename_in)
et.cleanup_namespaces(tree)
tree.write(filename_out)
In the output I noticed that although my process definition had been cleaned up from
<process name="CallHelloWorld"
targetNamespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld"
xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable"
xmlns:client="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld"
xmlns:ora="http://schemas.oracle.com/xpath/extension"
xmlns:bpelx="http://schemas.oracle.com/bpel/extension"
xmlns:bpel="http://docs.oasis-open.org/wsbpel/2.0/process/executable"
xmlns:ns1="http://xmlns.oracle.com/HelloWorld/HelloWorld/HelloWorld">
To
<process xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable" xmlns:bpelx="http://schemas.oracle.com/bpel/extension" name="CallHelloWorld" targetNamespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld">
Several namespaces were removed which were in use such as the ns1 namespace which was used in a partnerlink definition as part of an attribute value;
<partnerLink name="HelloWorld" partnerLinkType="ns1:HelloWorld" partnerRole="HelloWorldProvider"/>
My conclusion was that using prebuild functions like the above would most likely not help solve this problem. I did not find a way to limit the functionality to specific namespaces.
Determining used namespaces
I tried a different approach; determine namespaces used in the root element and try to find them on different locations in the BPEL file. Then rewriting the root element.
import lxml.etree as et
import copy
filename_in='D:\\dev\\HelloWorld\\CallHelloWorld\\CallHelloWorld.bpel'
filename_out='D:\\dev\\HelloWorld\\CallHelloWorld\\CallHelloWorld.bpel.out'
tree = et.parse(filename_in)
root=tree.getroot()
nsmap=root.nsmap
nsmapnew= copy.deepcopy(nsmap)
#print (nsmap.values())
namespaces=set(nsmap.values())
print ("Namespaces found: "+str(len(nsmap)))
for nsitem in nsmap:
found=0
nscount=0;
if nsitem != None:
print ("Processing prefix: "+nsitem+" Namespace: "+nsmap.get(nsitem))
#processing all elements
walkAll = tree.getiterator()
for elt in walkAll:
#check element
eltns=elt.xpath('namespace-uri(/*)')
if eltns==nsmap.get(nsitem):
found=1
#print("Found namespace as element namespace")
break
if str(elt.text).find(nsitem+":") != -1:
found=1
#print("Found prefix as part of element text")
break
#check attributes
for attribute in elt.attrib:
if attribute.startswith("{"+nsmap.get(nsitem)+"}"):
found=1
#print ("Found namespace as attribute name namespace")
break
if str(elt.attrib[attribute]).find(nsitem+":") != -1:
found=1
#print ("Found prefix as part of attribute value")
break
else:
#default namespace not removing
found=1
if found==0:
print("Not found")
del nsmapnew[nsitem]
print ("Namespaces remaining: "+str(len(nsmapnew)))
new_root = et.Element(root.tag, attrib=root.attrib,nsmap=nsmapnew)
new_root[:] = root[:]
#to add the top level comment
try:
firstcomment=root.getprevious()
new_root.addprevious(firstcomment)
except:
None
tree = et.ElementTree(new_root)
tree.write(filename_out, xml_declaration=True, encoding='utf-8',pretty_print=True)
This rewrote my XML like I wanted it to. Even if namespaces were used in XPATH expressions in the BPEL file, they were being recognized as being used. One drawback however was the namespace prefix which was added for all the subelements even though these elements were in the default namespace. I considered using a transformation to fix this. This would however remove the comments from the file so I decided not to. The script output was as followed;
Namespaces found: 6
Processing prefix: ns1 Namespace: http://xmlns.oracle.com/HelloWorld/HelloWorld/HelloWorld
Processing prefix: ora Namespace: http://schemas.oracle.com/xpath/extension
Not found
Processing prefix: client Namespace: http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld
Processing prefix: bpel Namespace: http://docs.oasis-open.org/wsbpel/2.0/process/executable
Processing prefix: bpelx Namespace: http://schemas.oracle.com/bpel/extension
Namespaces remaining: 5
The created output file was as followed;
<?xml version='1.0' encoding='UTF-8'?>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Oracle JDeveloper BPEL Designer
Created: Thu Apr 11 12:23:10 CEST 2013
Author: Maarten
Type: BPEL 2.0 Process
Purpose: Synchronous BPEL Process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<bpel:process xmlns:ns1="http://xmlns.oracle.com/HelloWorld/HelloWorld/HelloWorld" xmlns:client="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld" xmlns:bpel="http://docs.oasis-open.org/wsbpel/2.0/process/executable" xmlns:bpelx="http://schemas.oracle.com/bpel/extension" xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable" name="CallHelloWorld" targetNamespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld"><bpel:import namespace="http://xmlns.oracle.com/HelloWorld/CallHelloWorld/CallHelloWorld" location="CallHelloWorld.wsdl" importType="http://schemas.xmlsoap.org/wsdl/"/>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
PARTNERLINKS
List of services participating in this BPEL process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<bpel:partnerLinks>
<!--
The 'client' role represents the requester of this service. It is
used for callback. The location and correlation information associated
with the client role are automatically set using WS-Addressing.
-->
<bpel:partnerLink name="callhelloworld_client" partnerLinkType="client:CallHelloWorld" myRole="CallHelloWorldProvider"/>
<bpel:partnerLink name="HelloWorld" partnerLinkType="ns1:HelloWorld" partnerRole="HelloWorldProvider"/>
</bpel:partnerLinks>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
VARIABLES
List of messages and XML documents used within this BPEL process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<bpel:variables>
<!-- Reference to the message passed as input during initiation -->
<bpel:variable name="inputVariable" messageType="client:CallHelloWorldRequestMessage"/>
<!-- Reference to the message that will be returned to the requester-->
<bpel:variable name="outputVariable" messageType="client:CallHelloWorldResponseMessage"/>
<bpel:variable name="InvokeHelloWorld_process_InputVariable" messageType="ns1:HelloWorldRequestMessage"/>
<bpel:variable name="InvokeHelloWorld_process_OutputVariable" messageType="ns1:HelloWorldResponseMessage"/>
</bpel:variables>
<!--
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
ORCHESTRATION LOGIC
Set of activities coordinating the flow of messages across the
services integrated within this business process
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-->
<bpel:sequence name="main">
<!-- Receive input from requestor. (Note: This maps to operation defined in CallHelloWorld.wsdl) -->
<bpel:receive name="receiveInput" partnerLink="callhelloworld_client" portType="client:CallHelloWorld" operation="process" variable="inputVariable" createInstance="yes"/>
<bpel:assign name="Assign1">
<bpel:copy>
<bpel:from>$inputVariable.payload/client:input</bpel:from>
<bpel:to>$InvokeHelloWorld_process_InputVariable.payload/ns1:input</bpel:to>
</bpel:copy>
</bpel:assign>
<bpel:invoke name="InvokeHelloWorld" partnerLink="HelloWorld" portType="ns1:HelloWorld" operation="process" inputVariable="InvokeHelloWorld_process_InputVariable" outputVariable="InvokeHelloWorld_process_OutputVariable" bpelx:invokeAsDetail="no"/>
<bpel:assign name="Assign2">
<bpel:copy>
<bpel:from>$InvokeHelloWorld_process_OutputVariable.payload/ns1:result</bpel:from>
<bpel:to>$outputVariable.payload/client:result</bpel:to>
</bpel:copy>
</bpel:assign>
<!-- Generate reply to synchronous request -->
<bpel:reply name="replyOutput" partnerLink="callhelloworld_client" portType="client:CallHelloWorld" operation="process" variable="outputVariable"/>
</bpel:sequence>
</bpel:process>
The ora namespace had been removed from the process attribute. The file had also become smaller. The ora namespace is however a default Oracle namespace and to make sure I didn't break anything, I tried to compile and deploy the altered process. This was successful. I also tested it with more complex processes. The resulting BPEL file was still fully functional.
Possible followups could be;
- expand the script and allow recursively processing of multiple files and filetypes
- determine used and unused namespaces for every element, not just the root element
- link the results of unused namespaces within a project to XSD's which could also be removed from a project if they are not used by any file in the project
- if for example problems start to occur with XPATH expressions after removing default Oracle namespaces; exclude specific namespaces from cleaning
- remove the namespace prefix for the default namespace










