Thursday, June 6, 2019

Graceful shutdown of forked workers in Python and JavaScript running in Docker containers

You might encounter a situation where you want to fork a script during execution. For example if the amount of forks is dependent on user input or another specific situation. I encountered such a situation in which I wanted to put load on a service using multiple concurrent processes. In addition, when running in a docker container, only the process with PID=1 receives a SIGTERM signal. If it has terminated, the worker processes receive a SIGKILL signal and are not allowed a graceful shutdown. In order to do a graceful shutdown of the worker processes, the main process needs to manage them and only exit after the worker processes have terminated gracefully. Why do you want processes to be terminated gracefully? In my case because I store performance data in memory (disk is too slow) and only write the data to disk when the test has completed.

This seems relatively straightforward, but there are some challenges. Also I implemented this in JavaScript running on Node and in Python. Python and JavaScript handle forking differently.

Saturday, June 1, 2019

Performance! 3 reasons to stick to Java 8 for the moment

It is a smart thing to move to newer versions of Java! Support such as security updates and new features are just two of them but there are many more. Performance might be a reason to stick to Java 8 though. In this blog post I'll show some results of performance tests I have conducted showing Java 11 has slower startup times and slightly slower throughput compared to Java 8 when using the same Java code. Native images (a GraalVM feature) have greatly reduced startup time and memory usage at the cost of throughput. You can only compile Java 8 byte-code to a native image though (at the moment).

Saturday, March 23, 2019

6 tips to make your life with Vagrant even better!

HashiCorp Vagrant is a great tool to quickly get up and running with a development environment. In this blog post I'll give some tips to make your life with Vagrant even better! You can find an example which uses these tips here.

Saturday, March 16, 2019

Using Python to performancetest an Oracle DB

Performance testing is a topic with many opinions and complexities. You can not do it in a way which will make everyone happy. It is not straightforward to compare measures before and after a change. Environments are often not stable (without change in itself and its environment). When performing a test, the situation at the start or end of the test are also often not the same. For example the test might write data in a database.

There are various ways to look at performance. You can look at user experience, generate load similar to what application usage produces or you can do more basic things like query performance. What will you be looking at? Resource consumption and throughput are the usual suspects.

I'll look at a simple example in this blog post. I'll change database parameters and look at throughput of various actions which are regularly performed on databases. This takes away the complexity of distributed systems. I used a single Python script for this which can be downloaded here.

Summary of conclusions: Exposing database functionality using a DAD is not so much influenced by the tested settings. Setting FILESYSTEMIO_OPTIONS to SETALL improved the performance of almost all database actions. This has also been observed at different customers. Disabling Transparent HugePages and enabling the database to use HugePages seemed to have little effect. PL/SQL native compilation also did not cause a massive improvement. From the tested settings FILESYSTEMIO_OPTIONS is the easiest to apply. Query performance and actions involving a lot of data improved with all (and any of) these settings.

Tuesday, February 26, 2019

Filesystem events to Elasticsearch / Kibana through Kafka Connect / Kafka

Filesystem events are useful to monitor. They can indicate a security breach. They can also help  understanding how a complex system works by looking at the files it reads and writes.

When monitoring events, you can expect a lot of data to be generated quickly. The events might be interesting to process for different systems and at a different pace. Also it would be nice if you could replay events from the start or a specific moment. Enter Kafka. In order to put the filesystem events in Kafka (from an output file), the Kafka Connect FileSourceConnector is used. In order to get the data from Kafka to Elasticsearch, the Kafka Connect ElasticsearchSinkConnector is used. Both connectors can be used without Enterprise license.

Saturday, February 16, 2019

Minikube on KVM on Linux Mint 19.1

In a previous blog post I wrote about running Minikube on Windows. I ended with the suggestion that getting Minikube working might be much easier on Linux. Thus I installed Linux Mint (as dual-boot) on my laptop and gave it a shot. The steps I took to get it working are described here.

Friday, February 15, 2019

Some challenges with Oracle Reports

Oracle Reports has been around for a long time and future versions will most likely not be created  (see here). Hence this is going to be my first and also last blog post on this product. Installing Reports is not an easy task. It requires several steps which are not well documented. This blog post contains a few pointers. The main source of inspiration is here.