Python: Batch resize and pad files

When using AI models like Stable Diffusion, sometimes input images need to be of a specific size. In case of Stable Diffusion, multiples of 64 are required. Stable Diffusion (at least 1.5) works best with images of 512 pixels in width or height. If you for example take an image of 599 x 205 pixels and you resize it to 1496 x 512 (maintaining the aspect ratio), you end up with 1496, which is not a multiple of 64. In order to obtain a usable image, it needs to be padded to a size of 1536 x 512 to allow processing. In order to batch resize and pad images I created a Python script using OpenCV. Why create a script? Doing this for a large amount of images quickly becomes a chore and writing a script for this is fun. Why OpenCV? It is powerful, easy to use and popular.

Apache NiFi: Filter events and only let through the latest in a timeframe

In the IoT world, some devices generate large volumes of events that can be difficult for back-end systems to process in real time. Of course you can use NiFi to throttle messages. However, this will not be sufficient if the flow of events is consistently higher than what can be handled by the back-end system. A way to deal with this is to let Apache NiFi group and filter messages based on a specific attribute and only letting through the latest message for a specific device, in a certain timeframe. In this blog post I'll illustrate how you can do this. The trick is to merge several messages together using the MergeContent processor and then select the latest one using a Jolt transformation.

Reduce your gas and electricity bill!

Gas and electricity costs have increased significantly in the Netherlands and perhaps throughout Europe as a result of the current difficulties on the energy market. In this blog post, I'll make some recommendations for how to lower your gas and electricity costs. Some of these are specific to the Netherlands.

My goal is to use gas and energy as economically as possible. Consequently, the goal is to lower the gas and/or electricity costs. You may also be motivated by other factors, such as a desire to stop using Russian gas or a desire to lower CO₂ emissions. These motives could lead to different choices. Everyone lives in a different house and there are differences in the laws/regulations of the various countries, which can, of course, also affect the decisions you make.

N.b. I'm no gas/energy expert but the below measures are things which I have looked at and which seemed to help. 

General suggestions

Reduce gas

• Insulate your house
  This is an obvious one and, especially in older houses, you can save a lot of money by this. You can do large operations like insulating your walls, roof, loft or floor but you can also relatively easily also do small things like 
• use weatherstrips
• insulate pipes. Especially if they transport hot water.
• use thick curtains or thermo curtains to keep heat inside
• making sure your mailbox when it is build inside your front door does not cause a draft. You could reduce this with a product like this.
• install radiator foil when you do not have floor heating for the radiators which are against walls which are connected to the outside. Radiator foil is cheap and easy to install. See for example here. It works best if you can directly attach it to your radiator.
• close doors when you are not inside the room. Also remember heat goes up. The rooms at the ground floor, especially if they have high ceilings, are more difficult to keep warm. If you prevent air movement by closing doors, the heat stays in the same place longer.
• Reduce your room temperature
  You can save a lot of gas by reducing your room temperature. This probably requires you to wear more clothes. The tip here is to use thermo underwear and layers. In your bed woollen blankets can help keep you warm at night. Often also you do not need to have all rooms at a high temperature so consider only heating the rooms you use often.
• Tweak your central heating system
  A central heating boiler (CV in Dutch) produces hot water for the tap and hot water for the radiators. The following (Dutch) site has some suggestions on this (here). If you are looking for manuals, you can check out this site (Dutch).
• The hot water for the tap needs to be at least 60 degrees of Celsius to prevent the Legionella bacteria from causing issues. Higher is often not necessary!
• The hot water for the radiators can usually be lowered to 60 degrees of Celsius when your house is well insulated and you have reasonably new radiators. The default setting could be as high as 95 degrees of Celsius which is not needed. Mind that a lower setting means it will take longer for your house to warm up.
• Usually the central heating boiler has several settings related to heating. You can usually put in on 'comfort', 'eco' or 'off'. This setting determines how long it will take until you have hot water when you open the tap or start the shower. When the setting is on comfort, it will use more gas than when set to eco. This can be a waste if you don't mind waiting a bit longer for hot water.
• The radiators can be tweaked to optimize the water flow going through and the heat it transfers. This is called hydronic balancing (see here). Look into it! It can make your radiators more efficient and help with keeping different rooms at a similar temperature. When you let someone do this, they can at the same time do maintenance on your central heating boiler.
• A smart thermostat which turns the radiator heating lower or off when you are not there and for example at night, can also save a lot of gas. If you still have a manual thermostat, you will probably sometimes forget to turn it down when you are going to bed or when going on a holiday.
• There are radiator ventilators which help in making radiators more efficient. See for example here.
• Prevent air and rust build-up inside your central heating system. Bleef your radiators once in a while (see for example here). Air is bad for heat transfer and can cause rust inside your pipes which is also not good for heat transfer. Rust can also cause issues inside the central heating boiler, especially if it is a relatively new one (when the pump uses direct current). You can prevent rust/air from accumulating inside your central heating system by using dirt separators and air vents (see for example here). A dirt separator should be used for the water going into the central heating boiler and an air separator for the water leaving the central heating boiler. 
• Replace old central heating systems
  Old central heating systems are often less efficient than new ones. New central heating systems have several features to achieve this which old ones mostly don't have yet. Do look into whether you would benefit most from a heat pump, a hybrid central heating system (which can heat on gas and electricity) or another solution. Also consider legislation. For example in the Netherlands in the future, it will not be allowed anymore to buy a central heating system which only works on gas.
• Dry air is easier to heat
  If the air in your house is relatively dry, the air is easier to heat. Read for example here (there are many other sources). If you want to save on having to heat your house (gas or electricity), reduce humidity to below 40% in winter. This also helps reduce condensation of water on cold surfaces, which, as an additional benefit, can also prevent moulds from forming (see here). The best way to go about this is to first measure the humidity inside your house and next take measures if required. You can buy a dehumidifiers to reduce humidity and of course make sure when taking a shower or bath, that the room is well ventilated.
• Shower
  A rain shower uses a lot of water which needs to be heated. Heating requires gas or electricity. You can save gas/electricity if you use a water saving shower head instead. Personally I prefer them anyway. You can of course also save on gas/electricity if you shower shorter. Taking a bath uses more gas/water than taking a shower if the shower is shorter than around 10 minutes (see here).

Reduce electricity

• Reduce standby electricity usage
  When a device, for example a charger, is plugged in, it consumes electricity. Even when it is not in use. You can measure this by using relatively cheap devices like these. With the current electricity prices I noticed that my home workplace would use more than €150 euro of electricity a year when not in use! This can easily be avoided by using a power strip with an off button and turn it off when not in use.
• Use LEDs
  LED light bulbs are the most energy efficient light bulbs you can currently buy. Should you have 'old fashioned' light bulbs, you can save as much as 10 times the electricity for the same amount of light. Of course LED lights do not help you in heating your house. If you use smart LEDs, there will be slightly more electricity consumption, even when it is turned off since it needs to be able to respond to signals for turning it on (usually using WiFi or Bluetooth). This additional electricity usage is not that much though (see here).
• Use a hot water dispenser instead of a Quooker or water boiler
  A Quooker keeps water warm so you can have hot water when you need it. This however requires continuously keeping water warm / electricity. A regular water cooker often (depends on how you use it) heats too much water to a too high temperature. You can consider using a water dispenser (for example here. usually between 50€ and 100€). Using such a device you can configure the temperature and amount of water to use which makes more efficiënt use of electricity. Boiling water using a water boiler or a kettle has similar cost (currently).
• Replace old devices
• Old freezers, refrigerators, washing machines, dryers, televisions (especially plasma or LED) are often a lot less efficient with their energy than modern versions.
• Use laptops instead of desktop PCs. Laptops use a lot less power (they are made to last long on a battery). They usually have a lower life expectancy than desktops though and they are often more expensive for similar hardware when compared to desktops.
• If you are running a server at home, especially if it is a desktop or similar, consider using hardware for it which uses less power. For example, a server often does not need a graphics card. An SSD is a lot more energy efficient than an HDD (here). Would a Raspberry Pi or something similar not be enough for your needs?
• Use the eco programs on your dishwasher and washing machine
  The eco program, if your machine has one, usually takes longer to complete than the fast or regular programs. They are however more energy efficient. See for example here.
• Tweak your television
  Suppose you have a large OLED TV. It uses a lot of power when turned on and even when in standby (can be avoided with a power strip with off button as mentioned earlier). You can tweak your TV to reduce power usage. See for example here for some suggestions for if you own a recent LG TV.
• Use timers
  You can use timers such as these to cut power to devices you do not use structurally at certain times a day. For example why power your modem/WiFi router when you are asleep?
• Freezer
  Make sure there is no ice in your freezer. It will use less electricity without ice around the cooling units. Read here. There are stories that a filled fridge uses less electricity but there does not appear to be solid evidence for this (read here).
  

Are your electricity and gas meter accurate?

The gas and electricity meters might be inaccurate. Check for example here. If you suspect this might be the case, you can check for yourself first and if the meters  measurements are clearly inaccurate, you can contact the supplier. They can perform a more accurate test. Usually you have to pay for this test but if you are right and the measures were off, you will get a refund and will receive compensation for the estimated amount you have paid too much in the past.

Specific to the Netherlands

• Solar cells good, heat pump bad
  Choose your energy/gas conservation measures carefully
• Currently in the Netherlands solar cells have a really short return of investment. Definitely get them if you do not have them already. An air/water heat pump though has a very long return of investment if you can even save enough to make it worth the initial investment. See for example here. If you have low gas usage and no underfloor heating, buying a heat pump in the Netherlands is currently not economically interesting. An air/air heat pump, also called airco (which can also do heating) is a lot more efficient than a mobile airco. It does use electricity and it will probably be most useful to cool during summer (using the electricity from your solar cells). City heating is not necessarily cheaper or more environment friendly when compared to using your own central heating system (does it burn wood or gas? is the price of what it burns used to calculate what consumers need to pay?). A boiler and heat collectors might be interesting to look at since they can reduce gas usage but do require electricity to pump water around and require quite some space. If you totally want to get rid of using gas, your total gas/electricity bill will probably rise. If for example you want to cook using induction, even during this energy crisis, you will still have to pay more than using gas. This is the current situation. It might change in the future.
• Check out government subsidies
  You do not have to pay for everything by yourself! The government provides several subsidies to make your home more gas/energy efficient or improve your home in general. In Groningen you can take a look at what you might get for earthquake damages (here). If you have confirmed damages, you can get the SNN subsidy to increase the value of your home. If you are lucky, you can also get an additional 10K (here). If you have used the money to improve your home (for some subsidies that is a requirement), you can also use the same measures to request a more general subsidy for which you don't have to live in Groningen (see here for some suggestions).
• The price ceiling. Gas users benefit most
  A relatively new policy sets a price ceiling for electricity and gas starting at januari 2023 but gas/energy are allowed to implement it as early as november 2022. Until a certain amount of usage you pay at maximum a specific price. Above that usage, you pay the market price which is most likely higher. If you manage to stay below the price ceiling, gas and electricity will be relatively cheap. A challenge is mainly high electricity usage. For example you do not have (enough) solar cells to cover your electricity usage (e.g. charging electric cars and using electricity to warm your house). You could end up above the ceiling for electricity. It might be worthwhile to consider whether it will be cheaper heating your house on gas than on electricity if you have the choice.

Apache NiFi: Monitoring metrics and provenance events using Azure Log Analytics

There are several cases where you might want to use Azure Log Analytics to monitor your NiFi instances. An obvious one is when NiFi is running in Azure. Azure Log Analytics can also be used as single monitoring/alerting solution for multiple applications making operations easier by providing a single interface for this. You might want this if you want to monitor business processes which span multiple applications and you want to monitor the entire process to for example identify bottlenecks.

In this blog post I'll show you how easy it is to achieve this using the AzureLogAnalyticsReportingTask and the AzureLogAnalyticsProvenanceReportingTask from NiFi and what you need to configure in Azure Log Analytics to make this work.

Apache NiFi: Importing and exporting parameters

When you import a new process group or upgrade an existing one, missing parameters contexts and parameters will automatically be added. The new parameters will be filled with values from the environment where the process group was committed to the Registry (except sensitive parameter values). This is usually a development environment. NiFi 1.15 adds parameter context inheritance. If however you are on a lower version and have many similar process groups, you can have many copies of parameter contexts. If you add a large number of flows and need to add a large number of parameter contexts, it can be bothersome to have to manually update all the parameter contexts used by the new process groups. 

In most cases when you use deployment tooling the environment configuration is kept separately from the application and applied upon deployment. This is also one of the principles of the 12 factor application (read here). For example in Azure DevOps you can use variable groups, in XL Deploy environments, a Spring Boot applications usually uses property files and K8s resources can use Kustomize templates. In this blog post I created a script which allows you to use a similar method. You can export parameters from an environment (except for sensitive parameter values). This results in a CSV file. This CSV file can be imported into a different environment. This also allows you to keep a CSV parameter file per environment which can be applied on deployment. Updating parameters in a CSV is a lot easier than doing the same manually and you can easily check if everything is correct.

You can check out my code here. Specifically look at the export_parameters and import_parameters methods.

Having fun with Jolt transformations

Jolt is a Java library which can be used to transform JSON to JSON (here). A Jolt transformation itself is also a JSON file. You can use it in products such as Apache NiFi and Apache Camel. In this blog post I'll describe my first experiences with Jolt transformations. 

For me personally Jolt transformations are not intuitive and not that powerful when for example compared to the capabilities of XSLT for transforming XML. It is available in Apache NiFi though and can be used without the 'execute code' permission, that is why I decided to use it to transform JSON files. I might misunderstand basic Jolt concepts which can cause suggested solutions to be overly complex or in other ways suboptimal.

Apache NiFi: JSON to SOAP

Apache NiFi is a powerful open source integration product. A challenge you might encounter when integrating systems is that one system can produce JSON messages and the other has a SOAP API available. In this blog post I'll show how you can use NiFi to convert JSON input to a SOAP service call. This involves abstracting an AVRO schema for the JSON, converting it to XML and transforming the XML to a SOAP message. 

In this example I'm using several publicly available websites. You should of course be careful. Do not copy/paste sensitive XML or JSON on these sites!

Apache NiFi: Automating tasks using NiPyAPI

Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps make them more reproducible. It allows you to incorporate tasks in for example a CI/CD system without requiring human intervention. A NiFi feature to help you automate tasks is its powerful API. In order to more easily use this API from Python, NiPyAPI is available. In this blog post I'll describe some things you can do with NiPyAPI, some challenges I encountered and how I fixed them. You can find my sample code here.

Apache NiFi: Avoid these common pitfalls

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It has a powerful UI which can be used for both development and operations. In addition, the NiFi Registry is available to make promoting software from one environment to the next, easier. In order to use NiFi efficiently, I'd like to point out some common pitfalls when using NiFi.

Vagrant and Hyper-V: Don't do it!

I've used Vagrant since 2015 in combination with Virtualbox for creating development machines. Recently however I'm experiencing more issues with Virtualbox. For example CPUs getting stuck when assigning multiple CPUs to a VM and issues with auto adjusting the guest resolution when I resize the VM window. These annoyances drove me to try out Vagrant with Hyper-V (running an Ubuntu 21.04 guest on a Windows 11 host). In this blog post I'll describe my experiences. In summary, it did not make me happy. A lot of things which work out of the box with Vagrant and VirtualBox require effort to get working in Hyper-V. Not only that but several alternative solutions are required outside of Hyper-V because of lack of features. I think I should try VMWare next to see if it will provide a better experience. You can download my Vagrantfile and provisioning script here.

Merge AVRO schema and generate random data or Java classes

Previously I wrote about generating random data which conforms to an AVRO schema (here). In a recent use-case, I encountered the situation where there were several separate schema files containing different AVRO types. The message used types from those different files. For the generation of random data, I first needed to merge the different files into a single schema. In addition, I wanted to generate Java classes for the complete message which required importing dependent types in the pom.xml. In this blog post I'll describe how I did that.

Java: Validating JSON against an AVRO schema

AVRO schema are often used to serialize JSON data into a compact binary format in order to for example transport it efficiently over Kafka. When you want to validate your JSON against an AVRO schema in Java, you will encounter a challenge. The JSON which is required to allow validation against an AVRO schema from the Apache AVRO libraries is not standard JSON. It requires explicit typing of fields. Also when the validation fails, you will get errors like: "Expected start-union. Got VALUE_STRING" or "Expected start-union. Got VALUE_NUMBER_INT" without a specific object, line number or indication  of what is expected. Especially during development, this is insufficient.

In this blog post I'll describe a method (inspired by this) on how you can check your JSON against an AVRO schema and get usable validation results. First you generate Java classes of your AVRO schema using the Apache AVRO Maven plugin (which is configured differently than documented). Next you serialize a JSON object against these classes using libraries from the Jackson project. During serialization, you will get clear exceptions. See my sample code here.

Generating random JSON data from an AVRO schema in Java

Recently I was designing an AVRO schema and wanted to test how data would look like which conformed to this schema. I developed some Java code to generate sample data. This of course also has uses in more elaborate tests which require generation of random events. Because AVRO is not that specific, this is mainly useful to get an idea of the structure of a JSON which conforms to the definition. Here I'll describes a simple Java (17 but will also work on 11) based solution to do this.

Apache NiFi: Forwarding HTTP headers

Apache NiFi can be used to expose various flavors of webservices. Using NiFi in such a way provides benefits like quick development using a GUI and of course data provenance. You know who called you with which data and where the data went. The NiFi is very scalable, delivery can be guaranteed and NiFi can help with features like back-pressure if a backend system cannot handle requests as quickly as they are offered. Exposing webservices by using NiFi, can have additional benefits such as service virtualization (decoupling). When exposing HTTP(S) webservices, a regular requirement is to pass through HTTP headers. This blog post is about how you can do that using the NiFi processors ListenHTTP, InvokeHTTP, HandleHttpRequest and HandleHttpResponse. I've used the environment which is described here.