Apache NiFi: Filter events and only let through the latest in a timeframe

In the IoT world, some devices generate large volumes of events that can be difficult for back-end systems to process in real time. Of course you can use NiFi to throttle messages. However, this will not be sufficient if the flow of events is consistently higher than what can be handled by the back-end system. A way to deal with this is to let Apache NiFi group and filter messages based on a specific attribute and only letting through the latest message for a specific device, in a certain timeframe. In this blog post I'll illustrate how you can do this. The trick is to merge several messages together using the MergeContent processor and then select the latest one using a Jolt transformation.

Having fun with Jolt transformations

Jolt is a Java library which can be used to transform JSON to JSON (here). A Jolt transformation itself is also a JSON file. You can use it in products such as Apache NiFi and Apache Camel. In this blog post I'll describe my first experiences with Jolt transformations. 

For me personally Jolt transformations are not intuitive and not that powerful when for example compared to the capabilities of XSLT for transforming XML. It is available in Apache NiFi though and can be used without the 'execute code' permission, that is why I decided to use it to transform JSON files. I might misunderstand basic Jolt concepts which can cause suggested solutions to be overly complex or in other ways suboptimal.