Apache NiFi: Monitoring metrics and provenance events using Azure Log Analytics

There are several cases where you might want to use Azure Log Analytics to monitor your NiFi instances. An obvious one is when NiFi is running in Azure. Azure Log Analytics can also be used as single monitoring/alerting solution for multiple applications making operations easier by providing a single interface for this. You might want this if you want to monitor business processes which span multiple applications and you want to monitor the entire process to for example identify bottlenecks.

In this blog post I'll show you how easy it is to achieve this using the AzureLogAnalyticsReportingTask and the AzureLogAnalyticsProvenanceReportingTask from NiFi and what you need to configure in Azure Log Analytics to make this work.

Java Agent: Rewrite Java code at runtime using Javassist

There are situations where you want to change Java code not at the source code level but at runtime. For example when you want to instrument code for logging purposes but do not want to change the source code (because you might not have access to it). This can be done by using a Java Agent. For example, Dynatrace uses a Java agent to collect data from inside the JVM. Another example is the GraalVM tracing agent (here) which helps you create configuration for the generation of native images. Logging is one use-case but you can also more dramatically alter runtime code to obtain a completely different runtime behavior.

This blog post is not a step by step introduction for creating Java Agents. For that please take a look at the following. In this blog post I have created a Java agent which rewrites synchronized methods to use a ReentrantLock instead (see here). The use-case for this is to allow applications to use Project Loom's Virtual Threads more efficiently.

You can find the code of the agent here.