Showing posts with label nipyapi. Show all posts
Showing posts with label nipyapi. Show all posts

Wednesday, May 18, 2022

Apache NiFi: Importing and exporting parameters

When you import a new process group or upgrade an existing one, missing parameters contexts and parameters will automatically be added. The new parameters will be filled with values from the environment where the process group was committed to the Registry (except sensitive parameter values). This is usually a development environment. NiFi 1.15 adds parameter context inheritance. If however you are on a lower version and have many similar process groups, you can have many copies of parameter contexts. If you add a large number of flows and need to add a large number of parameter contexts, it can be bothersome to have to manually update all the parameter contexts used by the new process groups. 

In most cases when you use deployment tooling the environment configuration is kept separately from the application and applied upon deployment. This is also one of the principles of the 12 factor application (read here). For example in Azure DevOps you can use variable groups, in XL Deploy environments, a Spring Boot applications usually uses property files and K8s resources can use Kustomize templates. In this blog post I created a script which allows you to use a similar method. You can export parameters from an environment (except for sensitive parameter values). This results in a CSV file. This CSV file can be imported into a different environment. This also allows you to keep a CSV parameter file per environment which can be applied on deployment. Updating parameters in a CSV is a lot easier than doing the same manually and you can easily check if everything is correct.

You can check out my code here. Specifically look at the export_parameters and import_parameters methods.

Friday, April 15, 2022

Apache NiFi: Automating tasks using NiPyAPI

Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps make them more reproducible. It allows you to incorporate tasks in for example a CI/CD system without requiring human intervention. A NiFi feature to help you automate tasks is its powerful API. In order to more easily use this API from Python, NiPyAPI is available. In this blog post I'll describe some things you can do with NiPyAPI, some challenges I encountered and how I fixed them. You can find my sample code here.