Vagrant + Docker Compose: A quick and easy Apache NiFi development environment

Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared resources and of course NiFi itself. Setting this up yourself can be cumbersome. That's why I've created some scripts to help you do this quickly. You can find them here. 

Since manually scripting the installations of all these products can be a challenge / work, I decided to use Docker images which often already provide an automatic installation (so I don't have to do that myself) and used Docker Compose to easily allow the containers to find each other and have a docker-compose.yml which contained my environment variables so I wouldn't have to supply them on the commandline.

Setup

Vagrant

The Vagrantfile I've used, first installs plugins to update the base image disk size and automatically install the VirtualBox guest additions. It uses Ubuntu 20.04 as the base image (currently the most recent LTS version). The Vagrantfile ends with calling a provision.sh script. This script does the rest of the provisioning. 

The provision.sh script does the following (in 50 lines):

• It installs packages required to offer basic GUI functionality in Ubuntu
• It installs tools like Firefox, Docker and Docker Compose, Offset Explorer (previously Kafkatool). 
• It adds a developer with sufficient privileges to work efficiently.
• Copies the required files to build a Docker image for Kafka
• Pulls the required images for Apache NiFi and NiFi Registry

Docker Compose

I've used the scripts from here as base for my docker-compose.yml file and expanded it with NiFi and NiFi registry. If of course you want a more complete Kafka set-up, you can also base your docker-compose.yml on what the Confluent Platform provides (here). This also contains for example the REST Proxy and Schema Registry (for JSON Schema, AVRO and Protobuf schemas).

Try it out

Preparations

You need an installation of VirtualBox and Vagrant for this.You can easily download and install them from their respective websites or use a package manager (for example Chocolatey on Windows) if you want to be able to easily keep them up to date.

Clone and build

Next clone the GitHub repository (here). Go to the provisioning/ubuntu-dev folder inside the repository and execute:

vagrant up

Now Vagrant will create your VirtualBox VM for you and make things ready. After it is done, the VirtualBox VM 'Ubuntu NiFi Dev' will be shutdown again so you can start it and use it.

Use

You can login using the user developer with password Welcome01.

After you have logged in, you can open up a terminal and go to the /home/developer/comp directory. In that directory you can do 

docker-compose up

This will first build the Kafka image, start Kafka, start Zookeeper, start NiFi and start the NiFi Registry.

If you want to know the status, you can do:

docker-compose ps

After it is complete, you can open Firefox and go to https://localhost:8443/nifi and login using user admin and password Welcome010101 (it was required to be at least 12 characters long). Ignore the certificate errors (the certificate used is self-signed) and have fun!

If you want to remove everything, for example to start clean again after you have messed up, you can do:

docker-compose down