Apache NiFi can be used to accelerate big data projects by allowing easy integration between various data sources. Using Apache NiFi it is easy to track what happened to your data (data provenance) and to provide features like guaranteed ordered delivery and error handling. In this example I'm going to configure NiFi to read COVID data from a REST API, split the data into individual records per country and publish the result to a Kafka topic. I've used the environment described here.
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared resources and of course NiFi itself. Setting this up yourself can be cumbersome. That's why I've created some scripts to help you do this quickly. You can find them here.
Since manually scripting the installations of all these products can be a challenge / work, I decided to use Docker images which often already provide an automatic installation (so I don't have to do that myself) and used Docker Compose to easily allow the containers to find each other and have a docker-compose.yml which contained my environment variables so I wouldn't have to supply them on the commandline.
Your customer has a specific challenge and wants to have it solved by means of software. They ask you as a software/solution architect to advise them on this topic and they want to be able to choose a solution based on your advise by the end of next week (we're not talking about something as large as a government tender). How can you approach this challenge? When using a formalized approach, although usually thorough, you will probably not make it by next week and thus you are forced to make some shortcuts.
It boils down to establishing a set of prioritized requirements and evaluating them against potential solutions. In this blog post I'll provide a general outline of a 'quick and dirty' (not so formal) product selection process which can be done relatively quickly. I'll start with identification and classification of stakeholders. Next I'll suggest several topics to talk to the stakeholders about in order to determine and prioritize requirements. Ending with several suggestions on how to obtain possible candidates for the solution and compare them.
Disclaimer
There is probably overlap with existing approaches to accomplish the same. Please inform me so I can cross check this, learn and add references. I've used CMMI-DEV as an inspiration among others. The below approach is not a company standard. It is an approach I've personally tried and have good experiences with.
Quality is something which is hard to define. Different areas of expertise use their own definitions of what quality is. Without an objective standard which carries weight, anyone can claim to provide a quality product or service according to some standard. This makes it difficult to compare products and to formalize which characteristics a product or service needs to have. In this blog post I'll provide an introduction to ISO/IEC-5055 which is a quality standard which allows us to measure the quality of source code objectively.
Sometimes you want to record online presentations to maybe re-watch them later or share them with colleagues. It is not always possible to easily do this from within the tool which is used for the presentation. An easy alternative is to record such sessions using OBS Studio, a free and open source tool. In this blog post I'll explain how you can do this.
This is my 200th blog post. Also I reached 1M page-views recently. In those 200 blog posts, I used approximately 325K words. An average non-fiction book is somewhere between 70K to 80K words. So that would amount to somewhere around 4 books. Why did I write so many blog posts, what did I write about, which are my favorites, what do readers think? In this rather personal blog post I'm going to look back on the past 10 years of blogging.
Jenkins is a solid CI/CD platform which has proven itself over the years. Many organizations use it to build, test and deploy their applications. In Jenkins it is possible to define credentials or to use an external credential store. You can then use these inside your pipelines and jobs. Direct access to credentials can be limited. Even with limited access, there are still various ways in which you can extract credentials.
This blog post illustrates how you can display credentials in log files as base64 encoded strings so they are not masked and you can easily copy / paste / base64 decode them to obtain and (ab)use them. The method described is not specific to Jenkins but can also be used in various other CI/CD platforms on-premises and in the cloud (such as GitHub Actions and Bamboo).
Note: this is not meant as an encouragement to break rules or laws. Often legislation does not allow you to try and access systems you are not officially authorized to. It is meant to create awareness and to allow you to think about if and how you might want to prevent this in your own CI/CD platform.
SonarCloud is a hosted SonarQube SaaS solution which does not allow 3rd party plugins to be installed. This puts some limitations on the kind of data you can put in SonarCloud. For Java this is limited to Checkstyle, PMD and SpotBugs results. OWASP provides a Dependency-Check plugin to identify vulnerable dependencies in for example your pom.xml file. In this blog post I'll show how to get OWASP Depedency-Check data in SonarCloud without using a 3rd party plugin! Disclaimer: this solution has been created in very little time (~2 hours) and has not been seriously tested, optimized or used in production environments. Use at your own risk!
GitHub Actions allow you to do most CI/CD tasks for free, directly from your GitHub repository. One of the challenges however is that there is no build-in facility like for example SonarQube to manage code quality. Luckily, SonarSource provides SonarCloud; a SonarQube SaaS offering which is free for public projects! It is also easy to feed SonarCloud from GitHub Actions. In this blog post I'll describe how you can do this.
I'm a regular user of GitHub. Recently I discovered GitHub also has a build-in CI/CD workflow solution called GitHub Actions. Curious about how this would work I decided to try it out. I had previously build a Jenkins Pipeline to perform several static and dynamic application security tests on a Java project and decided to try and rebuild this pipeline using GitHub Actions. This blog described my first experiences and impressions.
OWASP ZAP or Zed Attack Proxy is an open source dynamic application security testing (DAST) tool. It is available here and has a website with documentation here. I recently encountered it when looking for open source security test tools to embed in a CI/CD pipeline (here). I was surprised by how versatile this tool is. In this blog post I'll summarize several ways how you can use it.
It is often expected of a DevOps team to also take security into consideration when delivering software. Often however, this does not get the attention it deserves. In this blog post I'll describe some easy to use, CI/CD pipeline friendly, open source tools you can use to perform several checks during your Java software delivery process which will help you identify and fix issues with minimal effort.
You can view my sample project here which implements all these tools. There is also a docker-compose.yml file supplied. SonarQube en Jenkins however do not come preconfigured in this setup. You can look at the Jenkinsfile (pipeline definition) to see what Jenkins configuration is required (installing plugins and creating credentials).
This is provided as an overview and small example only. It is not meant as a walkthrough to setup a CI/CD environment or a manual how to use the different tools. Also the mentioned tools are not all the tools which are available but a set of tools which are freely available, popular and which I managed to get working without too much effort. Using them together allows you to improve the security of your Java applications during your software delivery process and provide quick feedback to developers. This can also help increase security awareness. When you have some experience with these tools you can implement more strict policies (which can let a build fail) and quality gates.
There are situations where you want to change Java code not at the source code level but at runtime. For example when you want to instrument code for logging purposes but do not want to change the source code (because you might not have access to it). This can be done by using a Java Agent. For example, Dynatrace uses a Java agent to collect data from inside the JVM. Another example is the GraalVM tracing agent (here) which helps you create configuration for the generation of native images. Logging is one use-case but you can also more dramatically alter runtime code to obtain a completely different runtime behavior.
This blog post is not a step by step introduction for creating Java Agents. For that please take a look at the following. In this blog post I have created a Java agent which rewrites synchronized methods to use a ReentrantLock instead (see here). The use-case for this is to allow applications to use Project Loom's Virtual Threads more efficiently.
You can find the code of the agent here.
There are a lot of use-cases in which you might want to automate a web-browser. For example to automate tedious repetitive tasks or to perform automated tests of front-end applications. There are also several tools available to do this such as Selenium, Cypress and Puppeteer. Several blog posts and presentations by Lucas Jellema picked my interest in Playwright so I decided to give it a try. I'm not a great fan of JavaScript so I decided to go with Python for this one. I also did some tests with wrk, a simple yet powerful HTTP bench-marking tool, to get an indication about how Playwright would handle concurrency and was not disappointed.
