Apache NiFi can be used to accelerate big data projects by allowing easy integration between various data sources. Using Apache NiFi it is easy to track what happened to your data (data provenance) and to provide features like guaranteed ordered delivery and error handling. In this example I'm going to configure NiFi to read COVID data from a REST API, split the data into individual records per country and publish the result to a Kafka topic. I've used the environment described here.
Vagrant can be used to quickly create development environments in for example VirtualBox, VMWare or Hyper-V. I decided to use Vagrant to create a quick Apache NiFi development environment. For Apache NiFi development, you also often require input/output for which Kafka can be used, the NiFi Registry to manage shared resources and of course NiFi itself. Setting this up yourself can be cumbersome. That's why I've created some scripts to help you do this quickly. You can find them here.
Since manually scripting the installations of all these products can be a challenge / work, I decided to use Docker images which often already provide an automatic installation (so I don't have to do that myself) and used Docker Compose to easily allow the containers to find each other and have a docker-compose.yml which contained my environment variables so I wouldn't have to supply them on the commandline.
Your customer has a specific challenge and wants to have it solved by means of software. They ask you as a software/solution architect to advise them on this topic and they want to be able to choose a solution based on your advise by the end of next week (we're not talking about something as large as a government tender). How can you approach this challenge? When using a formalized approach, although usually thorough, you will probably not make it by next week and thus you are forced to make some shortcuts.
It boils down to establishing a set of prioritized requirements and evaluating them against potential solutions. In this blog post I'll provide a general outline of a 'quick and dirty' (not so formal) product selection process which can be done relatively quickly. I'll start with identification and classification of stakeholders. Next I'll suggest several topics to talk to the stakeholders about in order to determine and prioritize requirements. Ending with several suggestions on how to obtain possible candidates for the solution and compare them.
Disclaimer
There is probably overlap with existing approaches to accomplish the same. Please inform me so I can cross check this, learn and add references. I've used CMMI-DEV as an inspiration among others. The below approach is not a company standard. It is an approach I've personally tried and have good experiences with.
Quality is something which is hard to define. Different areas of expertise use their own definitions of what quality is. Without an objective standard which carries weight, anyone can claim to provide a quality product or service according to some standard. This makes it difficult to compare products and to formalize which characteristics a product or service needs to have. In this blog post I'll provide an introduction to ISO/IEC-5055 which is a quality standard which allows us to measure the quality of source code objectively.
Sometimes you want to record online presentations to maybe re-watch them later or share them with colleagues. It is not always possible to easily do this from within the tool which is used for the presentation. An easy alternative is to record such sessions using OBS Studio, a free and open source tool. In this blog post I'll explain how you can do this.
This is my 200th blog post. Also I reached 1M page-views recently. In those 200 blog posts, I used approximately 325K words. An average non-fiction book is somewhere between 70K to 80K words. So that would amount to somewhere around 4 books. Why did I write so many blog posts, what did I write about, which are my favorites, what do readers think? In this rather personal blog post I'm going to look back on the past 10 years of blogging.
Jenkins is a solid CI/CD platform which has proven itself over the years. Many organizations use it to build, test and deploy their applications. In Jenkins it is possible to define credentials or to use an external credential store. You can then use these inside your pipelines and jobs. Direct access to credentials can be limited. Even with limited access, there are still various ways in which you can extract credentials.
This blog post illustrates how you can display credentials in log files as base64 encoded strings so they are not masked and you can easily copy / paste / base64 decode them to obtain and (ab)use them. The method described is not specific to Jenkins but can also be used in various other CI/CD platforms on-premises and in the cloud (such as GitHub Actions and Bamboo).
Note: this is not meant as an encouragement to break rules or laws. Often legislation does not allow you to try and access systems you are not officially authorized to. It is meant to create awareness and to allow you to think about if and how you might want to prevent this in your own CI/CD platform.
