Software artefacts are developed using a build pipeline. This pipeline consists of several steps to provide quick feedback on software quality by means of code quality checks, automated tests, test coverage checks, etc. When the software is done (adhering to a Definition of Done (DoD)), it is released. This release as a whole is then tested and promoted until it reaches a production environment. In the meantime, work on a next release has already started. This process is shown in the below image. This is a minimal example (especially on the test part). In the below image, you can see there are 3 releases in progress. This is only in order to illustrate the process. You should of course try to limit the number of releases which are not in production to reduce the overhead of fixes on those releases.
Automation of the release process is often a challenge. Why is this difficult? One of the reasons is the identification of what should be in a release. This is especially so when the process of creating a release is not automated. There is a transition phase (in the image between Test phase 2 and Test phase 3) when the unit (artefact) build pipeline stops and when the release as a whole continues to be promoted through the environments. In the image of the process above, you can easily identify where you can automate the construction of a release; at the end of the unit build pipeline. This is where you can identify the unit which has been approved by means of the different test phases / quality checks in the unit build pipeline and you know the release the unit has to be put in to be propagated as a whole. Why not add the unit to the release in an automated fashion there?
Automation
Artifact repository, Maven POM and dependencies
A common practice is to put artifacts in an artifact repository. Artifact repositories are often Maven compliant (i.e. Nexus, Artifactory) so you can identify your artifact with Maven GAV coordinates (groupId, artifactId, version). What better way to describe a collection of artifacts which can be identified with GAV attributes than a Maven POM file? You can define your artifacts as dependencies in your POM file. The Maven assembly plugin can then download those artifacts and put them in a specific structure to allow easy deployment. In case a dependency is already there, you want to update the version in the POM. If it is not there, you want to add it.
Automating adding dependencies to a POM
Below is a short Python (2.7) script to add dependencies to a POM file if the dependency is not there yet or to update the version of the dependency if it already is there.
import os
import xml.etree.ElementTree as ET
import xml.dom.minidom as minidom
import sys,re
import argparse
#script updates a pom.xml file with a specific artifactid/groupid/version/type/classifier dependency
#if the dependency is already there, the version is checked and updated if needed
#if the dependency is not there, it is added
#the comparison of dependencies is based on artifactid/groupid/type (and optionally classifier). other fields are ignored
#the pom file should be in UTF-8
#set the default namespace of the pom.xml file
pom_ns = dict(pom='http://maven.apache.org/POM/4.0.0')
ET.register_namespace('',pom_ns.get('pom'))
#parse the arguments
parser = argparse.ArgumentParser(description='Update pom.xml file with dependency')
parser.add_argument('pomlocation', help='Location on the filesystem of the pom.xml file to update')
parser.add_argument('artifactid', help='ArtifactId of the artifact to update')
parser.add_argument('groupid', help='GroupId of the artifact to update')
parser.add_argument('version', help='Version of the artifact to update')
parser.add_argument('type', help='Type of the artifact to update')
parser.add_argument('--classifier', help='Classifier of the artifact to update',default=None)
args = parser.parse_args()
pomlocation=args.pomlocation
artifactid=args.artifactid
groupid=args.groupid
version=args.version
type=args.type
classifier=args.classifier
#read a file and return a ElementTree
def get_tree_from_xmlfile(filename):
if os.path.isfile(filename):
tree = ET.parse(filename)
return tree
else:
raise Exception('Error opening '+filename)
#obtain a specific element from an ElementTree based on an xpath
def get_xpath_element_from_tree(tree,xpath,namespaces):
return tree.find(xpath, namespaces)
#returns the content of an element as a string
def element_to_str(element):
return ET.tostring(element, encoding='utf8', method='xml')
#returns an ElementTree as a pretty printed string
def elementtree_to_str(et):
root=et.getroot()
ugly_xml = ET.tostring(root, encoding='utf8', method='xml')
dom=minidom.parseString(ugly_xml)
prettyXML=dom.toprettyxml('\t','\n','utf8')
trails=re.compile(r'\s+\n')
prettyXML=re.sub(trails,"\n",prettyXML)
return prettyXML
#creates an Element object with artifactId, groupId, version, type, classifier elements (used to append a new dependency). classifier is left out if None
def create_dependency(param_groupid,param_artifactid,param_version,param_type,param_classifier):
dependency_element = ET.Element("dependency")
groupid_element = ET.Element("groupId")
groupid_element.text = param_groupid
dependency_element.append(groupid_element)
artifactid_element = ET.Element("artifactId")
artifactid_element.text = param_artifactid
dependency_element.append(artifactid_element)
version_element = ET.Element("version")
version_element.text = param_version
dependency_element.append(version_element)
type_element = ET.Element("type")
type_element.text = param_type
dependency_element.append(type_element)
if param_classifier is not None:
classifier_element = ET.Element("classifier")
classifier_element.text = param_classifier
dependency_element.append(classifier_element)
return dependency_element
#adds a dependency element to a pom ElementTree. the dependency element can be created with create_dependency
def add_dependency(pom_et,dependency_element):
pom_et.find('pom:dependencies',pom_ns).append(dependency_element)
return pom_et
#update the version of a dependency in the pom ElementTree if it is already present. else adds the dependency
#returns the updated ElementTree and a boolean indicating if the pom ElementTree has been updated
def merge_dependency(pom_et,param_artifactid,param_groupid,param_type,param_version,param_classifier):
artifactfound=False
pom_et_changed=False
for dependency_element in pom_et.findall('pom:dependencies/pom:dependency',pom_ns):
checkgroupid = get_xpath_element_from_tree(dependency_element,'pom:groupId',pom_ns).text
checkartifactid = get_xpath_element_from_tree(dependency_element,'pom:artifactId',pom_ns).text
checktype = get_xpath_element_from_tree(dependency_element,'pom:type',pom_ns).text
if param_classifier is not None:
checkclassifier_el = get_xpath_element_from_tree(dependency_element,'pom:classifier',pom_ns)
if checkclassifier_el is not None:
checkclassifier=checkclassifier_el.text
else:
checkclassifier=None
else:
checkclassifier = None
if (checkgroupid == param_groupid and checkartifactid == param_artifactid and checktype == param_type and (checkclassifier == param_classifier or param_classifier is None)):
artifactfound=True
print 'Artifact found in '+pomlocation
pomversion=dependency_element.find('pom:version',pom_ns).text
if pomversion != param_version:
print "Artifact has different version in "+pomlocation+". Updating"
dependency_element.find('pom:version',pom_ns).text=param_version
pom_et_changed=True
else:
print "Artifact already in "+pomlocation+" with correct version. Update not needed"
if not artifactfound:
print 'Artifact not found in pom. Adding'
dependency_element = create_dependency(param_groupid,param_artifactid,param_version,param_type,param_classifier)
pom_et = add_dependency(pom_et,dependency_element)
pom_et_changed=True
return pom_et,pom_et_changed
#read the file at the pomlocation parameter
pom_et = get_tree_from_xmlfile(pomlocation)
#merge the dependency into the obtained ElementTree
pom_et,pom_et_changed=merge_dependency(pom_et,artifactid,groupid,type,version,classifier)
#overwrite the pomlocation if it has been changed
if pom_et_changed:
print "Overwriting "+pomlocation+" with changes"
target = open(pomlocation, 'w')
target.truncate()
target.write(elementtree_to_str(pom_et))
target.close()
else:
print pomlocation+" does not require changes"
The script can deal with an optional classifier. When not specified it updates dependencies without looking at the classifier so be careful with this.
Also the script does some pretty printing when updating. This makes it easy to compare the POM file after a version control commit to for example compare different releases.
Seeing it work
Example pom.xml file.
<?xml version="1.0" encoding="utf8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>nl.amis.smeetsm.release</groupId>
<artifactId>Release</artifactId>
<packaging>pom</packaging>
<version>1.0</version>
<dependencies>
<dependency>
<groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId>
<artifactId>HelloWorld_FU</artifactId>
<version>1.0</version>
<type>pom</type>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<configuration>
<descriptors>
<descriptor>release-assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
</plugins>
</build>
</project>
The artifact HelloWorld_FU contains dependencies to other artifacts ending in SCA or SB to indicate if it is a SOA Suite SCA composite artifact or a Service Bus artifact. The release-assembly.xml file below puts the different types in different directories and zips the result. This way a release zip file is created.
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>release</id>
<formats>
<format>zip</format>
</formats>
<dependencySets>
<dependencySet>
<outputDirectory>/composite</outputDirectory>
<includes>
<include>nl.amis.smeetsm.*:*_SCA</include>
</includes>
</dependencySet>
<dependencySet>
<outputDirectory>/servicebus</outputDirectory>
<includes>
<include>nl.amis.smeetsm.*:*_SB</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>
Updating a dependency version: releasescript.py pom.xml HelloWorld_FU nl.amis.smeetsm.functionalunit.HelloWorld 2.0 pom
<?xml version="1.0" encoding="utf8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>nl.amis.smeetsm.release</groupId>
<artifactId>Release</artifactId>
<packaging>pom</packaging>
<version>1.0</version>
<dependencies>
<dependency>
<groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId>
<artifactId>HelloWorld_FU</artifactId>
<version>2.0</version>
<type>pom</type>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<configuration>
<descriptors>
<descriptor>release-assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
</plugins>
</build>
</project>
Adding a dependency version: releasescript.py pom.xml ByeWorld_FU nl.amis.smeetsm.functionalunit.ByeWorld 2.0 pom
<?xml version="1.0" encoding="utf8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>nl.amis.smeetsm.release</groupId>
<artifactId>Release</artifactId>
<packaging>pom</packaging>
<version>1.0</version>
<dependencies>
<dependency>
<groupId>nl.amis.smeetsm.functionalunit.HelloWorld</groupId>
<artifactId>HelloWorld_FU</artifactId>
<version>2.0</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>nl.amis.smeetsm.functionalunit.ByeWorld</groupId>
<artifactId>ByeWorld_FU</artifactId>
<version>2.0</version>
<type>pom</type>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<configuration>
<descriptors>
<descriptor>release-assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
</plugins>
</build>
</project>
Finally
When you want to start with the next release, you should create a branch in your version control system of the current release. This way you can separate releases in version control and can also easily create fixes on existing releases.
No comments:
Post a Comment