In this article I will provide a simple example on how you can quickly add value to your software project by improving traceability. This is done by combining the information from version control (SVN) and issue management (JIRA) to generate release notes and enforcing some version control rules.
To allow this to work, certain rules need to be adhered to.
- code is committed using a commit message or tag which allows linking of code to issue or change
- it should be possible to identify the code which is part of a release from version control
- the bug tracking system should allow a selection of issues per release
Version control; link code to function
In this example I'll talk about Subversion since I have most experience with this. Git also supports a similar mechanism of commit hooks. SVN can easily be installed and a repository be created by doing what is described on: http://www.civicactions.com/blog/2010/may/25/how_set_svn_repository_7_simple_steps
First you need to make sure you can link your code to your functionality. This is easily done with commit messages. In a small team you can quickly agree on a set standard and use that. When the team grows larger and more distributed, enforcing standards, becomes more of a challenge. SVN provides pre-commit hooks which can provide the needed functionality to require a certain format in the commit message. This avoids deviations of the agreed standard and allows more easily to extract (reliable) information from version control commit messages.
After creation of this repository, there will be a 'hooks' folder underneath the specified directory. Templates for hooks are provided there. Those are in shell scripts however and I prefer Perl for this. Mind though that the pre-commit hook script (even if it is a Perl file) should be executable!
In the below script I check for the format of a JIRA issue. You can also look at: http://stackoverflow.com/questions/10499098/restricting-subversion-commits-if-the-jira-issue-key-is-not-in-the-commit-messag. This allows commits to be prevented by directly checking Jira. If you want to allow check-ins specifying a JIRA ID while not checking JIRA itself, you can use the below example. It also checks the directory (myproject directly under the repository root). Usually multiple projects use the same repository and you don't want to bother everyone with your beautiful commit standards.
#!/usr/bin/perl -w
use strict;
my $repos = $ARGV[0];
my $txn = $ARGV[1];
my $svnlook = '/usr/bin/svnlook';
my $require = '\[([A-Z_0-0]+-[0-9]+)\]';
my $checklog = "N";
foreach my $line (`$svnlook changed -t "$txn" "$repos"`)
{
chomp($line);
if ($line !~ /^\s*(A|D|U|UU|_U)\s*(.+)$/)
{
die "!!Script Error!! Can't parse line: $line\n";
} else {
if ($2 =~ /^myproject.*$/)
{
$checklog = "Y";
}
}
}
if ($checklog ne "N")
{ my $log = `$svnlook log -t $txn $repos`;
if ($log =~ /$require/) {
exit 0;
} else {
die "No JIRA issue specified. Commit aborted!\n";
}
}
[maarten@localhost trunk]$ svn commit -m'Please kick me'
Adding trunk/test.txt
Transmitting file data .svn: Commit failed (details follow):
svn: Commit blocked by pre-commit hook (exit code 255) with output:
No JIRA issue specified. Commit aborted!
[maarten@localhost trunk]$ svn commit -m'[ABC-1]: Nice commit message'
Adding trunk/test.txt
Transmitting file data .
Committed revision 386327.
Extract issue numbers
From JIRA
The JIRA API can be used to extract issues using a selection. Your selection might differ. Below is just an example giving me issues of project ABC assigned to user smeetsm with password "password". It is a nice example of how simple the JIRA API is. Also it gives an example on how to extract specific information using the commandline from a JSON string. You can see the same regular expression as the one used in the pre-commit hook.
/usr/bin/curl -u smeetsm:password http://jira/rest/api/2/search?jql=project=ABC%20and%20assignee=smeetsm
| grep -Eho '"key":"([A-Z_0-0]+-[0-9]+)"'
This command can have output like:
"key":"ABC-1"
"key":"ABC-2"
"key":"ABC-3"
If you pipe this to a file (issues.txt), you can easily convert this to an XML by using something like:
echo \<issues\>;cat issues.txt | sed 's/"key"=\"\(.*\)"/\<issue\>\1\<\/issue\>/'; echo \</issues\>
This will have as output:
<issues>
<issue>ABC-1</issue>
<issue>ABC-2</issue>
<issue>ABC-3</issue>
</issues>
I choose this method of converting the JSON to XML since I wanted minimal overhead in my process (quick, easy, as few as possible external dependencies).
From SVN
You can use the following Python script to parse the SVN log and get the issues checked in from there. The script requires Python 2.7 and the lxml library. The lxml library (+installer) can be downloaded at: https://pypi.python.org/pypi/lxml/ or you can download it using the Python package manager PIP (supplied with Python 2.7.9+).
I have specified a duration between 2015-03-09 and now (HEAD) to identify the release. Identifying a release is usually done by looking at a release branch but the method is similar. You can again see the same regular expression which has been used in the pre-commit hook and in the Jira API call.
import os
import xml.etree.ElementTree as ET
import re
import subprocess
def getsvnlog():
p = subprocess.Popen(['/usr/bin/svn','log','--verbose','--xml','-r','{2015-03-09}:HEAD','file:///home/maarten/myrepository/myproject'],stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
return out
def getfilesfromlogentry(logitem):
result=[]
for path in logitem.findall("./paths/path[@kind='file']"):
result.append(path.text)
return result
def getfieldfromlogentry(logitem,fieldname):
result=[]
for item in logitem.findall("./"+fieldname):
result.append(item.text)
return result
def parse_svnlog():
svnlog = getsvnlog()
root = ET.fromstring(svnlog)
uniquelist={}
print "<issues>"
for logitem in root.findall("./logentry"):
for msg in getfieldfromlogentry(logitem,"msg"):
p = re.compile('([A-Z_0-0]+-[0-9]+)')
iterator = p.finditer(msg)
for match in iterator:
print "<issue>"+ msg[match.start():match.end()]+"</issue>"
print "</issues>"
return root
parse_svnlog()
This will yield a result like:
<issues>
<issue>ABC-1</issue>
<issue>ABC-3</issue>
<issue>ABC-4</issue>
</issues>
Generate release notes
Once you have issue numbers from version control and from issue management, you can do interesting things like generating release notes or just a report. The nice thing here is that by comparing the issues from version control and issue management, you can draw interesting conclusions.
If for example you have the following issues from SVN (svnissues.xml):
<issues>
<issue>ABC-1</issue>
<issue>ABC-3</issue>
<issue>ABC-4</issue>
</issues>
And the following from Jira (jiraissues.xml):
<issues>
<issue>ABC-1</issue>
<issue>ABC-2</issue>
<issue>ABC-3</issue>
</issues>
You'll notice ABC-4 is only present in SVN and ABC-2 is only present in JIRA. Why is that? Has the developer checked in code he was not supposed to? Has the developer checked in the code in the correct release branch? Is the JIRA issue status correct? It is something which should be investigated and corrected.
You can use the following Python script combined with the following XSL to produce output. The layout and contents of the release notes is of course greatly simplified. This is usually very customer specific.
transform.pl
from xml.dom.minidom import *
import lxml.etree as ET
dom1 = ET.Element("dummy")
xslt = ET.parse("transform.xsl")
transform = ET.XSLT(xslt)
print(ET.tostring(transform(dom1), pretty_print=True))
The XSLT shows how you can load XML files and use a reusable template call to compare the results.
transform.xsl:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:variable name="issues1" select="document('issuessvn.xml')"/>
<xsl:variable name="issues2" select="document('issuesjira.xml')"/>
<xsl:template match="/">
<html>
<body>
<xsl:for-each select="$issues1/issues/issue">
<xsl:call-template name="getIssue">
<xsl:with-param name="search" select="."/>
<xsl:with-param name="content" select="$issues2"/>
<xsl:with-param name="ident1" select="'SVN'"/>
<xsl:with-param name="ident2" select="'JIRA'"/>
<xsl:with-param name="showfound" select="true()"/>
<xsl:with-param name="shownotfound" select="true()"/>
</xsl:call-template>
</xsl:for-each>
<xsl:for-each select="$issues2/issues/issue">
<xsl:call-template name="getIssue">
<xsl:with-param name="search" select="."/>
<xsl:with-param name="content" select="$issues1"/>
<xsl:with-param name="ident1" select="'JIRA'"/>
<xsl:with-param name="ident2" select="'SVN'"/>
<xsl:with-param name="showfound" select="false()"/>
<xsl:with-param name="shownotfound" select="true()"/>
</xsl:call-template>
</xsl:for-each>
</body>
</html>
</xsl:template>
<xsl:template name="getIssue">
<xsl:param name="search"/>
<xsl:param name="content"/>
<xsl:param name="ident1"/>
<xsl:param name="ident2"/>
<xsl:param name="showfound"/>
<xsl:param name="shownotfound"/>
<xsl:choose>
<xsl:when test="$content/issues/issue[text()=$search]">
<xsl:if test="$showfound">
<p>Issue <xsl:value-of select="$search"/> found in <xsl:value-of select="$ident1"/> and <xsl:value-of select="$ident2"/></p>
</xsl:if>
</xsl:when>
<xsl:otherwise>
<xsl:if test="$shownotfound">
<p>Issue <xsl:value-of select="$search"/> found in <xsl:value-of select="$ident1"/> but not in <xsl:value-of select="$ident2"/></p>
</xsl:if>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
</xsl:stylesheet>
Finally
Take a look at the sample generated release notes below. Of course a very simple sample only focusing on version control and issue management.
Issue ABC-1 found in SVN and JIRA
Issue ABC-3 found in SVN and JIRA
Issue ABC-4 found in SVN but not in JIRA
Issue ABC-2 found in JIRA but not in SVN
You now have a means to check whether the developer was allowed to check the code into version control and what the status of the change/bug was. You are now also able to identify which parts of other issues might also be part of the release (by accident?). If you allow developers to indicate which issues are part of the release, they will most likely not be 100% accurate (describe release content with developer prejudice). If you automate this, you can be at least more accurate. Because you check version control against issue management, you also have a means to make the issue management information more accurate. Maybe for example someone forgot to update the issue status or put in the correct fix release. Both improve traceability from code to release.
Small note
You can do many things with version control hooks. You can do code compliance checks, check character sets, check filename conventions. All of these will help improve code quality. You can provide people with all kinds of interesting reports from version control and issue management about developer productivity and the quality of work they provide. Be careful with this and keep the custom scripts small and maintainable (unless of course you want to stay there forever).
No comments:
Post a Comment