To accomplish this I've used Maven with several plugins;
- jmeter-maven-plugin (http://wiki.apache.org/jmeter/JMeterMavenPlugin) for executing JMeter tests from Maven
- xml-maven-plugin (http://mojo.codehaus.org/xml-maven-plugin/) for transforming the resulting JTL file to a readable HTML report
- exec-maven-plugin (http://mojo.codehaus.org/exec-maven-plugin/) for calling scripts
- maven-postman-plugin (http://doc.fortysix.ch/maven/maven-postman-plugin/) for sending e-mails
I've also used conditional profiles which act on the result from previous build steps to only send a mail in case of errors. In the following post it is described you shouldn't be doing this; http://www.blackbuild.com/how-to-really-use-maven-profiles-without-endangering-your-karma/ ('Using a profile should not result in a different artifact than building without the profile.') but I did it anyway because it seemed useful in this case.
I encountered several challenges which I will describe in this post and how I fixed them. The sample code can be downloaded here; https://dl.dropboxusercontent.com/u/6693935/blog/jmeter-test.zip
Implementation
JMeter
I had encountered JMeter before;
- during the SOA Blackbelt training to do performance tests (http://javaoraclesoa.blogspot.nl/2013/07/oracle-soa-blackbelt-training-june-2013.html)
- it was used in the Oracle SOA Suite 11g Performance Tuning Cookbook (http://javaoraclesoa.blogspot.nl/2013/08/book-review-oracle-soa-suite-11g.html).
Since I have more experience with SOAP UI, this seemed a nice opportunity to try out JMeter. What I did was create a Test Plan to test the availability (HTTP response code 200) of several services on different environments.
In this sample I've used several Google sites (www.google.com, www.google.de, www.google.nl) to simulate environments and several URL's which are available at the different sites in order to simulate services.
To achieve this I've created a loop in JMeter to process environments and within that loop another loop to loop over services. The Test Plan looks as followed;
In order to create the loop within loop construction, I've used posts like; like http://stackoverflow.com/questions/12993754/how-to-implement-nested-loop-in-jmeter
Variable usage
To use a user defined variable like below (envLoopSize)
BeanShell PreProcessor
The BeanShell PreProcessor allows you to introduce some flexibility. For example setting environment variables based on the loopcounter;
Results
The combination of the Response Assertion and the Assertion Results produces output which can be transformed to a report. I'm checking for HTTP status 200 (OK).
Project structure
I've used two POM files. One master POM file for the logic of executing the tests and generating the HTML report. Another POM using a conditional profile for sending the e-mail.
Executing JMeter tests
The main POM contains several interesting parts;
<plugin>
<groupId>com.lazerycode.jmeter</groupId>
<artifactId>jmeter-maven-plugin</artifactId>
<version>1.8.1</version>
<executions>
<execution>
<id>jmeter-tests</id>
<phase>verify</phase>
<goals>
<goal>jmeter</goal>
</goals>
</execution>
</executions>
<configuration>
<ignoreResultFailures>true</ignoreResultFailures>
<testResultsTimestamp>false</testResultsTimestamp>
</configuration>
</plugin>
To execute JMeter tests, the above plugin is used. This executes all JMX files (JMeter Executable) from the \src\test\jmeter folder. The results are put in target\jmeter\results. I want to ignore failures since I want all tests to be executed and mail the results; the build should not fail in case of errors. Since I want to have a predictable output filename for the reports, I set the testResultsTimestamp to false. This makes sure the results file has the same name (except extension) as the JMeter Executable.
Transforming the results file to a readable report
The following plugin is used to transform the result;
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>xml-maven-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>transform</goal>
</goals>
</execution>
</executions>
<configuration>
<transformationSets>
<transformationSet>
<dir>${project.build.directory}/jmeter/results</dir>
<stylesheet>${project.basedir}/src/main/jmeter-results-report_21.xsl</stylesheet>
<outputDir>${project.build.directory}/jmeter/results</outputDir>
<fileMappers>
<fileMapper implementation="org.codehaus.plexus.components.io.filemappers.RegExpFileMapper">
<pattern>(.*?)\s(.*?)</pattern>
<replacement>$1$2</replacement>
<replaceAll>true</replaceAll>
</fileMapper>
<fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
<targetExtension>.html</targetExtension>
</fileMapper>
</fileMappers>
</transformationSet>
</transformationSets>
</configuration>
</plugin>
I have used two filemappers to remove spaces from the filename. I've had some issues with spaces which I thought I could solve this way. Even though this construction works, project.basedir and project.build.directory contain spaces so I had to work around this another way. Since it can be useful code in some cases I left it in. The jmeter-results-report_21.xsl is supplied with JMeter in the extras folder. Other report forms are available also.
Profiles
Maven profiles seem flexible at first impression, however activation conditions are evaluated before execution is started so you can not use the result from a previous build step to make choices. At first I tried with setting properties. The properties however could not be set while executing (at least it didn't work for me quickly enough). My second try was using the presence of a file for conditional activation of a profile. This didn't work in the same POM since the profile activation is evaluated before and not during the build. I decided to split the project up into a main and a child POM file and call the child POM with the exec-maven-plugin. This way the child POM is executed in a separate process and the activation property for the profile is evaluated. It took some effort to have a file present in case of failed tests. I'll describe this in the next paragraph;
<execution>
<id>send-mail</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<workingDirectory>${project.basedir}/sendmail</workingDirectory>
<executable>execmvn.bat</executable>
<arguments>
<argument>${project.build.directory}\jmeter\results</argument>
<argument>HTTPRequestDefaults.html</argument>
<argument>My Project</argument>
</arguments>
</configuration>
</execution>
Since if the arguments in the exec-maven-plugin contain spaces, they are put within quotes (even the CDATA trick didn't help; http://stackoverflow.com/questions/11544581/pass-string-with-white-space-using-commandlineargs-in-maven-commandlineargs-plug), I could not succesfully use -Dhtmldir=${project.build.directory}\jmeter\results since it would come out as "-Dhtmldir=C:\Path With Spaces\jmeter\results" which of course could not be processed correctly ("-Dhtmldir=C:\Path With Spaces\jmeter\results" is not a valid Maven goal). Because of this I decided to use a custom DOS BATCH file to wrap the execution. This way passing the parameters would always go correctly since it would not matter if quotes would be present around a filename anymore.
The part in the POM for activation of the profile looks like this;
<profiles>
<profile>
<id>sendmails</id>
<activation>
<file>
<exists>failures.txt</exists>
</file>
</activation>
<exists>${project.basedir}\failures.txt</exists> did not work so I used <exists>failures.txt</exists> which did work and set the workingDirectory argument in the exec-maven-plugin.
Have a file present in case of failed assertions in the test
In order to have a file present in case the report contained failures, I used the following;
<execution>
<id>determine-failures</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${project.basedir}/src/main/findstrcmd.bat</executable>
<arguments>
<argument>failed</argument>
<argument>${project.build.directory}\jmeter\results\HTTPRequestDefaults.html</argument>
<argument>${project.basedir}\sendmail\failures.txt</argument>
</arguments>
<successCodes>
<successCode>1</successCode>
<successCode>0</successCode>
</successCodes>
</configuration>
</execution>
<execution>
<id>remove-empty</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${project.basedir}/src/main/delifempty.bat</executable>
<arguments>
<argument>${project.basedir}\sendmail\failures.txt</argument>
</arguments>
</configuration>
</execution>
I first considered using the Maven enforcer plugin which has several checks; http://maven.apache.org/enforcer/enforcer-rules/index.html but I don't want to skip tests or have the build failed because a file is present so I decided not to use it. Then I decided to use findstr (a Windows command) to find occurrences of the string failed in the generated report. I used a wrapper dos batch file to circumvent the quote problem. In this batch file I piped the result to a file; failures.txt. The result code of findstr would be 1 if there were no results so I needed to add a successCodes part to make it continue for either result. Then I deleted the file if it was empty (no results) so presence of the file would indicate failed tests. I used again a wrapper batch file to do this to avoid issues with spaces in the filename.
The code for the batch file to delete an empty file (if present) is as follows;
@echo off
set str=%1
set str=%str:"=%
if exist "%str%" (
>nul findstr "^" "%str%" || del "%str%"
)
It can handle presence or absence of quotes in the parameter.
Sending mail; maven-postman-plugin
For sending mail I've used maven-postman-plugin. I've tried two versions of maven-mail-plugin but both didn't work for me out of the box. This one did and looks quite extensive in functionality (http://doc.fortysix.ch/maven/maven-postman-plugin/send-mail-mojo.html; for example SSL support) and it worked without much effort. I've used parameters which are passed from the exec-maven-plugin to the execmvn.bat file and passed as properties to the build.
<plugin>
<groupId>ch.fortysix</groupId>
<artifactId>maven-postman-plugin</artifactId>
<version>0.1.2</version>
<executions>
<execution>
<id>send a mail</id>
<phase>install</phase>
<goals>
<goal>send-mail</goal>
</goals>
<inherited>false</inherited>
<configuration>
<from>misterjenkins@localhost.localdomain</from>
<subject>Test results voor ${applicatienaam}</subject>
<failonerror>true</failonerror>
<mailhost>smtp.localhost.localdomain</mailhost>
<htmlMessageFile>${htmldir}/${htmlfilename}</htmlMessageFile>
<receivers>
<receiver>maarten.smeets@localhost.localdomain</receiver>
</receivers>
<fileSets>
<fileSet>
<directory>${htmldir}</directory>
<includes>
<include>${htmlfilename}</include>
</includes>
</fileSet>
</fileSets>
</configuration>
</execution>
</executions>
</plugin>
Results
If there are failed tests, a mail is send. This mail looks like this;
However the report is not send since the failures.txt file is not present. This causes the profile which defines the plugin execution not to be activated.
Conclusion
Maven has several strong points. It provides an IDE independent project definition and strong dependency management. When using Maven however you are very dependent on plugin availability and the way Maven wants you to do things. It can be difficult to deal with these constraints. It is possible to use workarounds to allow some conditional execution during your build. Using Maven profiles for this is not recommended. Even when using Maven plugins, these plugins provide additional constraints such as that arguments are altered if they contain spaces.
There are many Maven plugins, such as for example for execution of JMeter tests and sending of e-mails. These worked great out of the box and allow integration tests to be embedded into your build and mailed to interesting parties. Also XML transformations are easily accomplished.
The code can be downloaded here; https://dl.dropboxusercontent.com/u/6693935/blog/jmeter-test.zip
No comments:
Post a Comment