Making Performance Testing part of your Continuous Integration Environment

Continuous integration quickly became one of the key principles of agile development and DevOps as it makes our life easier in so many ways. It is important to realise, however, that every passed build still might have a negative effect on overall performance. That is why performance testing should be always included in the CI process. Let’s take a look at how to do it.

The necessary first step in making performance testing part of continuous integration is enabling automatic test evaluation – automatically determining whether the test (or one of its parts) failed or passed.

We call this feature acceptance criteria. It allows you to create your own criteria to evaluate, for example, whether the average response time did not exceed a certain threshold or the CPU was not overloaded and so on:

Step 1: Follow this acceptance criteria tutorial

After you create the criteria you can proceed to setting the CI tool. In this article we’ll take a look at two most common tools:

Jenkins
Bamboo

The principles are very similar in any CI tool, so it doesn’t really matter which tool you want to use.

Load testing in Jenkins

Let’s assume we have this simple pipeline.

1 load testing continuous integration

First of all, Jenkins pulls the source files from VCS and run unit tests. If unit tests pass, the application is built (packaged as JAR) and deployed to Test environment. Then Jenkins starts the performance test.

When the test is finished, SmartMeter.io will evaluate acceptance criteria and only if they pass, the application is deployed to production.

Don’t forget that it is SmartMeter.io evaluating the criteria, not Jenkins, so there is no dependency on Jenkins and you can use any other CI tool.

How does Jenkins know whether the test passed or failed? If the test passes, SmartMeter.io exits with code 0, if test fails it exits with code 2. Non-zero exit code is usually interpreted as a failure in any CI tool.

The great thing is that you immediately know which commits (and by whom) might have negatively affected the performance.

2 load testing continuous integration

This is how you run the performance test (in Non-GUI mode).

cd PATH-TO-SMARTMETER

./SmartMeter.sh runTestNonGui your-test.jmx

3 load testing continuous integration

Load testing in Bamboo

The process of setting up load testing in Bamboo is very similar to Jenkins as you can see in the video.

Here is a snippet of code used in the video:

Start SmartMeter.io test

runTestNonGui test-short.jmx -Jetn_reports_path=/home/vernjan/temp/reports -Jetn_report_name=${bamboo.buildNumber}

Collect artifacts

/home/vernjan/temp/reports/report-${bamboo.buildNumber}

Tips & Tricks

SmartMeter.io report is in HTML format, CI tools usually embeds a web server which can very conveniently display the report for you (in Jenkins we recommend HTML Publisher Plugin).
To set where the report is saved use etn_reports_path property
To set how the report is named use etn_report_name property

For example for Jenkins

cd /home/vernjan/apps/SmartMeter/

./SmartMeter.sh runTestNonGui test-short.jmx -Jetn_reports_path=$WORKSPACE ‑Jetn_report_name=$BUILD_NUMBER

*Note the -J prefix if passing as an argument.

You can also find out more in the documentation

Did this tutorial help you or you could use any other guidance? Let us know in the comments.

You can use acceptance criteria and continuous integration even in the free version of SmartMeter.io, so don’t hesitate and try it for your own!

Download SmartMeter.io for free

Filed under: Performance Testing / Continuous Integration