Running SoapUI tests on Jenkins

Introduction

It is fine to run small and ad hoc API and functional tests from SoapUI GUI environment but it is not suitable for running automated regression tests that are running on a test automation server by using some form of scheduling mechanism. Fortunately there is a possibility to run SoapUI tests in fully automated fashion on e.g. Jenkins server.

In this article we are going to explain all the steps needed to enable fully automated SoapUI test on Jenkins server.

Automating SoapUI to run on Jenkins

Easiest way to automate SoapUI test in order to run them on Jenkins server is to follow this steps:

  1. Split big SoapUI project
  2. Create managing IDE project
  3. Prepare Maven build
  4. Enable versioning
  5. Running tests on Jenkins

Split big SoapUI project

Big SoapUI can create a lot of problem for management and versioning as a project file, essentially XML file, can become too big and thus cumbersome to: maintain, version and fix, if necessary. In a screenshot below we can see such a big project that is composed of many different Test Suites. Each Test Suite is than composed of many different Test Cases and in turn each Test Case is composed of various Test Steps:

Project with many test suites, test cases and test steps initialization

Fig. 1. Project with many Test Suites, Test Cases and Test Steps

More manageable way is to split this big project in many smaller projects each containing essentially one Test Suite:

Big project split in many smaller

Fig. 2. Big project split in many smaller

NOTE: Split projects have to be saved in specific folder (see below)

Create managing IDE project

Because we have to deal with versioning, Maven and Jenkins files, it is easier if we create project with specific folder structure in development environment of choice. Here we have used IntelliJ to create such a project:

IntelliJ project for SoapUI projects Fig. 3. IntelliJ project for SoapUI projects

Project structure

To run successfully on Jenkins, IntelliJ project structure has to be composed as follows:

├── Jenkinsfile
├── README.md
├── elemif.iml
├── ext
│   └── ojdbc8-12.2.0.1.jar
├── global-groovy.log
├── pom.xml
├── soapui-errors.log
├── soapui-settings.xml
├── soapui.log
├── src
│   └── test
│       ├── resources
│       │   ├── UAT.jks
│       │   ├── database.json
│       │   └── qa.jks
│       └── soapui
│           ├── EMIF-archive.xml
│           ├── EMIF-compression.xml
│           ├── EMIF-content-type.xml
│           ├── EMIF-duplicate-messages.xml
│           ├── EMIF-header-issues.xml
│           ├── EMIF-library.xml
│           ├── EMIF-security.xml
│           └── EMIF-wrong-message.xml
└── target
    ├── surefire-reports
    │   ├── Archive-Initialization-InitDB-0-OK.txt
    │   ├── Archive-Initialization-InitLib-0-OK.txt
    │   └── TEST-Archive.xml
    └── test-classes
        ├── UAT.jks
        ├── database.json
        └── qa.jks

Where:

Normally SoapUI is storing preferences file (soapui-settings.xml) in users home folder. We can copy this file to our IDE project location and update it every time we do some important preference changes trough SoapUI GUI or we can discard file from original location and make symbolic link to the file in our IDE project folder.

Setting absolute and relative paths

When we are running our tests from within SoapUI GUI environment we have to supply absolute paths but when we are running our tests from command line or from e.g. Jenkins server than we have to use relative paths. The same goes for specifying location of Java Keystore used to sign and/or encrypt our messages. So in a project file (e.g. EMIF-archive.xml) we have to serch for the location of Java Keystore database (e.g. qa.jks) and than assure that absolute path is used:

Absolute Java Keystore path in project file Fig. 4. Absolute Java Keystore path in project file

When wi then open our project in SoapUI and we open Project->Show Project View menu, we can verify that SoapUI can recognize and verify our Keystore

Absolute Java Keystore path in Show Project View dialog box Fig. 5. Absolute Java Keystore in Show Project View dialog

However to run the same project on Jenkins server we have to supply relative paths. Thus we have to modify project file to use relative path:

Relative Java Keystore path in project file Fig. 6. Relative Java Keystore path in project file

When we than open the same project in SoapUI we can see that SoapUI is not able to recognise location of Java Keystore and thus not able to use it:

Relative Java Keystore path in Show Project View dialog box Fig. 7. Relative Java Keystore in Show Project View dialog

The same principle has to be applied to all other files. Unfortunately SoapUI is not capable to work with relative paths so in order to test our test cases in SoapUI we have to use absolute paths but to run it on Jenkins server we have to use relative paths.

Prepare Maven build

SoapUI Maven artefacts are not stored on Maven Central repository and thus we have to add additional repository location to Maven settings file which is located in users home folder (usually: $HOME/.m2/settings.xml), assuming that we have dowloaded and installed Maven support previously (Installation instructions can be found here).

The easiest way, thus to modify settings.xml file is by adding additional <profile> XML element under the <profiles> section and than reference it in <activeProfiles> section, as in this code excerpt from settings.xml:

<settings>
  .
  .
  .
 <profiles>
    <profile>
      <id>maven.central</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>central</id>
          <name>Maven2 Central</name>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </snapshots>
          <url>https://repo.maven.apache.org/maven2</url>
          <layout>default</layout>
        </repository>
      </repositories>
    </profile>

    <profile>
      <id>com.smartbear</id>

      <activation>
        <activeByDefault>false</activeByDefault>
      </activation>

      <repositories>
        <repository>
            <id>smartbear-sweden-plugin-repository</id>
            <url>http://www.soapui.org/repository/maven2/</url>
        </repository>
      </repositories>

      <pluginRepositories>
        <pluginRepository>
            <id>smartbear-sweden-plugin-repository</id>
            <url>http://www.soapui.org/repository/maven2/</url>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>

  <activeProfiles>       
     <activeProfile>com.smartbear</activeProfile>
  </activeProfiles>
</settings>

After we have enabled additional repository location for SoapUI specific artefacts, we are able to specify content of pom.xml which is Maven build file and is located at the root of our IDE project location. Content of pom.xml file can look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <artifactId>emif</artifactId>
    <version>1.0-SNAPSHOT</version>


    <build>
        <plugins>

            <plugin>
                <groupId>com.smartbear.soapui</groupId>
                <artifactId>soapui-maven-plugin</artifactId>
                <version>5.4.0</version>

                <executions>
                  <execution>
                       <id>EMIF-archive</id>
                       <configuration>
                           <settingsFile>soapui-settings.xml</settingsFile>
                           <projectFile>src/test/soapui/EMIF-archive.xml</projectFile>

                           <outputFolder>target/surefire-reports</outputFolder>
                           <testSuite>Archive</testSuite>
                           <!--<testCase>1\. Verify request and response are logged for all interfaces for accepted messages</testCase>-->
                           <!--<testCase>Initialization</testCase>-->
                           <!--<testCase>CommonFunctions</testCase>-->
                           <junitReport>true</junitReport>
                           <exportAll>true</exportAll>
                           <printReport>true</printReport>
                           <testFailIgnore>true</testFailIgnore>
                       </configuration>
                       <goals>
                           <goal>test</goal>
                       </goals>
                       <phase>test</phase>
                   </execution>

                    <execution>
                        <id>EMIF-compression</id>
                        <configuration>
                            <settingsFile>soapui-settings.xml</settingsFile>
                            <projectFile>src/test/soapui/EMIF-compression.xml</projectFile>

                            <outputFolder>target/surefire-reports</outputFolder>
                            <testSuite>Compression</testSuite>
                            <!--<testCase>2\. Wrong Organization</testCase>-->
                            <junitReport>true</junitReport>
                            <exportAll>true</exportAll>
                            <printReport>true</printReport>
                            <testFailIgnore>true</testFailIgnore>
                        </configuration>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <phase>test</phase>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

</project>

The Maven plugin responsible for executing build process is: soapui-maven-plugin and this plugin allows for one or more <execution> blocks inside <executions> element. Each <execution> block is responsible for running one SoapUI project. Which project file to run, which test suite to run or even specific test case to run, is defined with: <projectFile>, <testSuite> and <testCase> elements.

Now we can test our build and execute all tests from command line by executing following maven command:

mvn clean test

If everything is successful Maven will execute all tests specified in pom.xml file, print results on the console and store them in text files in target/surefire-reports folder

Enable versioning

To have full Continuous Integration (CI) cycle we have to initialize our IDE project folder as version controlled folder. For GIT version control system it is done with command:

git init
git add .
git commit -m "first commit"

Than we have to create repository on the version control server that will contain our project. For GitHub it is done by clicking on the “New” button under Repositories section. After we have given the name to repository we can upload our project to version control server. For GitHub:

git remote add origin https://github.com/<your user name>/emif.git
git push -u origin master

Than we can use:

git commit -a -m "description of change"
git push

each time we have modified our projects and we would like to push changes to Jenkins

Running tests on Jenkins

Prepare Jenkins file

We want to use new Jenkins pipeline build facilities that treats build process as a code. Therefore we have to create Jenkins file that contains build instructions in a declarative form or in a form of Groovy code. Our Jenkins file looks like this:

pipeline {
    options {
        disableConcurrentBuilds()
        buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '50'))
    }

    agent any

    tools {
        maven 'maven3'
        jdk 'jdk8'
        allure 'allure2'
    }

    triggers {
        pollSCM 'H/5 * * * *'
          cron '''TZ=Europe/Oslo
        H H(13-14) * * *'''
    }

    stages {
        stage('Test') {
            steps {
                sh '''
                    mvn clean test
                '''
            }
            post {
                always {
                    archiveArtifacts 'target/surefire-reports/**/*.xml'
                    junit 'target/surefire-reports/**/*.xml'
                }
            }
         }
    }
}

It essentially says that we do not allow parallel builds, we use maven to build, build will start whenever we post changes to source version control server and than on daily schedule. It also says that the test outputs are going to be available in: target/surefire-reports/ folder on the server.

Prepare Jenkins server

After we have prepared Jenkins file we have to create new Jenkins project by clicking on New Item button in Jenkins classic user interface web console:

 Jenkins classic user interface web console Fig. 8. Jenkins classic user interface web console

From the new page that opens we can chose to create “Pipeline” project or “Multibranch pipeline” if we plan to have multiple branches in our version control system (e.g. “master”, “stage”) that we build separately with different options:

 Miltibranch Pipeline creation option Fig. 9. Miltibranch Pipeline creation option

When we have chosen appropriate option we have to configure project. First we establish connection with version control system:

 Connection to version control system Fig. 10. Connection to version control system

Finally we define how our build process is going to be executed. As we have opted to use Jenkins file to define our build process, we are going to specify that option in “Build Configuration” tab:

 Build configuration tab Fig. 11. Build configuration tab

Checking test executions on Jenkins

After we have saved project we are ready to run our first test build. Build can be run automatically by pushing changes to version control server, waiting on a schedule or running it manually. To running it manually, we have to go to the Jenkins web console, and than click project name we gave to Jenkins project in previous step. Than we can chose version control branch we want to execute:

Jenkins project page Fig. 12. Jenkins project page

By clicking on a branch name the page that contains executions statuses and links to reports is opened. We can than click on “Build Now” button to start our build manually. To check build results we click on build execution number:

Branch execution status page Fig. 13. Branch execution status page

Clicking on build execution number opens the page that give us access to various execution status information. Like: access to text console output, pipeline steps, etc. and most important test results:

Execution status information Fig. 14. Execution status information

Test result page displays all successfully executed tests and all failed tests:

Test results page Fig. 15. Test results page

By clicking on the failed test name link we can see detailed explanation why test failed that allows for easier investigation and fix:

Failure details page Fig. 16. Failure details page

Conclusion

Setting up and running SoapUI tests on Jenkins is not trivial process but but is worth time and effort investment, because this allows us to run fully automated regression test that will allow for automatic checking of the system on the regular basis. This process also generates report for every failure that is discovered in the process allowing us to apply fixes and patches immediately or to roll back our deployments.