Introduction

You can perform analysis and testing by executing the Jtest goals for Maven for the main project directory. It is not necessary to build the tested project prior to analysis unless your project has never been built before and you need to ensure that external project dependencies are available, or you test a multi-module project. We recommend compiling multi-module projects before running analysis to enable Jtest to use artifacts from the local repository, reducing the amount of time necessary to test and analyze code.

As an aggregator, the Jtest Plugin for Maven must be executed only once during the build.

Running Static Analysis

To perform static analysis on your code:

  1. Ensure that the Jtest Plugin for Maven is set up (see Configuring the Jtest Plugin for Maven).
  2. Execute the jtest:jtest goal. Your command line may resemble the following:

    mvn jtest:jtest
  3. Review the analysis results (see Reviewing Results).

If Jtest reports compilation problems related to importing dependencies in the Maven test scope, try running the jtest:jtest goal with the mvn test command:

mvn test jtest:jtest

(info) By default, test sources are excluded from analysis. To analyze test code, disable the excludeTestSources option; see Jtest Goals Reference for Maven.

Executing and Collecting Coverage for Unit Tests

You can include unit test results in the Jtest report by running your tests with the jtest:jtest and jtest:agent goals, and the dedicated Unit Tests built-in test configuration:

  1. Ensure that the Jtest Plugin for Maven is set up (see Configuring the Jtest Plugin for Maven).
  2. Execute the jtest:agent and jtest:jtest goals with Maven. Ensure that the tests and classes are compiled before the jtest:agent goal is run.  Your command line may resemble the following:

    mvn clean test-compile jtest:agent test jtest:jtest -Djtest.config="builtin://Unit Tests"

By default, Jtest collects coverage data for executed tests. To disable collecting coverage, enable the jtest.coverage.skip option (see Jtest Goals Reference for Maven for details).

Executing the jtest:agent goal before the classes are compiled, may result in empty coverage. If your build uses Tycho plugins, or any other plugins that are detached from the default lifecycle, invoke the dedicated (non-default) compiler to ensure that the classes are properly compiled and coverage information is collected. Your command line may resemble the following:

mvn clean tycho-compiler:compile jtest:agent verify jtest:jtest -Djtest.config="builtin://Unit Tests"

(info) If you skip a test by calling a method from the org.junit.Assume (JUnit 4) or org.junit.jupiter.api.Assumptions (JUnit 5) class, Jtest collects coverage for code lines executed before the method is called. This coverage information is not associated with any test in the coverage report if the tests are executed with the JUnit 4 runner.

Jtest collects test coverage for Maven parallel builds, enabled with the -T option. For details, see https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3. For example:

mvn -T 1C clean test-compile jtest:agent test jtest:jtest -Djtest.config="builtin://Unit Tests"

Jtest supports collecting coverage for Surefire parallel test execution, but the coverage may not be correctly matched to each executed test.

About the jtest:agent Goal

The jtest:agent goal generates the Jtest agent settings based on the build modules, and sets the javaagent VM agument in the properties specified with the agentPropertyNames parameter in the initialize phase.

By default, it tries to inject javaagent VmArg into maven-surefire-plugin and eclipse-test-plugin through specific properties.

Prerequisites for using the jtest:agent goal with the Maven test plugins:

  • tests must be executed in the forked process (for example, with the parameters forkCount or forkMode)
  • additional VmArgs must contain Jtest javaagent VmArgs (the argLine parameter)

The jtest:agent goal attempts to automatically configure the Maven test plugins (maven-surefire-plugin and tycho-surefire-plugin) by setting up the properties of their VmArgs parameters (by default through the argLine and tycho.testArgLine properties). You can customize the default values of these parameters if one of the following ways:

  • Through the default property. The jtest:agent goal will extend the properties with the Jtest javaagent VmArg :

    <project>
      <!-- ... -->
      <properties>
        <!-- argLine property which will be extended by jtest:agent goal --->
        <argLine>-Xmx=1024m -XX:MaxPermSize=256m</argLine>
      </properties>
      <!-- ... -->
      <build>
        <plugins><!-- or pluginManagement -->
          <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
    		<configuration>
              <forkCount>1</forkCount>
              <!-- do not configure argLine parameter here -->
            </configuration>
          </plugin>
        </plugin>
      </build>
    </project>
  • Through a parameter value with an injected custom property:

    <project>
      <!-- ... -->
      <build>
        <plugins>
          <plugin>
            <groupId>com.parasoft.jtest</groupId>
            <artifactId>jtest-maven-plugin</artifactId>
            <version>2020.1.0</version>
    		<configuration>
              <!-- jtest:agent goal will set Jtest Java agent VmArg into properties below --> 
              <agentPropertyNames>jtest.argLine</agentPropertyNames>
              <!-- optional coverage block with parameters identical as in offline coverage -->
              <coverage>
                <!-- ... -->
              </coverage> 
            </configuration>
          </plugin>
          <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <argLine>${jtest.argLine} -Xmx=1024m -XX:MaxPermSize=256m</argLine>
            </configuration>
          </plugin>
        </plugin>
      </build>
    </project>

Executing Tests and Collecting Test Coverage for Tycho Builds

Jtest supports executing JUnit 4 tests and collecting test coverage for Tycho builds. However, due OSGi limitations, tests executed with the tycho-surefire-plugin require extended tycho-surefire configuration:

  1. Deploy the runtime.jar file shipped with Jtest in [INSTALL_DIR]/integration/coverage to one of the following:

    - The Maven repository. Your command line may resemble the following:

    mvn deploy:deploy-file -Dfile=C:\parasoft\jtest\integration\coverage\runtime.jar -DrepositoryId=REPOSITORY_ID -Durl=REPOSITORY_URL -DartifactId=runtime-jtest -DgroupId=com.parasoft.jtest -Dversion=2020.1.0

    - The local Maven repository. Your command line may resemble to following:

    mvn install:install-file -Dfile=<C:\parasoft\jtest\integration\coverage\runtime.jar -DgroupId=com.parasoft.jtest -DartifactId=runtime-jtest -Dversion=2020.1.0 -Dpackaging=jar
  2.  Extend  your tycho-surefire configuration by adding deployed the artifact as a framework extension. The configuration may resemble the following:

    <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>tycho-surefire-plugin</artifactId>
        <version>${tycho-version}</version>
            <configuration>
                <frameworkExtensions>
                    <frameworkExtension>
                        <groupId>com.parasoft.jtest</groupId>
                        <artifactId>runtime-jtest</artifactId>
                        <version>2020.1.0</version>
                    </frameworkExtension>
                </frameworkExtensions>
              ...
    </plugin>

Alternatively, if you execute test without collecting coverage information, you can configure Jtest to obtain test results using the deprecated XML processing mechanism by adding the following option in the jtestcli.properties configuration file:

jtest.unittest.xml.results.processing.enabled=true

This will enable Jtest to include test results for Tycho builds in the Jtest report. Coverage data will not be included.

Note that enabling the deprecated XML processing mechanism may slow down test execution.

Collecting Application Coverage

Jtest's coverage agent allows you to collect coverage data during manual or automated tests performed on a running application. See Application Coverage for information about collecting application coverage with Jtest.

Test Impact Analysis

Introduction

You can extend the capabilities of the Jtest Plugin for Maven with test impact analysis. It allows you to you to identify and re-run only the tests that are affected by your changes, eliminating the time and effort required to execute a large number of unaffected tests. To perform test impact analysis of your project, you need to:

  1. Add the test impact analysis plugin (tia-maven-plugin) shipped with Jtest to your Maven build.
  2. Configure the plugin.
  3. Execute the affected-tests goal.

Jtest does not support test impact analysis for Maven parallel builds.

Prerequisites

  • Jtest 10.4.1 or higher
  • Apache Maven 2.2.1 or higher (versions 3.3.1 and 3.3.3 are not supported)
  • Surefire 2.19.1 or higher
  • JUnit 4 or 5

(info) The Jtest Plugin for Maven must be configured to ensure access to the Maven repository shipped with Jtest; see Initial Setup.

(tick) Test impact analysis may require additional memory. We recommend increasing the memory allocated to the Maven build.

Integrating with the Test Impact Analysis Plugin

You can integrate with the test impact analysis plugin by adding tia-maven-plugin to the list of plugins in the POM file:

pom.xml
<build>
    <plugins>
        <plugin>
            <groupId>com.parasoft.jtest.tia</groupId>
            <artifactId>tia-maven-plugin</artifactId>
            <version>2020.1.0</version>
        </plugin>
    </plugins>
</build>

Configuring the Plugin

You can customize test impact analysis of your project in the POM file or in the command line by configuring tia-maven-plugin. At minimum, you must provide the paths to the following files that are generated by Jtest during execution:

  • coverage.xml  

  • report.xml

You must provide an absolute path to the coverage.xml file if one or more tests are in a different project than the tested source code to to ensure that all affected tests are re-run.

See Jtest Goals Reference for Maven for the complete list of available options.

In the POM file

If you configure the plugin in the POM file, you can add the tia-maven-plugin properties to:

  • plugin declaration

    pom.xml
    <plugin>
        <groupId>com.parasoft.jtest.tia</groupId>
        <artifactId>tia-maven-plugin</artifactId>
        <version>2020.1.0</version>
        <configuration>
            <referenceCoverageFile>target/jtest/coverage.xml</referenceCoverageFile>
            <referenceReportFile>target/jtest/report.xml</referenceReportFile>
            <runFailedTests>false</runFailedTests>
            <runModifiedTests>true</runModifiedTests>
            <jtestHome>${jtest.home}</jtestHome>
            <settings>jtestcli.properties</settings>
        </configuration>
    </plugin>
  • plugin execution

    pom.xml
    <plugin>
        <groupId>com.parasoft.jtest.tia</groupId>
        <artifactId>tia-maven-plugin</artifactId>
        <version>2020.1.0</version>
        <executions>
            <execution>
                <goals>
                    <goal>affected-tests</goal>
                </goals>
                <configuration>
                    <referenceCoverageFile>target/jtest/coverage.xml</referenceCoverageFile>
                    <referenceReportFile>target/jtest/report.xml</referenceReportFile>
                    <runFailedTests>false</runFailedTests>
                    <runModifiedTests>true</runModifiedTests>
                    <jtestHome>${jtest.home}</jtestHome>
                    <settings>jtestcli.properties</settings>
                </configuration>
            </execution>
        </executions>
    </plugin>
  • Maven properties

    pom.xml
    <properties>
        <jtest.referenceCoverageFile>target/jtest/coverage.xml</jtest.referenceCoverageFile>
        <jtest.referenceReportFile>target/jtest/report.xml</jtest.referenceReportFile>
        <jtest.runFailedTests>false</jtest.runFailedTests>
        <jtest.runModifiedTests>true</jtest.runModifiedTests>
        <jtest.home>${env.JTEST_HOME}</jtest.home>
        <jtest.settings>jtestcli.properties</jtest.settings>
    </properties>

In the Command Line

If you customize test impact analysis in the command line, pass the tia-maven-plugin properties with the -D switch. The properties must include the "jtest" prefix (see Jtest Goals Reference for Maven for details). Your command line may resemble the following:

Comand Line
mvn tia:affected-tests test -Djtest.referenceCoverageFile=target/jtest/coverage.xml -Djtest.referenceReportFile=target/jtest/report.xml -Djtest.runFailedTests=false -Djtest.runModifiedTests=true -Djtest.home=$JTEST_HOME -Djtest.settings=jtestcli.properties

Configuring and Executing the affected-tests Goal

You can configure execution of the affected-tests goal from the command line or by configuring the goal in the POM file.

In the Command Line

When tia-maven-plugin is included in the Maven build file (see Integrating with the Test Impact Analysis Plugin), you can execute the tia:affected-tests goal from the command line. Ensure it is executed before the test goal:

Command Line
mvn tia:affected-tests test

In the POM file

You can configure execution of the affected-tests goal in the build plugins or in the the build plugins of the profile.

  • If you include the affected-tests goal in the build plugins:

    pom.xml
    <build>
        <plugins>
            <plugin>
                <groupId>com.parasoft.jtest.tia</groupId>
                <artifactId>tia-maven-plugin</artifactId>
                <version>2020.1.0</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>affected-tests</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    it will be automatically executed with the Maven test goal:

    Command Line
    mvn test
  • If you include the affected-tests goal in the build plugins of the profile:

    pom.xml
    <profile>
        <id>tia</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>com.parasoft.jtest.tia</groupId>
                    <artifactId>tia-maven-plugin</artifactId>
                    <version>2020.1.0</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>affected-tests</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>

    you must execute the Maven test goal using the profile:

    Command Line
    mvn test -P tia

Re-executing Test Suites

Test impact analysis re-runs the entire test suite when at least one test included in that test suit is affected by code changes. Note that this may result in re-running some tests that are not affected if they are included in the same test suite as affected tests.

Re-executing Nested Tests

By default, the Maven surefire plugin is unable to re-run nested tests. To ensure that nested tests are re-executed when you run test impact analysis with Maven, add the <include></include> and <exclude></exclude> elements to the Maven surefire plugin configuration, for example:

 <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <testFailureIgnore>true</testFailureIgnore>
                        <includes>
                            <include></include>
                        </includes>
                        <excludes>
                            <exclude></exclude>
                        </excludes>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>

Manually Filtering Tests to Re-run

Test impact analysis relies on Surefire's default include/exclude mechanism, as well as on custom includes and excludes configured in the current POM file. In rare cases, test impact analysis may be unable to recognize the include or exclude pattern. This may result in failing to re-run affected tests that match the pattern and/or re-running tests that are not affected by code changes.

As a workaround, you can specify the file pattern in the command line with the -Dparasoft.testFilter option. Your command line may resemble the following:

mvn tia:affected-tests test -Djtest.referenceCoverageFile=target/jtest/coverage.xml -Djtest.referenceReportFile=target/jtest/report.xml -Djtest.settings=jtestcli.properties -Djtest.testFilter="**/*Test.java, !**/ProblemTest.java"

Note that Surefire's built-in -Dtest option for executing individual tests is not supported for test impact analysis with Jtest. Using the -Dtest option results in overriding the includes and excludes configured for test impact analysis in the POM file and in the command line.


  • No labels