The test impact analysis (TIA) functionality analyzes the coverage data for your application under test and outputs a coverage report that identifies which tests cover which code in the application. You can process the coverage report using the Selenic plug-in for Maven and run the subset of your Selenium tests that have been affected by changes to the application. In this section:
Test impact analysis is intended to be implemented as part of an automated process. The following overview describes the test impact analysis process:
- Configuration: Configure the coverage agent (agent.jar) shipped with Selenic and attach it to your AUT. Integrate the Selenic plug-in for Maven into your project.
- Collect information about what test cases cover: Run your Selenium tests so that the agent can collect data about the tests and the code they cover.
- Generate a baseline coverage report: Run the TestImpactAnalysis script to process the data collected by the agent to create a baseline coverage report.
- Run the tests affected by change: When a new version of the application is available, deploy it to the server and run your Selenium tests using the Selenic plug-in for Maven. Specify the baseline coverage file during execution and only the tests affected by change will run.
TIA is currently only supported for Java 8 - 12. See System Requirements for additional information.
Perform the following steps to enable TIA.
Package your Application
Package your application under test into a deployable .war file.
Configuring the Selenic Plug-in for Maven
Add the plug-in to your project pom.xml file. The following example shows how to add the dependency for Selenic version 2020.1, but you should use the current version in your build file.
Add the s
elenic-maven-plugin profile and repository to your Maven settings file. The settings may be configured globally or locally. By default, the global settings.xml file is in the <MAVEN_INSTALL>/conf/ directory. The local settings.xml file is in your <USER_HOME>/.m2 directory. If the local file does not exist, copy the settings.xml file from the global directory to the local directory and add the following
<profiles> element to the <
Configuring the TIA_OPTS Environment Variable
The Selenic plug-in for Maven includes an environment variable for passing parameters to the Java VM when executing the tests identified by test impact analysis (see Executing Tests Affected by Change). Add the
TIA_OPTS variable to the machine running Maven and specify any JVM parameters you want to include. For example, you could increase the available memory when running TIA to 2GB:
Attaching the Coverage Agent to the AUT
Selenic includes a Java agent that generates the coverage information necessary to determine which tests are affected by changes.
The agent is shipped in the <INSTALL>/test_impact_analysis/integration/coverage directory. It takes configuration settings from the agent.properties file in the same directory. You should copy the coverage directory that contains the agent.jar and agent.properties files to the machine where the AUT is running.
Configuring the Coverage Agent
Application servers usually contain more than one application. Additionally, common server classes or application libraries do not need to be instrumented. The agent only needs to collect coverage for application source code. Instrumenting all classes would be too time-consuming. For this reason, properly setting the scope of the coverage agent is very important.
You can configure the coverage agent by modifying the properties in the agent.properties and passing the properties to the
-javaagent argument. The agent supports several parameters (see Coverage Agent Parameters), but configuring the following settings is suitable for most cases:
Enabling Multi-session Mode
You can enable multi-session mode if multiple users or test processes access the application while the tests are running. This is so that code is executed by the automated tests and code executed by other users or test processes are distinguishable. In multi-session mode, a user ID is associated with the coverage data associated with distinct users or processes. The user ID is specified with the
coverageAgentUserID argument during execution (see coverageAgentUserID for details).
Add the following option to the agent.properties file to enable multi-session mode:
See Collecting Test and Coverage Data in Multi-session Mode for additional details.
Coverage Agent Parameters
The following table describes all properties that can be set for the agent:
Enables/disables associating coverage with particular tests; the default value is false.
Specifies a location on the application server for the agent to store the coverage data it collects at runtime.
A comma-separated list of patterns that specify classes to be instrumented. The following wildcards are supported:
* matches zero or more characters
** matches multiple directory levels
A comma-separated list of patterns that specify classes to be excluded from instrumentation. The following wildcards are supported:
* matches zero or more characters
** matches multiple directory levels
Enables/disables automatic runtime data collection; the default is
Sets up agent communication port; the default is
Enables/disables verbose output to console; the default is
Enables/disables collecting coverage information for test cases. The default value is
Enables/disables collecting web application coverage for multiple users; the default value is
Activates the agent.
When the properties are configured, add a
-javaagent argument when starting your application server to attach the agent and include the agent configuration file:
For your convenience, the coverage directory includes a script that will generate the
-javaagent arguments. Run either the agent.sh or agent.bat script and copy the output to your server startup script.
In the following example, the agent is attached to a Tomcat server with a JAVA_OPTS variable at the beginning of the catalina.sh (Linux) or catalina.bat (Windows) scripts:
Verifying that the Agent is Configured
Start the application and verify that the agent is ready by opening
<host>:8050/status in your browser. You should see a JSON object that contains test, session, and testCase properties, e.g.:
You can also check the directory you specified with the runtimeData property (/home/TIA/coverage_storage in the example above). The directory should contain a set of coverage data files. The files are generated when the agent is started.
Collecting Test and Coverage Data
Run your Selenium tests using the Selenic agent. The Selenic agent should be configured to collect coverage and to connect to the host and port where the AUT and coverage agent are running. You should also specify a local directory for downloading the coverage information from the server:
You can also configure the agent to save coverage data after test execution. The following table describes all parameters related to test impact analysis usage:
|Enables the agent to collect coverage.|
Specifies the host where the AUT (.war file) and coverage agent are running.
Specifies the coverage agent's port.
Specifies a user ID so that different users can be distinguished when collecting coverage in multi-session mode.
Specifies a local directory for the coverage data to be downloaded from the server.
Delete coverage data from server.
Refer to the Command Line chapter for additional information about Selenic agent command line arguments.
Collecting Test and Coverage Data in Multi-session Mode
Enable multi-session mode if multiple users or test processes are concurrently accessing the application during Selenium test execution. Multi-session mode enables you to identify the that comes from the Selenium tests being executed. Perform the following actions to run in multi-session mode:
- Enable multi-session mode in the agent.properties file (see Enabling Multi-session Mode).
- Start the SOAtest Web Proxy Server before running the agent to collect coverage during full test runs. The SOAtest Web Proxy Server is deployed as part of the Parasoft Recorder installation. See Installing Parasoft Recorder for additional information.
- Specify the user ID using the
coverageAgentUserIDargument during execution. Example:
Generating Baseline Coverage
Selenic includes a script for creating the baseline coverage report. The script analyzes the coverage data collected by the Selenium test run and the WAR file for the AUT to generate the report. Depending on your operating system, you can run either the TestImpactAnalysis.bat (Windows) or TestImpactAnalysis.sh (Linux/macOS) script located in the <INSTALL>/test_impact_analysis directory.
TEMP Directory for Linux/MacOS
You may need to modify the .sh script and specify a temporary directory if the variable is not already set, e.g.:
After the test run finishes, run the TestImpactAnalysis.sh or TestImpactAnalysis.bat script using the following syntax to generate the coverage report.
-appflag specifies the deployable .war. You should specify the same artifact deployed as the AUT for the automated test run (see Package your Application).
-runtimeCoverageflag specifies the directory specified with the
coverageDirparameter you configured in the
-javaagentarguments when executing your tests (see Collecting Test and Coverage Data).
-outputReportflag is optional and specifies where to output the coverage report.
-includeflag is optional and specifies a space-separated list of patterns to include in the analysis. By default, all test classes are included.
-excludeflag is optional and specifies a space-separated list of patterns to exclude in the analysis. Classes that match patterns specified with the -include flag override excluded patterns.
- Add the
-keepRuntimeCoverageoption if you want to keep the runtime coverage data files used to generate the baseline coverage XML report. This parameter does not take a value. If it is not included in the command, the runtime coverage data files will be removed at the end of the execution.
When the script finishes, the coverage.xml report will be saved to the location specified with the
-outputReport flag. If the flag is not included, the file will be saved to the <USER_HOME>/parasoft/test_impact_analysis/reports/ directory by default.
Executing Tests Affected by Change
When a new version of the application is available, you can execute your tests with the Selenic plug-in for Maven and specify the coverage file, deployed WAR file, and path to the Selenic home directory to execute only the tests reported by TIA:
Any JVM arguments you configured with the TIA_OPTS will also be used (see Configuring the TIA_OPT Environment Variable).
Test Impact Analysis Video Tutorial