The test impact analysis (TIA) functionality analyzes the coverage data for the application under test and generates a list of tests that have been affected by changes since the previous build. The list of tests are saved to a .lst file that you can pass as a resource to the SOAtest command line interface. SOAtest will only execute the subset of tests affected identified by TIA in order to validate the changes. The following overview describes the test impact analysis process:
- Configuration: Configure the coverage agent (agent.jar) shipped with SOAtest and , attach it to your application under test (AUT).Configure your Test Configuration to AUT, and enable SOAtest to communicate with the agent and configure settings for creating the baseline coverage report..
- Collect information about what test cases cover: Run your full test suite with the new configuration to collect information about what test cases cover and generate the baseline coverage report.suite so that the agent can collect data about the tests and the code they cover.
- Generate a baseline coverage report: Process the data collected by the agent to create a baseline report.
- Generate the .lst file containing the tests affected by changes. : When a new version of the application is available, run the
TestImpactAnalysisscript to process the baseline coverage report. A .lst file containing the tests affected by change will be created.
- Run the tests affected by change. : Deploy the latest version of the application (.war) to your server and run a job that executes SOAtest using the .lst file as the input to verify the changes.
TIA is intended to be implemented as part of an automated process. Perform the following steps to enable TIA.
Package your Application
Package your application under test into a deployable .war file.
Attaching the Coverage Agent to the AUT
You can also check the directory you specified with the runtimeData property (
/home/TIA/coverage_storage in the example above). The directory should contain a set of static coverage data files. The files are generated when the agent is started.
Creating the Test Configuration
Configure a Test Configuration to report You will need to create a test configuration in SOAtest that reports test execution information to the coverage agent and generate the baseline coverage report. You can configure an existing Test Configuration or create and configure a new one (see Creating a Custom Test Configuration).
. The test configuration only needs to be created once.
- Choose Parasoft> Test Configurations from the SOAtest menu and expand the Builtin category.
- Right-click Demo Configuration and choose Duplicate.
- Specify a new name for the configuration (i.e., TIA_agent) and click the Execution tab.
- Click the Application Coverage tab and specify the host name or IP address where the application under test and coverage agent are hosted and the port number of the agent. The port number should match the value of the
jtest.agent.portsetting in the agent.properties file (default is
8050). You can click
- Click Test Connection to verify that SOAtest can communicate with the coverage agent.
- (Optional) Under Coverage agent user ID, you can specify a user ID so that coverage results can be associated with a specific user. A user ID should only be specified when application coverage for multiple users is enabled for the coverage agent.
- Enable the Generate baseline coverage report for test impact analysis option and specify the following:
- Application binaries location - The location that contains binaries of the application under test. You can specify the path to a folder or a .war, .jar, .zip, or .ear file.
- Report location - The output directory of the baseline coverage report.
- Enable the
jtest.agent.enableMultiuserCoverageproperty is set to
truein the agent.properties file. We recommend specifying agent user IDs when multiple users or automated test runs are configured to access the same instance of the AUT at the same time. This is so that their interactions do not affect the correlation between tests and code covered for other users.
- If you want test failures reported when the coverage agent connection fails, enable the Report coverage agent connection failures as test failures option for test failures to be reported when the coverage agent connection fails, or when the application binaries location or report location are misconfigured. Otherwise connection problems will be reported to the console, but will not cause the test to fail.
- Enable the Retrieve coverage data option and specify a directory in which to locally store the runtime coverage data. The data saved to this directory is analyzed to generate the baseline coverage report.
- Click Apply to save your changes.
Run your full test suite using the new Test Configuration to collect the data and generate the baseline coverage report.test configuration. If you already have an automated test run, you can modify the existing job to use the new Test Configurationtest configuration, e.g.:
soatestcli.exe -data <your workspace> -resource <your tests> -localsettings <properties file with SOAtest settings> -config <new test configuration>
See Testing from the Command Line Interface - soatestcli for details about building test execution commands with SOAtest. You can also manually run tests from the SOAtest GUI.
|title||TEMP Directory for Linux/MacOS|
Generating Baseline Coverage
SOAtest includes a script for creating the baseline coverage report
. The script analyzes the coverage data collected by the full test run and the WAR file for the AUT to generate the report. Depending on your operating system, you can run either the TestImpactAnalysis.bat (Windows) or TestImpactAnalysis.sh (Linux/macOS) script located in the <INSTALL>/test_impact_analysis directory.
You may need to modify the .sh script and specify a temporary directory , for exampleif the variable is not already set, e.g.:
When the test finishes, run the TestImpactAnalysis.sh or TestImpactAnalysis.bat script using the following syntax to generate the coverage report.
./TestImpactAnalysis.sh -app <PATH TO AUT WAR FILE> -runtimeCoverage <PATH_TO_RUNTIME_COVERAGE> -outputReport <PATH_TO_OUTPUT_REPORT_DIR> -include <SPACE_SEPARATED_PATTERNS> -exclude <SPACE_SEPARATED_PATTERNS>
Generating List of Tests Affected by Change
- flag specifies the deployable .war. You should specify the same artifact deployed as the AUT for the automated test run (see Package your Application).
-runtimeCoverageflag specifies the directory you configured in the test configuration.
-outputReportflag is optional and specifies where to output the coverage report.
-includeflag is optional and specifies a space-separated list of patterns to include in the analysis. By default, all test classes are included.
-excludeflag is optional and specifies a space-separated list of patterns to exclude in the analysis. Classes that match patterns specified with the
-includeflag override excluded patterns.
- Add the
-keepRuntimeCoverageoption if you want to keep the runtime coverage data files used to generate the baseline coverage XML report. This This parameter does not take a value and is primarily used for debugging purposes. If it is not included in the command, the runtime coverage data files will be removed at the end of the execution.
TestImpactAnalysis.bat ^ -app C:\tomcat\webapps\application.war ^ -runtimeCoverage C:\TIA\runtime-coverage ^ -outputReport C:\TIA\reports ^ -include com\myapp\data\**,com\myapp\common\** ^ -exclude com\myapp\transport\**,com\myapp\autogen\**
./TestImpactAnalysis.sh \ -app /home/tomcat/webapps/application.war \ -runtimeCoverage /home/TIA/runtime-coverage \ -outputReport /home/TIA/reports \ -include com/myapp/data/**,com/myapp/common/** \ -exclude com/myapp/transport/**,com/myapp/autogen/**
When the script finishes, the coverage.xml report will be saved to the location specified with the
-outputReport flag. If the flag is not included, the file will be saved to the <USER_HOME>/parasoft/test_impact_analysis/reports/ directory by default.
Generating List of Tests Affected by Change
When a new version of the application is available, run the TestImpactAnalysis.sh or TestImpactAnalysis.bat script using the following syntax:
TestImpactAnalysis.sh -app <path to new .war><PATH_TO_NEW_WAR> -coverageReport <path to coverage.xml report> -outputLst <path to .lst><PATH_TO_COVERAGE_XML_REPORT> -outputLst <PATH_TO_LST>
-appflag specifies the new deployable .war (see Package your Application).
-coverageReportflag flag should specify the baseline coverage.xml report generated in the directory configured in your Test Configuration with the Report location option (see Configuring the Test Configuration). Generating Baseline Coverage.
-outputLstflag is optional and specifies where to output the results.