Introduction

dotTEST allows you to monitor and collect coverage data during manual or automated functional tests performed on a standalone or web application. You can also send coverage data and test results to DTP, which merges and correlates the data. The application coverage information can be displayed in the DTP Coverage Explorer (see the "Coverage Explorer" chapter in the DTP user manual), which provides insights about how well the application is tested, as well as the quality of your tests. 

Process Overview

dotTEST ships with a component called the coverage agent, which makes it possible to collect coverage for applications. Coverage for a standalone application can be collected with or without REST client for coverage engine (agent_client.exe). Coverage for a web application should be collected using the REST client.

Metadata about the lines of code that can be covered (static coverage data) is collected either by running a dedicated test configuration or by scanning the application assemblies. During test execution, interactions with the coverage agent are written to dynamic coverage data files, which contains markers that specify which lines of code were touched.  

dotTEST processes the dynamic coverage and static coverage data. A coverage.xml file, which contains the coverage information, is produced and sent to DTP. When DTP receives the coverage data, it is loaded into a coverage image, which is a special tag that enables you to aggregate coverage data from runs with the same build ID. The coverage image enables you to associate coverage information with specific tests. 

dotTEST can also collect partial application coverage, which includes only runtime coverage and general information about the IL that have been executed without any details about the source code structure other that the lines that have been covered. A simplified report is then sent and appended to existing coverage data on DTP. The partial coverage data can be viewed if information about source code is already available on DTP. 

Test results are also sent to DTP from the tool executing the tests (i.e., SOAtest, manual tests, etc.) in a report.xml file. If the build IDs for the coverage data file and the report match, DTP is able to correlate the data and display the coverage information. 

You can use either dottestcov or the dottestcli to create a coverage report and upload the data to DTP. dottesctcov is a dedicated tool for collecting coverage and offers more options in terms of creating coverage reports, see Creating a Coverage Report and Uploading the Data to DTP. The dottestcov.bat file is located in the <INSTALL_DIR>/integration/coverage/dottestcov directory. dottestcov is a standalone tool that can be copied out from dotTEST installation directory and used independently. Therefore, you can easily integrate it within your testing flow. For details about dottestcov command line options, see Command Line Options for dotestcov.

Prerequisites for using the Coverage Agent

Ensure that the Visual C++ Redistributable Packages that are shipped with dotTEST are installed on the machine where the monitored application will be run. They are automatically installed when you install dotTEST with the wizard (see Installation). If you install dotTEST from a ZIP distribution, or if you are going to collect coverage on another machine, install the packages manually by running the executables located in the [INSTALL_DIR]\bin\prerequisites directory.

The monitored application can be installed on the same machine where dotTEST is installed, or on another machine.

Additionally, collecting coverage from web applications requires the LoadUserProfile option to be set to true in the ISS Management Console.

Prerequisites for using the dottestcov tool

Ensure the following are installed on your machine to be able to use the dottestcov tool:

• Java 11 Runtime
• .NET Framework 4.7.2
• Visual C++ Redistributable Packages

Collecting Coverage

For details on collecting coverage for a standalone application, see Application Coverage for Standalone Applications.

For details on collecting coverage for a web application, see Application Coverage for Web Applications.

Generating the Static Coverage File with Source Code Information

Generating the static coverage file is required if you want create to a report containing merged static coverage (including source code information) and runtime coverage.

Generate the static coverage file by running the following test configuration on the solution:

dottestcli.exe -config "builtin://Collect Static Coverage" -solution SOLUTION_PATH

The dottestcli console output indicates where the static coverage data is saved, for example:

Saving static coverage information into: 'C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\[FILE].xml

This static coverage file can be used with dottestcli and dottestcov to be merged with runtime coverage data and sent to DTP. 

Generating the Static Coverage File without Source Code Information

Generate the static coverage file by running the following command:

dottestcov.bat -app [dir] 

Where -app specifies the path to the Application Under Test directory.

Uploading Test Results to DTP

If you use dotTEST with CAM:

1. Go to Report Center in the DTP interface.
2. Click the gear icon and choose Report Center Settings> Additional Settings> Report Center Administration> Tools> Data Collector Upload Form (requires admin permissions).
3. Click Choose File and browse for the report.xml file you downloaded from CAM.
4. Click the Upload button to upload the file to DTP.

If you use SOAtest, test results are automatically sent to DTP during test execution. See the Application Coverage section in the SOAtest user guide at https://docs.parasoft.com.

Creating a Coverage Report and Uploading the Data to DTP

You can use either the dottestcov or the dottestcli command to create a coverage report and upload the data to DTP. For details about dottestcov command line options, see Command Line Options for dotestcov.

If you use dotTEST with CAM, there are 3 types of coverage report that can be generated:

1. A static and runtime coverage report: a report containing merged static coverage (including source code information) and runtime coverage. To create this report, run dottestcli with the -runtimeCoverage and -staticCoverage switches or dottestcov with the -runtime and -static switches.
2. An application coverage report: a static coverage report created by scanning application assemblies and runtime coverage. To create this report, run dottestcov with the -app and -runtime switches.
3. A runtime coverage report: a simplified report containing partial application coverage, which includes only runtime coverage and general information about the classes that have been executed without any details about the class structure (only the lines that have been covered). The report can be viewed if there is already existing application or static coverage-based data on DTP. To create this report, run dottestcov with the -runtime switch.

Follow the procedure below to create a coverage report and upload the data to DTP.

1. Ensure that dottestcli/dottestcov is properly configured, including DTP, scope, and authorship settings. See Connecting to DTP, Sending Results and Publishing Source Code to DTP, Configuration.
2. Configure the following settings in the .properties file in order to properly merge coverage data:
   - report.coverage.images -  Specifies a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. DTP supports up to three coverage images per report.
   - session.tag - Specifies a unique identifier for the test run and is used to distinguish different runs on the same build.
   - build.id - Specifies a build identifier used to label results. It may be unique for each build, but it may also label several test sessions executed during a specified build.
   - report.coverage.limit (optional) - Specifies the lower coverage threshold. Coverage results lower than this value are highlighted in the report (the default value is 40).
3. If you want to use dottestcli, copy the runtime coverage and static coverage files to the same machine and run dottestcli with the following switches:
   
   • -runtimeCoverage: Specifies the path to runtime coverage that you download with CAM (see Coverage Agent Manager (CAM) section of the DTP documentation for details). You can provide a path to an individual .data file with coverage information from one testing session, or a path to a folder that contains many .data files from multiple testing sessions.
   • -staticCoverage: Specifies the path to the static coverage file (see Generating the Static Coverage File with Source Code Information).
   • -config: Specifies the URL of the Calculate Application Coverage test configuration that must be run to merge the coverage data.

   • -publish: Sends the merged coverage to DTP.

     dottestcli.exe -runtimeCoverage [path] -staticCoverage [path] -publish 

     This ensures that dotTEST has access to the runtime coverage data generated during test execution, as well as the static coverage data, which is required to fill the coverage.xml file with runtime coverage data.

4. If you want to use dottestcov, run dottestcov with the following switches:
   • -app: Specifies the path to the Application Under Test directory or archive file.
   • -runtime: Specifies the path to runtime coverage that you download with CAM (see Coverage Agent Manager (CAM) section of the DTP documentation for details). You can provide a path to an individual .data file with coverage information from one testing session, or a path to a folder that contains many .data files from multiple testing sessions.
   • -static: Specifies the path to the static coverage file (see Generating the Static Coverage File with Source Code Information).

   • -publish: Sends the merged coverage to DTP.

     Example 1: Generating a report based on runtime data and static coverage from the monitor

     dottestcov.bat -static [path to static_coverage.xml file] -runtime [path/dir]
     -publish -settings [path to .properties]

     Example 2: Generating a report based on runtime data and coverage from the application jars

     dottestcov.bat -app [dir] -runtime [path/dir] -publish -settings [path to .properties]

      Example 3: Generating a report based on runtime data

     dottestcov.bat -runtime [path/dir] -publish -settings [path to .properties]

5. If you want to merge previously generated partial coverage reports, run the dottestcov merge command. To merge two or more reports, run

   dottestcov.bat merge -coverage <REPORT_FILE> -coverage <REPORT_FILE>
   [-coverage <REPORT_FILE>] [-report <REPORT_DIR>]

   • -coverage <REPORT_FILE>: Specifies one or more input coverage reports to merge and publish to DTP.
   • -report <REPORT_DIR>: (Optional) Generates an xml coverage report to the given folder. By default the report is stored to the .coverage/reports folder.

If you use SOAtest, static and runtime coverage data are merged and sent to DTP during test execution. See the Application Coverage section in the SOAtest user guide at https://docs.parasoft.com.

Customizing Scope of Coverage using dottestcli

By default, coverage is measured for the entire application. To narrow down the scope of collecting coverage, provide paths to the selected parts of the application with the -solution, -resource, -include, and/or -exclude options (see Configuring the Test Scope for details) when generating static coverage. As a result, dotTEST will generate a scope configuration file that would include the customized scope for collecting static coverage.

You do not need to modify the content of the scope configuration file. However, if you collect coverage for large applications, you may want to additionally fine-tune the scope of collecting runtime coverage (i.e. coverage collected while interacting with the application under test) to optimize performance. To customize collecting runtime coverage, open the scope.instumentation.txt file and remove the the assemblies for which you do not want to collect coverage data from the list of assemblies.

Customizing the Coverage Scope with the Coverage Agent REST Client

1. Add the paths to resources to your command line (see Generating the Static Coverage File with Source Code Information), for example:

   dottestcli.exe -config "builtin://Collect Static Coverage" 
   -solution "C:\Devel\FooSolution\FooSolution.sln" 
   -resource "FooSolution/QuxProject" 
   -include "C:\Devel\FooSolution\src\QuxProject\**\*.cs" 
   -exclude "C:\Devel\FooSolution\src\QuxProject\**\tests\**\*.cs"

   When static coverage is generated, the scope information is stored in a scope configuration file. The output from the console indicates the location of the scope configuration file, for example:

   Saving static coverage scope configuration into: 'C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\scope.instrumentation.txt'

2. For a standalone application:
   1. Open the runCamAgent.bat script (see Collecting Coverage with Coverage Agent REST Client).

   2. Add the path to the scope configuration file to the script using the -scope option, for example:

      -scope 'C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\scope.instrumentation.txt

3. For a web application: pass the path to the scope configuration file to the Agent Client tool (see Attaching the Coverage Agent to the AUT) with the -scope option, for example:

   agent_client.exe -scope 'C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\scope.instrumentation.txt'

Customizing the Coverage Scope without the Coverage Agent REST Client

To customize the scope for collecting coverage without the coverage agent REST client, you need to manually generate static coverage before you start interacting with the application.

1. Generate the static coverage file by running the built-in Collect Static Coverage test configuration and providing paths to the selected parts of the application with the -solution, -resource, -include, and/or -exclude options (see Configuring the Test Scope for details). Your command line may resemble the following:

   dottestcli.exe -config "builtin://Collect Static Coverage" 
   -solution "C:\Devel\FooSolution\FooSolution.sln" 
   -resource "FooSolution/QuxProject" 
   -include "C:\Devel\FooSolution\src\QuxProject\**\*.cs" 
   -exclude "C:\Devel\FooSolution\src\QuxProject\**\tests\**\*.cs"

   When static coverage is generated, the scope information is stored in a scope configuration file. The output from the console indicates the location of the scope configuration file, for example:

   Saving static coverage scope configuration into: 'C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\scope.instrumentation.txt'

2. Open the monitorCoverage.bat script and configure the PARASOFT_PROFILING_ASM_LIST_PATH environment variable that points to the location of the scope configuration file, for example:

   SET PARASOFT_PROFILING_ASM_LIST_PATH=C:\Users\[USER]\Documents\Parasoft\dotTEST\Coverage\Static\scope.instrumentation.txt

3. Run the monitorCoverage.bat script to run the application.
4. Interact with the application and perform your tests, then close the application.

5. Run dottestcli or dottestcov to create a coverage report and uploading the data to DTP. See steps 3 and 4 in Creating a Coverage Report and Uploading the Data to DTP.

Customizing Scope of Coverage using dottestcov

By default, coverage is measured for the entire application. To narrow down the scope of collecting coverage,

dottestcov.bat 
-include "C:\Devel\FooSolution\src\QuxProject\**\*.cs" 
-exclude "C:\Devel\FooSolution\src\QuxProject\**\tests\**\*.cs"
-app [dir] -runtime [path/dir] -publish -settings [path to .properties]

Reviewing Coverage in DTP

You can use the Coverage Explorer in DTP to review the application coverage achieved during test execution. For details on viewing coverage information, see the DTP user guide at https://docs.parasoft.com.

Command Line Options for dottestcov

This section lists the command line options for dottestcov.

coverage

Default command. Scans application and runtime coverage data in order to generate a coverage report. To generate a report, run:

dottestcov.bat coverage [-app <APP_LOCATION> | -static <BASE_FILE>]
-runtime <RUNTIME_COVERAGE_LOCATION> [-report <REPORT_DIR>] [-deleteruntime] 

-app <APP_LOCATION>

Specifies the location of the Application Under Test (AUT) directory.

-static <BASE_COVERAGE_REPORT_FILE> or <BASE_METADATA_FILE>

Specifies the XML coverage report or metadata file containing the coverage base of the AUT.

-runtime <RUNTIME_COVERAGE_LOCATION>

Specifies the location of the runtime coverage directory generated during test execution against the AUT.

-report <REPORT_DIR>

Generates an XML coverage report to the given folder. Optional parameter; by default the report is stored to the .coverage/reports folder.

-deleteruntime

Deletes the runtime coverage data files used to generate the coverage XML report.
If this parameter is present, the runtime coverage data files will be deleted after the report is created.

impacted

Scans the application and analyzes the baseline coverage report to find tests that need rerunning. In order to generate an LST file for the impacted tests, run: 

dottestcov.bat impacted -app <APP_LOCATION>
-baseline <BASELINE_COVERAGE_REPORT FILE> [-report <LST_FILE_DIR>]

-app <APP_LOCATION>

Specifies the location of the Application Under Test (AUT) directory.

-baseline <BASELINE_COVERAGE_REPORT_FILE>

Specifies the XML coverage report to use as the baseline.

-report <LST_FILE_DIR>

Generates an LST file in a given location. If an LST file already exists, it will be overwritten. Optional parameter; by default the LST file is stored to the .coverage/lsts folder.

merge

Merges previously generated partial coverage reports. In order to merge two or more reports, run:

dottestcov.bat merge -coverage <REPORT_FILE> -coverage <REPORT_FILE>
[-coverage <REPORT_FILE>] [-report <REPORT_DIR>]

-coverage <REPORT_FILE>

Specifies one or more input coverage XML reports to merge and publish to DTP.

-report <REPORT_DIR>

Generates an XML coverage report to the given folder. Optional parameter; by default the report is stored to the .coverage/reports folder.

help

Shows the dottestcov help.

Other arguments:

-include <PATTERN>
-exclude <PATTERN>

Specifies patterns to include/exclude elements during AUT scanning. Optional parameter; by default all elements are accepted.

-publish

Uploads the generated reports to the DTP Server. (Settings pattern: report.dtp.publish=true)

-settings <SETTINGS_FILE>

Reads the custom configuration from a file. <SETTINGS_FILE> should be an existing configuration file with <KEY=VALUE> entries (.properties).

-property

Specifies a custom configuration entry. You can use -property multiple times (if more than one entry includes the same key, the entry specified as first will overwrite all the following entries).

-showdetails

Increases output verbosity.
(Settings pattern: console.verbosity.level=high)

dottestcov Exit Codes

The process may exit with one of the following exit codes: