This topic provides information about how to use SOAtest as part of the Parasoft Application Coverage workflow. For complete information about using the Application Coverage solution, refer to the Collecting Coverage for Web Applications guide. In this section:

Introduction

The Parasoft Application Coverage solution enables you to measure application code coverage while executing tests against your running applications. You can collect coverage from a single running application or from an application that is made up of multiple backend services (you can collect coverage from all of them during execution of the same test cases). This enables you to gauge the effectiveness of your existing test suite, determine where additional tests need to be added, and more efficiently execute tests as the application evolves. You can collect application coverage by utilizing the coverage agent shipped with SOAtest for Java or dotNET, which allows you to monitor the application as tests are being performed.

Application coverage is obtained by collecting and uploading static and dynamic coverage data to DTP for viewing.

Static coverage data is a description of all source code that is available in your application.
- Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with SOAtest to analyze the application binaries.
- Static coverage files are uploaded to DTP by the tool that created them (that is, Jtest/dotTEST or the coverage tool).
Dynamic (runtime) data is coverage data that is collected by the coverage agents as tests execute. It measures which code is covered as tests execute.
- Runtime coverage data is collected and uploaded to DTP by SOAtest.
Data is merged on DTP automatically.

The following steps describe the Application Coverage workflow with SOAtest:

Generating the static coverage file. When you have access to the application source code, the static coverage file should be generated by Jtest or dotTEST in .xml or .data format as this method comes with several advantages over the alternative. You can also generate the static coverage file by analyzing the application binaries using the coverage tool shipped with SOAtest when you do not have access to the application source code. See Generating the Static Coverage File below for more information.
Attaching the coverage agent to the application under test (AUT). The coverage agent ships with SOAtest and allows you to monitor the code being executed when the AUT is running.
Configuring SOAtest to connect to the coverage agent and collect runtime coverage and upload the data to DTP.
Executing tests with SOAtest as part of an automated process.
Reviewing the application coverage on DTP in the Coverage Explorer.

Coverage reports produced by SOAtest 2023.1 that are uploaded to DTP need to be sent to DTP 2023.1. Earlier versions of DTP will not properly process coverage reports produced by SOAtest 2023.1.

Generating the Static Coverage File

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. See Jtest or dotTEST user guide at docs.parasoft.com for more information on generating static coverage files with those tools.

You can also generate the static coverage file using the coverage tool shipped with SOAtest, though it lacks the advantages of using the source code described above. To do so, you need to run the tool independently of SOAtest, passing the binary, filter, license, and other properties when you do (see the sample license properties file below). An example using the Jtest coverage tool (jtestcov) against Parabank is shown below:

java -jar jtestcov.jar -soatest -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties

The -soatest flag is used by the coverage tools to pass your SOAtest license, which is needed when you run the coverage tool independently of SOAtest as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched. You can see all available commands and options by running: java -jar jtestcov.jar help

Sample License Properties File

# === LICENSE ===

# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
#parasoft.eula.accepted=false

# === LOCAL LICENSE ===
# Enables local license - be sure to specify password.
#jtest.license.use_network=false

# Specifies password for the local license. 
#jtest.license.local.password=[LICENSE PASSWORD]

# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
#jtest.license.use_network=true

# Specifies type of the network license (edition).
# Supported editions: desktop_edition, desktop_compliance_edition, server_edition, server_compliance_edition
#jtest.license.network.edition=server_edition

# Enables specific list of license features
#jtest.license.network.edition=custom_edition
#jtest.license.custom_edition_features=Jtest, Automation, Desktop Command Line, DTP Publish, Coverage, Test Impact Analysis


# === DTP SERVER SETTINGS ===

# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443

# Specifies user name for DTP server authentication.
#dtp.user=admin

# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin

# Specifies name of the DTP project - this settings is optional. 
#dtp.project=[DTP Project Name]

# === DTP REPORTING ===

# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true

# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd

# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]

# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test

# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to high.
#console.verbosity.level=high

Configuring the Application Under Test for Coverage

Before you start collecting coverage with SOAtest, you need to configure the application under test (AUT) using Parasoft dotTEST or Parasoft Jtest. The process is outlined below; for more detailed instructions, see the Jtest or dotTEST user guide at docs.parasoft.com. Ensure that your SOAtest, dotTest, and JTest versions are in sync to avoid unexpected issues.

(Optional) Customize the coverage agent by configuring the coverage agent options. This step is required if you are collecting coverage information for multiple users that are simultaneously interacting with the AUT.
Attach the coverage agent to the AUT.

Configuring the Coverage Agent

Application servers usually contain more than one application. Additionally, common server classes or application libraries do not need to be instrumented. The agent only needs to collect coverage for application source code. Instrumenting all classes would be too time consuming. For this reason, properly setting the scope of the coverage agent is very important. There are coverage agents for Jtest and dotTEST.

Jtest Coverage Agent

You can configure the Jtest coverage agent by modifying the properties in the agent.properties and passing the properties to the -javaagent argument. The agent supports several parameters (see Coverage Agent Parameters), but configuring the default settings is suitable for most cases:

jtest.agent.serverEnabled=true
jtest.agent.includes=com/myapp/data,com/myapp/common/**
jtest.agent.excludes=com/myapp/transport/*,com/myapp/autogen/**

Coverage Agent Parameters

The following table describes all properties that can be set for the agent:

`jtest.agent.runtimeData`	Specifies a location on the application server for the agent to store the coverage data it collects at runtime.
`jtest.agent.includes`	A comma-separated list of patterns that specify classes to be instrumented. The following wildcards are supported: `` matches zero or more characters `` matches multiple directory levels In the following example, all classes from the `com.myapp.data` package and all classes from package and subpackages that start with `com.myapp.common` will be instrumented: `com/myapp/data/,com/myapp/common/**`
`jtest.agent.excludes`	A comma-separated list of patterns that specify classes to be excluded from instrumentation. The following wildcards are supported: `` matches zero or more characters `` matches multiple directory levels In the following example, all classes from the `com.myapp.transport` package and all classes from package and subpackages that start with `com.myapp.autogen` will be excluded from instrumentation: `com/myapp/transport/,com/myapp/autogen/**`
`jtest.agent.autostart`	Enables/disables automatic runtime data collection; the default is `true`.
`jtest.agent.port`	Sets up agent communication port; the default is `8050`.
`jtest.agent.debug`	Enables/disables verbose output to console; the default is `false`.
`jtest.agent.enableMultiuserCoverage`	Enables/disables collecting web application coverage for multiple users; the default is `false.`
`jtest.agent.serverEnabled`	Activates the agent.
`jtest.agent.enableJacoco`	Enables the agent to collect coverage using the JaCoCo engine; the default is `false.`

When the properties are configured, add a -javaagent argument when starting your application server to attach the agent and include the agent configuration file:

-javaagent:'/path/to/agent.jar'=settings='/path/to/agent.properties',runtimeData='/path/to/runtime_coverage'

For your convenience, the coverage directory includes a script that will generate the -javaagent arguments. Run either the agent.sh or agent.bat script and copy the output to your server startup script.

$ ./agent.sh 
Add this Java VM args to your process:
---------------------------------------------------
-javaagent:"/home/TIA/test_impact_analysis/integration/coverage/agent.jar"=settings="/home/TIA/test_impact_analysis/integration/coverage/agent.properties",runtimeData="/home/TIA/test_impact_analysis/integration/coverage/runtime_coverage"
---------------------------------------------------
Press any key to continue . . .

In the following example, the agent is attached to a Tomcat server with a JAVA_OPTS variable at the beginning of the catalina.sh (Linux) or catalina.bat (Windows) scripts:

Linux and macOS

if [ "$1" = "start" -o "$1" = "run" ]; then
JAVA_OPTS='-javaagent:"/home/TIA/test_impact_analysis/integration/coverage/agent.jar"=settings="/home/TIA/test_impact_analysis/integration/coverage/agent.properties",runtimeData="/home/TIA/coverage_storage"'
fi

Windows

if "%1"=="stop" goto skip_instrumentation
set JAVA_OPTS=-javaagent:"C:\TIA\test_impact_analysis\integration\coverage\agent.jar"=settings="C:\TIA\test_impact_analysis\integration\coverage\agent.properties",runtimeData="C:\TIA\coverage_storage"
:skip_instrumentation

Start the application and verify that the agent is ready by opening <host>:8050/status in your browser. You should see a JSON object that contains test, runtime_coverage, and testCase properties, for example:

{"test":null,"session":"runtime_coverage_20191008_1537_0","testCase":null}

You can also check the directory you specified with the runtimeData property (/home/TIA/coverage_storage in the example above). The directory should contain a set of static coverage data files. The files are generated when the agent is started.

dotTEST Coverage Agent

The dotTEST coverage agent can collect coverage for web applications and standalone applications. Each requires a different configuration.

Web Applications

The application under test (AUT) must be running on a machine using IIS version 7.5 or higher.

To attach the coverage agent to the AUT:

Copy the <INSTALL_DIR>/coverage/dotNET/dottest_agent folder to the machine where IIS is installed and the web application is deployed.
Run a console as an Administrator on this machine.
Invoke the Agent Client tool on this machine to enable runtime coverage collection inside IIS:
```
agent_client.exe
```
agent_client initializes the environment for the web server (IIS) and behaves like a service, enabling you to execute tests and collect coverage. The service is ready and waiting for commands as long as the following message is printed to the output:
```
Write 'exit' and hit Enter to close agent_client
```
A test session and test can be started even if the tested website or application has not been loaded yet.
Ensure that port 8050 (default port for the coverage agent) allows HTTP traffic in firewall settings on this machine. You can change the coverage agent port number if the default port is unavailable.
Start the application and verify that the agent is ready by opening <host>:8050/status in your browser. You should receive the following response:
```
{"session":null,"test":null}
```
Configure soatest.properties to collect coverage for the AUT. Several application coverage parameters are supported (see Application Coverage Settings), but configuring the following settings is suitable for most cases:
```
application.coverage.enabled=true
application.coverage.agent.url=http://localhost:8050
application.coverage.binaries=c:/<PATH_TO_APPLICATION_PDB_FILES>/bin
application.coverage.binaries.include=<INCLUDE_PATTERN>
application.coverage.binaries.exclude=<EXCLUDE_PATTERN>
```
Be sure the path for the application binaries is to the folder containing the application PDB files.

Standalone Applications

Most users will collect coverage for web applications, but you can also collect coverage information from standalone applications.

Copy the <INSTALL_DIR>/coverage/dotNET/dottest_agent folder to the machine where you intend to run the standalone application.
Run the coverage_wizard.exe in that folder, specifying the following:
- Full path the application
- Command line arguments
- Working directory
Select the .NET CLR version used by the application.
Enter the directory where the scripts generated by the wizard will be saved in the Target scripts directory field.
Enable the Use Coverage Agent option and enable or disable the Collect coverage without admin privileges option as appropriate.
- By default, admin privileges are required to collect coverage with coverage agent, but you can configure the tool to collect coverage by users without admin privileges. Doing so will cause two additional .bat scripts to be generated: runCamAgentInit.bat and runCamAgentUninit.bat. Run the runCamAgentInit.bat script with admin privileges to setup and initialize the coverage agent. When testing is completed, run the runCamAgentUninit.bat script to stop the process.
Click Finish. The wizard will generate scripts in target script directory specified above.
Run a console as an Administrator on this machine and change directories to where the scripts were generated.
Execute runCamAgent.bat.
Run another console as an Administrator and execute monitorCoverage.bat.
Configure soatest.properties to collect coverage for the AUT. Several application coverage parameters are supported (see Application Coverage Settings), but configuring the following settings is suitable for most cases:
```
application.coverage.enabled=true
application.coverage.agent.url=http://localhost:8050
application.coverage.binaries=c:/<PATH_TO_APPLICATION_PDB_FILES>/bin
application.coverage.binaries.include=<INCLUDE_PATTERN>
application.coverage.binaries.exclude=<EXCLUDE_PATTERN>
```
Be sure the path for the application binaries is to the folder containing the application PDB files.

Configuring SOAtest to Collect and Upload Coverage

With the proper configuration, SOAtest will collect application coverage during test execution. To do so, configure a SOAtest settings file (see Configuring Settings) with the following options, as appropriate:

build.id: Set to the build ID with which to correlate coverage results. If you want to merge SOAtest coverage results with results from other types of testing—such as unit testing and manual testing of the same application—they must all use the same build ID. The build ID set here must match the build ID set for dotTEST or Jtest when collecting the static and dynamic coverage files and uploading the coverage data.
dtp.project: This value must match the dtp.project value set for dotTEST or Jtest when collecting the static and dynamic coverage files.
report.dtp.publish: Set to true to enable publishing test result data to DTP.
application.coverage.enabled: Set to true. This enables application coverage.
application.coverage.agent.url: Specify the URL where the application under test and coverage agent are hosted. Include the protocol (HTTP/HTTPS) and port number of the agent. Separate multiple entries with a semi-colon ( ; ). For example: http\://<Agent Host>\:8050;http\://<Agent Host>\:9050.
application.coverage.user.id: Optional. Specify a user ID so that coverage results can be associated with a specific user. A user ID should only be specified when the coverage agent is configured to run in multi-user mode. See jtest.agent.enableMultiuserCoverage below.
application.coverage.dtp.publish: Set to true to upload coverage reports to DTP or false to not upload coverage reports to DTP. If you have configured multiple coverage agents, their coverage reports will be merged into a single coverage report for DTP. Defaults to the value of report.dtp.publish (see Configuring Settings) or to true if -publish is passed to soatestcli.
application.coverage.binaries: Optional. For Jtest, specify the URL or local file that contains binaries of the application under test (AUT). You can specify the path to a folder or a .war, .jar, .zip, or .ear file. For dotTEST, specify the folder containing the application PDB files. Not needed if the static coverage file is uploaded to DTP separately. Do not use if application.coverage.static.file has been specified.
application.coverage.binaries.exclude: Optional, but recommended. Specify a comma-separated pattern of application binaries to exclude during AUT scanning. If neither this nor application.coverage.binaries.include is configured, analysis of binaries will take longer and analyze more of the application than needed. Do not use if application.coverage.static.file has been specified. By default, all binaries are accepted. For example, application.coverage.binaries.exclude=jakarta/**,java/**,javax/**,org/apache/**,org/springframework/**
application.coverage.binaries.include: Optional, but recommended. If neither this nor application.coverage.binaries.exclude is configured, analysis of binaries will take longer and analyze more of the application than needed. Specify a comma-separated pattern of application binaries to include during AUT scanning. Do not use if application.coverage.static.file has been specified. By default, all binaries are accepted. For example, application.coverage.binaries.include=com/myapp/data/*,com/myapp/common/**
application.coverage.static.file: Optional. Specify the path to the static coverage file generated with dotTEST or Jtest in the .xml or .data format, or by the SOAtest coverage tool that analyzed the application binaries. Not needed if the static coverage file is uploaded to DTP separately. Do not use if application.coverage.binaries has been specified.
application.coverage.images: Specify a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. Separate multiple entries with a semi-colon ( ; ). For details, see the Parasoft DTP user guide. For example: ${dtp_project};${dtp_project}_Functional Test.

Executing Tests to Collect and Upload Coverage

Run your tests in command line mode using the Test Configuration and settings file you have configured (see Configuring SOAtest to Collect and Upload Coverage). This step should be part of an automated process. Coverage data will be collected as tests execute and, if publishing to DTP has been enabled, uploaded to DTP.

Reviewing Coverage in DTP

Go to DTP and open the Coverage Explorer to review the application coverage achieved by your SOAtest tests. For details on using the Coverage Explorer, see the Parasoft DTP user guide at https://docs.parasoft.com.

Page tree

Application Coverage