Page History

...

We assume that you are familiar with Parasoft technologies and have already deployed and licensed the following products:

• Parasoft CTP
• Parasoft DTP

Configure DTP

In DTP, create a project for each microservice for which you want to collect coverage. Both static and runtime coverage for a given microservice will be collected to and associated under its corresponding project. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.

...

Anchor
ConfigureDTP ConfigureDTP

The first step in DTP is project creation. With microservices, there are two common methods for organizing and building code, either in separate repositories or a shared repository. Parasoft supports both methods.

The Spring PetClinic project that is used as an example in this guide uses a shared repository, where all the microservices are maintained in one repository and there is a single build pipeline for the entire PetClinic system even though each microservice is deployed in a separate container. In this document, there will be times when there are subtle differences in the steps you need to take depending on which method you are using. These instances will be clearly marked as they are above to make it easy to follow.

Anchor
DTPPropertiesForCoverage DTPPropertiesForCoverage

DTP Properties for Coverage

There are two XML coverage reports that make up code coverage reporting for a project, a static coverage report that is generated during the build phase and a dynamic (runtime) coverage report that is generated during runtime when testing occurs. Because these reports are published to DTP separately at different stages of the lifecycle and need to be merged, it is important that the DTP properties associated with them are consistent for each project. This guide will describe how to define these properties during the various stages of the Application Coverage workflow, but it is important to remember that these settings are a critical factor in ensuring the data is published and merged correctly.

1. DTP Project
   This references the DTP project that will be used to aggregate coverage data for this microservices project. It is referred to as dtp.project

It is important that the following properties are set when generating and uploading both static and runtime coverage to DTP. Furthermore, in order to correctly associate between reported static and runtime coverage, these values must be identical on a per-microservice basis. For example, the project setting can and should differ to reference two different DTP projects for two different microservices, but for each microservice the static coverage settings and runtime coverage settings should reference the same project. That is, both the static coverage settings and runtime coverage settings for Microservice A should reference the same DTP project, Project A; and the static coverage settings and runtime coverage settings for Microservice B should reference the same DTP project, Project B.

1. DTP Project
   This references the DTP project that will be used to aggregate coverage data for this microservice. It is referred to as dtp.project in the settings for static coverage generation and is the DTP project field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
2. Build ID
   This refers to the build identifier used to label a collective set of associated results. It is referred to as build.id in the settings for static coverage generation and is the Build ID field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
3. Coverage Images
   This refers to a set of tags that can be used to differentiate different types of coverage in DTP. Since DTP aggregates code coverage from a variety of testing practices (unit testing, functional testing, and so on) the coverage image tags allow users to both merge coverage images to review overall Application coverage, as well as differentiate coverage between testing practices.
   It is referred to as report.coverage.images in the settings for static coverage generation and is the Coverage images field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice. The value %{dtp_project} will automatically give the coverage image the same name as the DTP project. If the coverage image has any other name, you will need to update your DTP filter.
   Generally speaking, users often define two coverage image tags for each testing type performed. An "all" tag that will associate coverage from multiple testing practices together, as well as a testing practice specific tag like "functional" so that DTP can report on coverage specifically from functional testing. Similarly, developers using Jtest or dotTEST would define coverage image tags for their unit tests like "all" and "unit", and DTP would merge the unit test coverage and functional test coverage into a single coverage metric. A common convention would be: %{dtp_project}-all and %{dtp_project}-functional and %{dtp_project}-unit.
4. Session Tag
   This refers to a tag which is a unique identifier for the test run, to differentiate from other test runs under a common Build ID. It is referred to as session.tag in the settings for static coverage generation and is the sessionTag property of the JSON request body in the CTP REST API endpoint used to upload runtime coverage to DTP.

Configure CTP

First, in CTP, go to administration (choose User Profile from the username menu in the upper-right corner) and configure your connection to the DTP where you defined your projects.

Image Removed

Next, create a system to represent the architecture of your application. Each microservice should have its own component in this system representation. You will come back to this System later in this guide to add an Environment, for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at https://docs.parasoft.com.

Image Removed

...

Image Removed

Build Phase

Step 1: Generate static coverage for each microservice.

• • Static coverage files are generated by the covtools that ship with CTP, or if your development team has adopted Jtest or dotTEST in their build pipeline they can also generate the static coverage files.

  • Static coverage files are published to Parasoft DTP and associated with a specific build of each microservice.

Deploy Phase

Step 2: Attach coverage agents to each microservice.

• • The coverage agents ship with CTP and allow you to monitor the code being executed when the AUT is running.

Step 3: Update CTP environment with coverage agent details.

Test Phase

Step 4: Trigger Parasoft CTP APIs from test framework.

• • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.

  • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

  • (Optional) Create baseline build in CTP

Review Phase

Step 5: Review test reports and application coverage in DTP.

...

1. DTP project field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
2. Build ID
   This refers to the build identifier used to label a collective set of associated results. It is referred to as build.id in the settings for static coverage generation and is the Build ID field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
3. Coverage Images
   This refers to a set of tags that can be used to differentiate different types of coverage in DTP. Since DTP aggregates code coverage from a variety of testing practices (unit testing, functional testing, and so on) the coverage image tags allow users to both merge coverage images to review overall Application coverage, as well as differentiate coverage between testing practices.
   It is referred to as report.coverage.images in the settings for static coverage generation and is the Coverage images field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice. The value %{dtp_project} will automatically give the coverage image the same name as the DTP project. If the coverage image has any other name, you will need to update your DTP filter.
   Generally speaking, users often define two coverage image tags for each testing type performed. An "all" tag that will associate coverage from multiple testing practices together, as well as a testing practice specific tag like "functional" so that DTP can report on coverage specifically from functional testing. Similarly, developers using Jtest or dotTEST would define coverage image tags for their unit tests like "all" and "unit", and DTP would merge the unit test coverage and functional test coverage into a single coverage metric. A common convention would be: %{dtp_project}-all and %{dtp_project}-functional and %{dtp_project}-unit.
4. Session Tag
   This refers to a tag which is a unique identifier for the test run, to differentiate from other test runs under a common Build ID. It is referred to as session.tag in the settings for static coverage generation and is the sessionTag property of the JSON request body in the CTP REST API endpoint used to upload runtime coverage to DTP.
   

   Note
   A common mistake is publishing multiple test reports to DTP, where DTP Project, Build ID, and other properties like the testing tool are identical, when the intention is not to overwrite the data. Unique session tags are necessary if you want test results from multiple reports to append instead of overwrite.

Configure CTP

Anchor
ConfigureCTP ConfigureCTP

First, in CTP, go to administration (choose User Profile from the username menu in the upper-right corner) and configure your connection to the DTP where you defined your projects.

Image Added

Next, create a system to represent the deployment architecture of your application. No matter how the microservices code is organized or built, here each microservice that runs as a separate process/container will have a dedicated coverage agent and should have its own component in the Environment Manager system representation.

You will come back to this System later in this guide to add an environment, but for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at https://docs.parasoft.com.

Image Added

Microservices Coverage Workflow

Anchor
MicroservicesCoverageWorkflow MicroservicesCoverageWorkflow

Image Added

Build Phase

Step 1: Generate static coverage for each microservices project.

• • Static coverage files are generated by the covtools that ship with CTP, or if your development team has adopted Jtest or dotTEST in their build pipeline they can also generate the static coverage files.

  • Static coverage files are published to Parasoft DTP and associated with a specific build of each microservices project.

Deploy Phase

Step 2: Attach coverage agents to each microservice.

• • The coverage agents ship with CTP and allow you to monitor the code being executed when the AUT is running.

Step 3: Update CTP environment with coverage agent details.

Test Phase

Step 4: Trigger Parasoft CTP APIs from test framework.

• • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.

  • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

  • (Optional) Create baseline build in CTP

Review Phase

Step 5: Review test reports and application coverage in DTP.

Step 1: Generate Static Coverage for Each Microservices Project

Anchor
GenerateStaticCoverage GenerateStaticCoverage

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with CTP to analyze the application binaries (jtestcov for Java applications or dottestcov for .NET applications). These coverage tools can be accessed either from the component instance editor page or from the links under the Coverage section of the Getting Started Widget, which can be added or found on the main entry page for CTP. The coverage tools can also be downloaded from a CTP endpoint if used in an automation pipeline.

If development has adopted Jtest or dotTEST in their CI pipelines, these tools can generate the static coverage file from source code and publish it to DTP as an alternative to using the coverage tools shipped with CTP.

Image Added

Generating static coverage from source code

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. It also improves Test Impact Analysis, as static coverage generated this way provides for more precise analysis so tests will only be flagged as impacted if they traverse any methods that have been changed. While you can use TIA with static coverage generated from application binaries, TIA results will be broader (since analysis is done at the level of classes) and some tests may be flagged as being impacted when they don't need to be. See Application Coverage (Jtest) or Application Coverage (dotTEST) for more information on generating static coverage files with those tools.

Generating static coverage with jtestcov

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

Use the jtestcov jar with a command like the one below running against Parabank:

java -jar <PATH_TO_jtestcov.jar> -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/** -settings c:/<PATH TO .PROPERTIES>/jtestcov.properties

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using dottestcov.bat with a command like the example below:

dottestcov.bat -ctp -include "C:\Devel\FooSolution\src\QuxProject***.cs" -exclude "C:\Devel\FooSolution\src\QuxProject*\tests**.cs" -app <DIR>\FooSolution.sln -publish -settings <PATH TO .PROPERTIES>\dottestcov

...

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with CTP to analyze the application binaries (jtestcov for Java applications or dottestcov for .NET applications). These coverage tools can be accessed either from the component instance editor page or from the links under the Coverage section of the Getting Started Widget, which can be added or found on the main entry page for CTP. The coverage tools can also be downloaded from a CTP endpoint if used in an automation pipeline.

If development has adopted Jtest or dotTEST in their CI pipelines, these tools can generate the static coverage file from source code and publish it to DTP as an alternative to using the coverage tools shipped with CTP.

Image Removed

Generating static coverage from source code

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. See Application Coverage (Jtest) or Application Coverage (dotTEST) for more information on generating static coverage files with those tools.

Generating static coverage with jtestcov

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

Use the jtestcov jar with a command like the one below running against Parabank:

java -jar <PATH_TO_jtestcov.jar> -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties

# === LICENSE ===
 
# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
#parasoft.eula.accepted=false
 
# === LOCAL LICENSE ===
# Enables local license - be sure to specify password.
#jtest.license.use_network=false
 
# Specifies password for the local license.
#jtest.license.local.password=[LICENSE PASSWORD]
  EULA.txt file included in the product installation directory.
parasoft.eula.accepted=true

# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
#jtest.license.use_network=true
 
# Specifies type of the network license (edition).
# Supported editions: desktop_edition, desktop_compliance_edition, server_edition, server_compliance_edition
#jtestctp.license.use_network=true
ctp.license.network.edition=servercustom_edition
 
# Enables specific list of license features
#jtest.license.network.edition=custom_edition
#jtest# Note: customize the CTP coverage agent tier based on what your CTP license has enabled
ctp.license.custom_edition_features=JtestCTP, Automation, Desktop Command Line, DTP Publish, Coverage, TestTier Impact5 Analysis
 
 
# === DTP SERVER SETTINGS ===
 
# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443
 
# Specifies user name for DTP server authentication.
#dtp.user=admin
 
# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin
 
# Specifies name of the DTP project - this settings is optional.
#dtp.project=[DTP Project Name]
 
# === DTP REPORTING ===
 
# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true
 
# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd
 
# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]
 
# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test
 
# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to to high.
#console.verbosity.level=high

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using either dottestcov.sh or dottestcov.bash, depending on your operating system, with a command like the example below running against Parabank:

dottestcov(.sh/.bash) coverage -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties

...

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

Image Removed

...

high.
#console.verbosity.level=high

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

Image Added

Anchor
AttachCoverageAgents AttachCoverageAgents

Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to collect dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. As part of this process, you will have the opportunity to review the agent's settings in the agent.properties file; while doing so, ensure that the include and exclude values that are used are the same as those defined in your static coverage process.

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where <HOST> is where the microservice is running and <PORT> is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

1. Run runCamAgent.bat to start the coverage agent.
2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at at http://<HOST>:<PORT>/status, where host <HOST> is where the microservice is running and port <PORT> is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

1. Run runCamAgent.bat to start the coverage agent.
2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

...

Anchor
UpdateCTPEnvironment UpdateCTPEnvironment

Step 3: Update CTP Environment with Coverage Agent Details

Go back the CTP System you created in the Environment Manager during setup and create an Environment for it. In this Environment, you want to create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. How you configure the coverage agent depends on whether you configured DTP for separate or shared repositories when you configured DTP during setup:

These Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice deployment this component represents.

...

• GET - /v3/environments/{environmentId}/config
• PUT - /v3/environments/{environmentId}/config /config

To see an example of how this was done with the Spring Petclinic project, see the following Jenkinsfile: https://github.com/gtrofimov/spring-petclinic-microservices/blob/master/jobs/Jenkinsfile.deploy

Step 4: Trigger Parasoft CTP APIs from Test Framework

...

After changes have been made to your microservices that trigger a new target build, the following steps define how to integrate Test Impact Analysis into your test execution.

Step 1: Generate New Static Coverage for Each

...

Microservices Project

Use the Step 1 of Microservices Coverage Workflow and generate new static coverage files for each microservice microservices project that changed, using new build IDs to describe the new builds that are to be tested.
These static coverage files will be published to DTP, which is how DTP will know what code changed between the baseline and target builds in order to calculate impacted tests.

...