In this guide:

Executive Summary

This guide is intended to help you collect code coverage information for microservices developed in .NET or Java during functional test execution and leverage that information to optimize subsequent test executions with test impact analysis.

The primary audience for this user guide is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

Setup

Prerequisites

We assume that you are familiar with Parasoft technologies and have already deployed and licensed the following products:

Configure DTP

In DTP, create a project for each microservice for which you want to collect coverage. Both static and runtime coverage for a given microservice will be collected to and associated under its corresponding project. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.

DTP Properties for Coverage

It is important that the following properties are set when generating and uploading both static and runtime coverage to DTP. Furthermore, in order to correctly associate between reported static and runtime coverage, these values must be identical on a per-microservice basis. For example, the project setting can and should differ to reference two different DTP projects for two different microservices, but for each microservice the static coverage settings and runtime coverage settings should reference the same project. That is, both the static coverage settings and runtime coverage settings for Microservice A should reference the same DTP project, Project A; and the static coverage settings and runtime coverage settings for Microservice B should reference the same DTP project, Project B.

  1. DTP Project
    This references the DTP project that will be used to aggregate coverage data for this microservice. It is referred to as dtp.project in the settings for static coverage generation and is the DTP project field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
  2. Build ID
    This refers to the build identifier used to label a collective set of associated results. It is referred to as build.id in the settings for static coverage generation and is the Build ID field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
  3. Coverage Images
    This refers to a set of tags that can be used to differentiate different types of coverage in DTP. Since DTP aggregates code coverage from a variety of testing practices (unit testing, functional testing, and so on) the coverage image tags allow users to both merge coverage images to review overall Application coverage, as well as differentiate coverage between testing practices.
    It is referred to as report.coverage.images in the settings for static coverage generation and is the Coverage images field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice. The value %{dtp_project} will automatically give the coverage image the same name as the DTP project. If the coverage image has any other name, you will need to update your DTP filter.
    Generally speaking, users often define two coverage image tags for each testing type performed. An "all" tag that will associate coverage from multiple testing practices together, as well as a testing practice specific tag like "functional" so that DTP can report on coverage specifically from functional testing. Similarly, developers using Jtest or dotTEST would define coverage image tags for their unit tests like "all" and "unit", and DTP would merge the unit test coverage and functional test coverage into a single coverage metric. A common convention would be: %{dtp_project}-all and %{dtp_project}-functional and %{dtp_project}-unit.
  4. Session Tag
    This refers to a tag which is a unique identifier for the test run, to differentiate from other test runs under a common Build ID. It is referred to as session.tag in the settings for static coverage generation and is the sessionTag property of the JSON request body in the CTP REST API endpoint used to upload runtime coverage to DTP.

Configure CTP

First, in CTP, go to administration (choose User Profile from the username menu in the upper-right corner) and configure your connection to the DTP where you defined your projects.

An image of CTP's DTP Configuration page under Administration

Next, create a system to represent the architecture of your application. Each microservice should have its own component in this system representation. You will come back to this System later in this guide to add an Environment, for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at https://docs.parasoft.com.

A CTP system diagram for an application with a microservice architecture.

Microservices Coverage Workflow


Build Phase

Step 1: Generate static coverage for each microservice.

Deploy Phase

Step 2: Attach coverage agents to each microservice.

Step 3: Update CTP environment with coverage agent details.

Test Phase

Step 4: Trigger Parasoft CTP APIs from test framework.

Review Phase

Step 5: Review test reports and application coverage in DTP.

Step 1: Generate Static Coverage for Each Microservice

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with CTP to analyze the application binaries (jtestcov for Java applications or dottestcov for .NET applications). These coverage tools can be accessed either from the component instance editor page or from the links under the Coverage section of the Getting Started Widget, which can be added or found on the main entry page for CTP. The coverage tools can also be downloaded from a CTP endpoint if used in an automation pipeline.

If development has adopted Jtest or dotTEST in their CI pipelines, these tools can generate the static coverage file from source code and publish it to DTP as an alternative to using the coverage tools shipped with CTP.

Generating static coverage from source code

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. It also improves Test Impact Analysis, as static coverage generated this way provides for more precise analysis so tests will only be flagged as impacted if they traverse any methods that have been changed. While you can use TIA with static coverage generated from application binaries, TIA results will be broader (since analysis is done at the level of classes) and some tests may be flagged as being impacted when they don't need to be. See Application Coverage (Jtest) or Application Coverage (dotTEST) for more information on generating static coverage files with those tools.

Generating static coverage with jtestcov

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

Use the jtestcov jar with a command like the one below running against Parabank:

java -jar <PATH_TO_jtestcov.jar> -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/** -settings c:/<PATH TO .PROPERTIES>/jtestcov.properties

The -ctp flag is used by the coverage tools to pass your CTP license, which is needed when you run the coverage tool independently of CTP as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched. You can see all available commands and options by running: java -jar jtestcov.jar help

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using dottestcov.bat with a command like the example below:

dottestcov.bat -ctp -include "C:\Devel\FooSolution\src\QuxProject***.cs" -exclude "C:\Devel\FooSolution\src\QuxProject*\tests**.cs" -app <DIR>\FooSolution.sln -publish -settings <PATH TO .PROPERTIES>\dottestcov.properties

The -ctp flag is used by the coverage tools to pass your CTP license, which is needed when you run the coverage tool independently of CTP as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched.

# === LICENSE ===
 
# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
parasoft.eula.accepted=true

# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
ctp.license.use_network=true
ctp.license.network.edition=custom_edition
# Note: customize the CTP coverage agent tier based on what your CTP license has enabled
ctp.license.custom_edition_features=CTP, Coverage Tier 5 

# === DTP SERVER SETTINGS ===
 
# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443
 
# Specifies user name for DTP server authentication.
#dtp.user=admin
 
# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin
 
# Specifies name of the DTP project - this settings is optional.
#dtp.project=[DTP Project Name]
 
# === DTP REPORTING ===
 
# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true
 
# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd
 
# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]
 
# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test
 
# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to high.
#console.verbosity.level=high

Defining includes and excludes is an important part of controlling how much your system processes, which can greatly affect how long it takes to generate your static coverage files. In many cases, there is no need to measure code coverage of third-party library code, so it is preferable to limit code coverage to project code using these settings. Includes and excludes should be expressed as comma-separated lists of patterns that specify classes to be instrumented. The following wildcards are supported:

* matches zero or more characters
** matches multiple directory levels

In the following example, all classes from the com.myapp.data package and all classes from package and subpackages that start with com.myapp.common will be instrumented:

-include com/myapp/data/*,com/myapp/common/**

While in this example, all classes from the com.myapp.transport package and all classes from package and subpackages that start with com.myapp.autogen will be excluded from instrumentation:

-exclude com/myapp/transport/*,com/myapp/autogen/**

If you are outside of development, working with the application binaries, and not sure what the appropriate include/exclude settings should be, reach out to your colleagues in development to make sure you use the correct patterns. There is usually a standard pattern for packaging company code, like com.myapp.**, that should help you determine the right setting to use.

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

An image of a DTP dashboard showing the coverage and build administration widgets for four different projects corresponding to four different microservices. Note that each coverage widget displays 0 out of some variable number of lines covered.

Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. As part of this process, you will have the opportunity to review the agent's settings in the agent.properties file; while doing so, ensure that the include and exclude values that are used are the same as those defined in your static coverage process.

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where <HOST> is where the microservice is running and <PORT> is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

  1. Run runCamAgent.bat to start the coverage agent.
  2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where <HOST> is where the microservice is running and <PORT> is specified in agent.properties; by default, it is 8050.

Step 3: Update CTP Environment with Coverage Agent Details

Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

An image of the coverage tab of the component instance manager for a component of the same system as the previous image. The Agent URL, DTP Project, Filter, Build ID, and Coverage Image fields are filled out.

If desired, you can additionally fill out other details for component instances. For example, filling out the real endpoints section of the components tab in the component instance manager will allow you to see the status of that endpoint from the environment diagram.

After environment configuration is complete, you can check to make sure all necessary components have their coverage agents configured and connected; components with coverage agents are denoted with a blue C symbol, and a green checkmark signifies a good status for the connection.

An image of the configured environment for the same system as the previous images. Four of the components have coverage agents configured in their component instances, denoted by a blue C symbol, with good connections denoted by a green checkmark.

Now that the Environment in CTP has been fully setup via the GUI, the CTP Components or the entire Environment can be automatically synchronized from your deployment pipeline via REST API. For example, when a new version of your microservice is built and deployed, it will have a new Build ID that CTP needs to have a reference to. The microservice's deployment endpoint, and thus coverage agent endpoint, may also change, and synchronizing CTP with the current state of your deployed microservices is essential in correctly measuring code coverage and associating it with the appropriate builds that have been deployed. If automation drives the deployment of your applications into a test environment, it is recommended to automate updating CTP at the same time.

You can use the following APIs to update CTP:

Step 4: Trigger Parasoft CTP APIs from Test Framework

4a: Record Runtime Coverage

To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the /agents group of REST endpoints.

You can collect coverage information for multiple users that are simultaneously accessing the same microservices. There are some additional steps necessary for this:

  • For .NET, launch the coverage agent with the -multiuser switch. Note that multiuser mode is only supported for web applications; standalone applications are not supported.
    dottest_iismanager.exe -multiuser
  • For Java, enable the coverage agent's multiuser mode by modifying the options in the agent.properties file. Enable both of the multiuser options as shown below.
    jtest.agent.enableMultiuserCoverage=true
jtest.agent.autoloadMultiuserLibs=true

  • If multiuser mode is enabled and you are using SOAtest, CTP, dotTEST, or Jtest 2024.1 or later, your tests need to inject the following HTTP request header: 
    baggage: test-operator-id=<USER>

The workflow goes as follows:

  1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

    where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

  2. Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API:

    specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases. Optionally, you may include an array of work items with the POST request in order to leverage requirements traceability in DTP.

  3. Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

  4. After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

    specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

  5. After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

4b: Upload Runtime Coverage to DTP

For a better understanding of how to effectively integrate into a third-party test framework, see this example of the Spring Petclinic with Selenium tests on GitHub. 

To upload the runtime coverage session generated by the previous steps to DTP, we will again use the CTP REST API. Make a REST request to the endpoint:

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image). The sessionTag in the POST request body must be unique each time results are published for a given build ID in order for DTP to aggregate the test results. This is especially useful when leveraging multi-user mode to collect coverage during parallel test execution where each parallel thread is represented as a different user executing a different set of tests.

It is critical to ensure the DTP Project, Build ID, and Coverage Image Tags in the environment match the properties used to generate your static coverage. Otherwise, DTP may fail to associate the runtime and static coverage.

4c: Create baseline build in CTP (Optional)

This step is only required if you are implementing the Test Impact Analysis workflow outlined in the following section. Test Impact Analysis requires a baseline build with coverage from a full regression test run to use for comparison when calculating impacted tests from a subsequent build. To mark a build as a baseline build to be compared against, send an additional REST request at the end of your test session to the CTP API endpoint at

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

It is important to note that DTP has a rolling log of how much build data it retains in its database and file system (see Data Retention Settings in DTP for details). When baselining a build to be referenced by many builds in the future, it is important that the build be archived in DTP, so the data is available when CTP queries DTP to retrieve impacted tests. If a build important to your baseline is not archived and new builds get published to the DTP project exceeding the project's rolling log settings, build data will be lost preventing the test impact analysis workflow from functioning.

If you are using CTP 2024.1 or earlier, see Locking and Archiving Builds in DTP for more information about archiving builds in DTP. DTP also has a REST endpoint that allows you to dynamically archive and unarchive builds as appropriate to your process:

PUT /v12/builds/{id}/persistence

Starting with version 2024.2, CTP automatically archives DTP builds when creating a baseline, tagging these builds with the description "Archived by CTP" in DTP. Be aware that when someone calls

DELETE /v3/environments/{environmentId}/coverage/baselines/{id}

and passes ?unarchive=true to delete a baseline, CTP will automatically unarchive associated builds tagged with the "Archived by CTP" description. To protect against a build accidentally getting unarchived this way, change the description for the build in DTP.

Step 5: Review Application Coverage on DTP

Coverage statistics should now be visible in DTP for the microservice projects.

By clicking on a coverage widget, we can drill down and view coverage statistics not only for the whole project but individual files and methods as well.

A screenshot of DTP's coverage explorer showing coverage percentages for individual files and folders.

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow

You can use CTP and DTP to not only measure code coverage of microservices during functional testing, but also to enable Test Impact Analysis: an automated process for optimizing which tests are included for execution based on code changes in an application. Code coverage from test executions against a baseline build is used in conjunction with a code diff between that baseline build and a target build to calculate which tests are impacted for testing the target build. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes made in a build.

Prerequisite: Modify your microservice(s).

Step 1: Generate New Static Coverage for Each Microservice.

Step 2: Attach Coverage Agents to Each Microservice.

Step 3: Update CTP Environment with New Build Details.

Step 4: Call CTP REST API to Retrieve Impacted Tests.

Prerequisite: Modify your Microservice(s)

To conduct Test Impact Analysis, it is necessary to have a baseline build with coverage information from a full regression test run to compare against. Follow the Microservices Coverage Workflow defined in the previous section, including Step 4.c where CTP is configured with a baseline build id that will be used as a reference point for comparing with a target build in this workflow.

After changes have been made to your microservices that trigger a new target build, the following steps define how to integrate Test Impact Analysis into your test execution.

Step 1: Generate New Static Coverage for Each Microservice

Use the Step 1 of Microservices Coverage Workflow and generate new static coverage files for each microservice that changed, using new build IDs to describe the new builds that are to be tested.
These static coverage files will be published to DTP, which is how DTP will know what code changed between the baseline and target builds in order to calculate impacted tests.

Step 2: Attach Coverage Agents to Each Microservice

This part of the deploy process is no different from Step 2 of the Microservices Coverage Workflow. The updated microservices should get deployed to your test environment in preparation for the testing phase.

While it is technically not necessary to deploy coverage agents to run impacted tests, it is recommended to maintain a consistent deployment process where the coverage agents are part of the standard deployment to your test environment.

Step 3: Update CTP Environment with New Build Details

Similar to Step 3 of the Microservices Coverage Workflow, the same CTP environment should get updated about the new microservice builds during the deploy phase of the CI/CD process. It is important for CTP to have a reference of the new Build IDs of each microservice, where they match the Build IDs used to generate the new static coverage files from Step 1 of this workflow.

Since CTP needs to be updated every time new microservice builds are deployed into a test environment, it is recommended to use CTP's REST API to automate the updates during this deploy phase of the pipeline.

The Test Impact Analysis workflow requires configuring the optional Filter setting in CTP's coverage settings for a Component Instance. DTP projects can have a number of Filters that are used to segment the data it is reporting. By default, each DTP project starts with one filter that auto-includes all reports published to that project, and the CTP component instance UI will initialize to this filter by default. If updating CTP Environments via REST API, be sure the Filter setting is retained. For advanced users that use multiple DTP filters per project, be sure to reference a filter that contains the coverage reporting that CTP publishes to the corresponding DTP projects for each microservice.

Step 4: Call CTP REST API to Retrieve Impacted Tests

At this point, both your test environment and Parasoft platform should be ready for your optimized test execution with Test Impact Analysis. Before triggering your test framework to run tests, you need to retrieve the list of impacted tests from CTP in order to parameterize your test framework with which Test IDs it should include in its execution. Send a REST request to the CTP API endpoint at

where environmentId is the ID of the environment used for both the baseline and the target (new) build. Use the baselineBuildId query parameter to specify the baseline build ID you chose in Step 4.c of the Microservices Coverage Workflow when running your baseline test build.

The endpoint will return a JSON list of tests which DTP has determined have been impacted by code changes between the two builds. In the case where there are multiple microservices with new builds, CTP will query DTP for impacted tests for each changed microservice and then eliminate duplicates so the correct set of impacted tests is returned for all changes across the distributed system that was newly deployed in the test environment. For example, here is a response consisting of a single impacted test:

[
  {
    "id": "5f6edb09-8e34-3885-aa7f-32435d3c8a8f",
    "testName": "first visit",
    "analysisType": "FT",
    "testSuiteName": "first visit",
    "toolName": "CTP"
  }
]

Troubleshooting

Takes a long time to create static coverage / process coverage

This usually happens when the system is processing more than it needs to. Check your includes and excludes and update them appropriately.