In this guide:

Executive Summary

This guide is intended to help you collect coverage information for microservices developed with .NET framework or Java technology.

The primary audience for this user guide is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

Prerequisites

We assume that you are familiar with Parasoft technologies and have already deployed and licensed the following products:

• Parasoft CTP
• Parasoft DTP

Microservices Coverage Workflow

1. Setup

   • Deploy and license Parasoft DTP.

     • You can pre-configure DTP projects for each microservice, or the projects will be automatically created when data is first published to a named project.

   • Deploy and license Parasoft CTP.

     • Configure CTP's connection to DTP.

     • Configure an environment that represents all of the deployed microservices you wish to collect code coverage for.

2. Build Phase

   • Generate static coverage for each microservice.

     • Static coverage files are generated by the covtools that ship with CTP, or if your development team has adopted Jtest or dotTEST in their build pipeline they can also generate the static coverage files.

     • Static coverage files are published to Parasoft DTP and associated with a specific build of each microservice.

3. Deploy Phase

   • Attach coverage agents to each microservice.

     • The coverage agents ship with CTP and allow you to monitor the code being executed when the AUT is running.

   • Configure CTP to connect to the coverage agents.

     • Your CI/CD pipeline can automatically synchronize CTP's configuration with coverage agent deployments via REST API.

4. Test Phase

   • Run tests against the microservices and report the results to CTP.

     • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.

     • Upload the runtime coverage to DTP.

       • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

5. Review Phase

   • Review test reports and application coverage in DTP.

Step 1: Deploy and License DTP and CTP

In DTP, create a project for each microservice for which you want to collect coverage. Both static and runtime coverage for a given microservice will be collected to and associated under its corresponding project. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.

DTP Properties for Coverage

It is important that the following properties are set when generating and uploading both static and runtime coverage to DTP. Furthermore, in order to correctly associate between reported static and runtime coverage, these values must be identical on a per-microservice basis. For example, the project setting can and should differ to reference two different DTP projects for two different microservices, but for each microservice the static coverage settings and runtime coverage settings should reference the same project. That is, both the static coverage settings and runtime coverage settings for Microservice A should reference the same DTP project, Project A; and the static coverage settings and runtime coverage settings for Microservice B should reference the same DTP project, Project B.

1. DTP Project
   This references the DTP project that will be used to aggregate coverage data for this microservice. It is referred to as dtp.project in the settings for static coverage generation and is the DTP project field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
2. Build ID
   This refers to the build identifier used to label a collective set of associated results. It is referred to as build.id in the settings for static coverage generation and is the Build ID field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
3. Coverage Images
   This refers to a set of tags that can be used to differentiate different types of coverage in DTP. Since DTP aggregates code coverage from a variety of testing practices (unit testing, functional testing, and so on) the coverage image tags allow users to both merge coverage images to review overall Application coverage, as well as differentiate coverage between testing practices.
   It is referred to as report.coverage.images in the settings for static coverage generation and is the Coverage images field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice. The value %{dtp_project} will automatically give the coverage image the same name as the DTP project. If the coverage image has any other name, you will need to update your DTP filter.
   Generally speaking, users often define two coverage image tags for each testing type performed. An "all" tag that will associate coverage from multiple testing practices together, as well as a testing practice specific tag like "functional" so that DTP can report on coverage specifically from functional testing. Similarly, developers using Jtest or dotTEST would define coverage image tags for their unit tests like "all" and "unit", and DTP would merge the unit test coverage and functional test coverage into a single coverage metric. A common convention would be: %{dtp_project}-all and %{dtp_project}-functional and %{dtp_project}-unit.
4. Session Tag
   This refers to a tag which is a unique identifier for the test run, to differentiate from other test runs under a common Build ID. It is referred to as session.tag in the settings for static coverage generation and is the sessionTag property of the JSON request body in the CTP REST API endpoint used to upload runtime coverage to DTP.

Step 2: Generate Static Coverage for Each Microservice

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with CTP to analyze the application binaries (jtestcov for Java applications or dottestcov for .NET applications). These coverage tools can be accessed either from the component instance editor page or from the links under the Coverage section of the Getting Started Widget, which can be added or found on the main entry page for CTP. The coverage tools can also be downloaded from a CTP endpoint if used in an automation pipeline.

If development has adopted Jtest or dotTEST in their CI pipelines, these tools can generate the static coverage file from source code and publish it to DTP as an alternative to using the coverage tools shipped with CTP.

Generating static coverage from source code

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. See Application Coverage (Jtest) or Application Coverage (dotTEST) for more information on generating static coverage files with those tools.

Generating static coverage with jtestcov

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

Use the jtestcov jar with a command like the one below running against Parabank:

java -jar <PATH_TO_jtestcov.jar> -soatest -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties

# === LICENSE ===
 
# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
#parasoft.eula.accepted=false
 
# === LOCAL LICENSE ===
# Enables local license - be sure to specify password.
#jtest.license.use_network=false
 
# Specifies password for the local license.
#jtest.license.local.password=[LICENSE PASSWORD]
 
# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
#jtest.license.use_network=true
 
# Specifies type of the network license (edition).
# Supported editions: desktop_edition, desktop_compliance_edition, server_edition, server_compliance_edition
#jtest.license.network.edition=server_edition
 
# Enables specific list of license features
#jtest.license.network.edition=custom_edition
#jtest.license.custom_edition_features=Jtest, Automation, Desktop Command Line, DTP Publish, Coverage, Test Impact Analysis
 
 
# === DTP SERVER SETTINGS ===
 
# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443
 
# Specifies user name for DTP server authentication.
#dtp.user=admin
 
# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin
 
# Specifies name of the DTP project - this settings is optional.
#dtp.project=[DTP Project Name]
 
# === DTP REPORTING ===
 
# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true
 
# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd
 
# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]
 
# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test
 
# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to high.
#console.verbosity.level=high

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using either dottestcov.sh or dottestcov.bash, depending on your operating system, with a command like the example below running against Parabank:

dottestcov(.sh/.bash) coverage -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

Step 3: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

1. Run runCamAgent.bat to start the coverage agent.
2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Step 4: Configure CTP to Connect to the Coverage Agents

First, in CTP, go to administration (choose User Profile from the username menu in the upper-right corner) and configure your connection to the DTP where you defined your projects.

Next, create a system to represent the architecture of your application. Each microsystem should have its own component in this system representation. For more details on creating systems in CTP, refer to the CTP user guide at https://docs.parasoft.com.

Once you have created a system, next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

If desired, you can additionally fill out other details for component instances. For example, filling out the real endpoints section of the components tab in the component instance manager will allow you to see the status of that endpoint from the environment diagram.

After environment configuration is complete, you can check to make sure all necessary components have their coverage agents configured and connected; components with coverage agents are denoted with a blue C symbol, and a green checkmark signifies a good status for the connection.

The CTP Components or the entire Environment can be synchronized from your deployment pipeline via REST API. For example, when a new version of your microservice is built and deployed, it will have a new Build ID that CTP needs to have a reference to. The microservice's deployment endpoint, and thus coverage agent endpoint, may also change, and synchronizing CTP with the current state of your deployed microservices is essential in correctly measuring code coverage and associating it with the appropriate builds that have been deployed. If automation drives the deployment of your applications into a test environment, it is recommended to automate updating CTP at the same time.

You can use the following APIs to update CTP:

• GET - /v3/environments/{environmentId}/config
• PUT - /v3/environments/{environmentId}/config 

Step 5: Run Tests Against the Microservices and Report the Results to CTP

To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the agent's group of REST endpoints.

The workflow goes as follows:

1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/session/start
   • GET /v3/environments/{environmentId}/agents/session/start

   where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

2. Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/test/start
   • GET /v3/environments/{environmentId}/agents/test/start
   • GET /v3/environments/{environmentId}/agents/test/start/{testId}

   specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases.

3. Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

4. After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/test/stop
   • GET /v3/environments/{environmentId}/agents/test/stop
   • GET /v3/environments/{environmentId}/agents/test/stop/{testId}

   specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

5. After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/session/stop
   • GET /v3/environments/{environmentId}/agents/session/stop

Integrating a Third-party Test Framework

For a better understanding of how to effectively integrate into a third-party test framework, see this example of the Spring Petclinic with Selenium tests on GitHub. 

Step 6: Upload the Runtime Coverage to DTP

To upload the runtime coverage session generated by the previous step to DTP, we will again use the CTP REST API. Make a REST request to the endpoint:

• POST /v3/environments/{environmentId}/coverage/{id}

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image).

Step 7: Review Application Coverage on DTP

Coverage statistics should now be visible in DTP for the microservice projects.

By clicking on a coverage widget, we can drill down and view coverage statistics not only for the whole project but individual files and methods as well.

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow

You can use CTP in conjunction with DTP to not only gather coverage information for your microservices for a particular build, but also use them to conduct Test Impact Analysis: examining the coverage data of two different builds to determine which tests have been impacted by code changes between the two builds. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes they made in a build.

1. Use the microservices coverage workflow to establish a baseline.
   A baseline build in DTP will be used to compare against a future build.

2. Modify your microservice(s).
   Make code or other changes to your microservices.

3. Use the microservices coverage workflow to generate new static coverage with your changes.
   This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

4. Use the CTP REST API to connect with DTP and get a list of impacted tests.

Step 1: Use the Microservices Coverage Workflow to Establish a Baseline

To conduct Test Impact Analysis, it is necessary to have a baseline build in DTP with coverage information to compare against. Use the Microservices Coverage Workflow outlined above to first generate coverage. Then, send a REST request to the CTP API endpoint at

• POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

Step 2: Modify your Microservice(s)

Make code or other changes to your microservices.

Step 3: Use the Microservices Coverage Workflow to Generate New Static Coverage with Your Changes

Use the Microservices Coverage Workflow and generate a new static coverage using new build ID to compare against the baseline build.

Be sure to use the same environment as in Step 1 and update the DTP Build IDs field in the coverage tab of your component instances configured for coverage agents to new Build IDs.

Step 4: Use the CTP REST API to Connect with DTP and Get a List of Impacted Tests

Once you have static coverage for a new build with your changes, send a REST request to the CTP API endpoint at

• GET /v3/environments/{environmentId}/coverage/impactedTests

where environmentId is the ID of the environment used for both the baseline and the target (new) build. Use the baselineBuildId query parameter to specify the baseline build ID you chose in Step 1.

The endpoint will return a JSON list of tests which DTP has determined have been impacted by code changes between the two builds. For example, here is a response consisting of a single impacted test:

[
  {
    "id": "5f6edb09-8e34-3885-aa7f-32435d3c8a8f",
    "testName": "first visit",
    "analysisType": "UT",
    "testSuiteName": "first visit",
    "toolName": "CTP"
  }
]

Troubleshooting

Takes a long time to create static coverage / process coverage

This usually happens when the system is processing more than it need to. Check your includes and excludes and update them appropriately.