Collecting Coverage for Microservices

In this guide:

Executive Summary

This guide is intended to help you collect code coverage information for microservices developed in .NET or Java during functional test execution and leverage that information to optimize subsequent test executions with test impact analysis.

The primary audience for this user guide is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

Setup

Prerequisites

We assume that you are familiar with Parasoft technologies and have already deployed and licensed the following products:

• Parasoft CTP
• Parasoft DTP

Configure DTP

In DTP, create a project for each microservice for which you want to collect coverage. Both static and runtime coverage for a given microservice will be collected to and associated under its corresponding project. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.

DTP Properties for Coverage

It is important that the following properties are set when generating and uploading both static and runtime coverage to DTP. Furthermore, in order to correctly associate between reported static and runtime coverage, these values must be identical on a per-microservice basis. For example, the project setting can and should differ to reference two different DTP projects for two different microservices, but for each microservice the static coverage settings and runtime coverage settings should reference the same project. That is, both the static coverage settings and runtime coverage settings for Microservice A should reference the same DTP project, Project A; and the static coverage settings and runtime coverage settings for Microservice B should reference the same DTP project, Project B.

1. DTP Project
   This references the DTP project that will be used to aggregate coverage data for this microservice. It is referred to as dtp.project in the settings for static coverage generation and is the DTP project field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
2. Build ID
   This refers to the build identifier used to label a collective set of associated results. It is referred to as build.id in the settings for static coverage generation and is the Build ID field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice.
3. Coverage Images
   This refers to a set of tags that can be used to differentiate different types of coverage in DTP. Since DTP aggregates code coverage from a variety of testing practices (unit testing, functional testing, and so on) the coverage image tags allow users to both merge coverage images to review overall Application coverage, as well as differentiate coverage between testing practices.
   It is referred to as report.coverage.images in the settings for static coverage generation and is the Coverage images field in the Coverage tab of the Component Instance manager in CTP where the coverage agent is configured for the microservice. The value %{dtp_project} will automatically give the coverage image the same name as the DTP project. If the coverage image has any other name, you will need to update your DTP filter.
   Generally speaking, users often define two coverage image tags for each testing type performed. An "all" tag that will associate coverage from multiple testing practices together, as well as a testing practice specific tag like "functional" so that DTP can report on coverage specifically from functional testing. Similarly, developers using Jtest or dotTEST would define coverage image tags for their unit tests like "all" and "unit", and DTP would merge the unit test coverage and functional test coverage into a single coverage metric. A common convention would be: %{dtp_project}-all and %{dtp_project}-functional and %{dtp_project}-unit.
4. Session Tag
   This refers to a tag which is a unique identifier for the test run, to differentiate from other test runs under a common Build ID. It is referred to as session.tag in the settings for static coverage generation and is the sessionTag property of the JSON request body in the CTP REST API endpoint used to upload runtime coverage to DTP.

Configure CTP

First, in CTP, go to administration (choose User Profile from the username menu in the upper-right corner) and configure your connection to the DTP where you defined your projects.

Next, create a system to represent the architecture of your application. Each microservice should have its own component in this system representation. You will come back to this System later in this guide to add an Environment, for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at https://docs.parasoft.com.

Microservices Coverage Workflow

Build Phase

Step 1: Generate static coverage for each microservice.

• • Static coverage files are generated by the covtools that ship with CTP, or if your development team has adopted Jtest or dotTEST in their build pipeline they can also generate the static coverage files.

  • Static coverage files are published to Parasoft DTP and associated with a specific build of each microservice.

Deploy Phase

Step 2: Attach coverage agents to each microservice.

• • The coverage agents ship with CTP and allow you to monitor the code being executed when the AUT is running.

Step 3: Update CTP environment with coverage agent details.

Test Phase

Step 4: Trigger Parasoft CTP APIs from test framework.

• • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.

  • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

  • (Optional) Create baseline build in CTP

Review Phase

Step 5: Review test reports and application coverage in DTP.

Step 1: Generate Static Coverage for Each Microservice

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

Static coverage files can be created by using Jtest or dotTEST to analyze the project's source code (the preferred method) or using the coverage tool shipped with CTP to analyze the application binaries (jtestcov for Java applications or dottestcov for .NET applications). These coverage tools can be accessed either from the component instance editor page or from the links under the Coverage section of the Getting Started Widget, which can be added or found on the main entry page for CTP. The coverage tools can also be downloaded from a CTP endpoint if used in an automation pipeline.

If development has adopted Jtest or dotTEST in their CI pipelines, these tools can generate the static coverage file from source code and publish it to DTP as an alternative to using the coverage tools shipped with CTP.

Generating static coverage from source code

If you have access to the application source code, you should generate the static coverage file using Jtest or dotTEST in .xml or .data format if at all possible. The method has several advantages, including containing metadata about user classes, methods, and lines, as well as supporting showing source code annotated with coverage data when viewing coverage results in DTP. See Application Coverage (Jtest) or Application Coverage (dotTEST) for more information on generating static coverage files with those tools.

Generating static coverage with jtestcov

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

Use the jtestcov jar with a command like the one below running against Parabank:

java -jar <PATH_TO_jtestcov.jar> -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/** -settings c:/<PATH TO .PROPERTIES>/jtestcov.properties

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using dottestcov.bat with a command like the example below:

dottestcov.bat -ctp -include "C:\Devel\FooSolution\src\QuxProject***.cs" -exclude "C:\Devel\FooSolution\src\QuxProject*\tests**.cs" -app <DIR>\FooSolution.sln -publish -settings <PATH TO .PROPERTIES>\dottestcov.properties

# === LICENSE ===
 
# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
parasoft.eula.accepted=true

# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
ctp.license.use_network=true
ctp.license.network.edition=custom_edition
# Note: customize the CTP coverage agent tier based on what your CTP license has enabled
ctp.license.custom_edition_features=CTP, Coverage Tier 5 

# === DTP SERVER SETTINGS ===
 
# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443
 
# Specifies user name for DTP server authentication.
#dtp.user=admin
 
# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin
 
# Specifies name of the DTP project - this settings is optional.
#dtp.project=[DTP Project Name]
 
# === DTP REPORTING ===
 
# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true
 
# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd
 
# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]
 
# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test
 
# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to high.
#console.verbosity.level=high

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

1. Run runCamAgent.bat to start the coverage agent.
2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Step 3: Update CTP Environment with Coverage Agent Details

Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

If desired, you can additionally fill out other details for component instances. For example, filling out the real endpoints section of the components tab in the component instance manager will allow you to see the status of that endpoint from the environment diagram.

After environment configuration is complete, you can check to make sure all necessary components have their coverage agents configured and connected; components with coverage agents are denoted with a blue C symbol, and a green checkmark signifies a good status for the connection.

Now that the Environment in CTP has been fully setup via the GUI, the CTP Components or the entire Environment can be automatically synchronized from your deployment pipeline via REST API. For example, when a new version of your microservice is built and deployed, it will have a new Build ID that CTP needs to have a reference to. The microservice's deployment endpoint, and thus coverage agent endpoint, may also change, and synchronizing CTP with the current state of your deployed microservices is essential in correctly measuring code coverage and associating it with the appropriate builds that have been deployed. If automation drives the deployment of your applications into a test environment, it is recommended to automate updating CTP at the same time.

You can use the following APIs to update CTP:

• GET - /v3/environments/{environmentId}/config
• PUT - /v3/environments/{environmentId}/config 

Step 4: Trigger Parasoft CTP APIs from Test Framework

4a: Record Runtime Coverage

To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the /agents group of REST endpoints.

The workflow goes as follows:

1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/session/start
   • GET /v3/environments/{environmentId}/agents/session/start

   where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

2. Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/test/start
   • GET /v3/environments/{environmentId}/agents/test/start
   • GET /v3/environments/{environmentId}/agents/test/start/{testId}

   specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases. Optionally, you may include an array of work items with the POST request in order to leverage requirements traceability in DTP.

3. Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

4. After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/test/stop
   • GET /v3/environments/{environmentId}/agents/test/stop
   • GET /v3/environments/{environmentId}/agents/test/stop/{testId}

   specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

5. After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

   • POST /v3/environments/{environmentId}/agents/session/stop
   • GET /v3/environments/{environmentId}/agents/session/stop

4b: Upload Runtime Coverage to DTP

To upload the runtime coverage session generated by the previous steps to DTP, we will again use the CTP REST API. Make a REST request to the endpoint:

• POST /v3/environments/{environmentId}/coverage/{id}

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image). The sessionTag in the POST request body must be unique each time results are published for a given build ID in order for DTP to aggregate the test results. This is especially useful when leveraging multi-user mode to collect coverage during parallel test execution where each parallel thread is represented as a different user executing a different set of tests.

4c: Create baseline build in CTP (Optional)

This step is only required if you are implementing the Test Impact Analysis workflow outlined in the following section. Test Impact Analysis requires a baseline build with coverage from a full regression test run to use for comparison when calculating impacted tests from a subsequent build. To mark a build as a baseline build to be compared against, send an additional REST request at the end of your test session to the CTP API endpoint at

• POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

PUT /v12/builds/{id}/persistence

DELETE /v3/environments/{environmentId}/coverage/baselines/{id}

Step 5: Review Application Coverage on DTP

Coverage statistics should now be visible in DTP for the microservice projects.

By clicking on a coverage widget, we can drill down and view coverage statistics not only for the whole project but individual files and methods as well.

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow

You can use CTP and DTP to not only measure code coverage of microservices during functional testing, but also to enable Test Impact Analysis: an automated process for optimizing which tests are included for execution based on code changes in an application. Code coverage from test executions against a baseline build is used in conjunction with a code diff between that baseline build and a target build to calculate which tests are impacted for testing the target build. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes made in a build.

Prerequisite: Modify your microservice(s).

• • Make code or other changes to your microservices.

Step 1: Generate New Static Coverage for Each Microservice.

• • This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

Step 2: Attach Coverage Agents to Each Microservice.

Step 3: Update CTP Environment with New Build Details.

Step 4: Call CTP REST API to Retrieve Impacted Tests.

Prerequisite: Modify your Microservice(s)

To conduct Test Impact Analysis, it is necessary to have a baseline build with coverage information from a full regression test run to compare against. Follow the Microservices Coverage Workflow defined in the previous section, including Step 4.c where CTP is configured with a baseline build id that will be used as a reference point for comparing with a target build in this workflow.

After changes have been made to your microservices that trigger a new target build, the following steps define how to integrate Test Impact Analysis into your test execution.

Step 1: Generate New Static Coverage for Each Microservice

Use the Step 1 of Microservices Coverage Workflow and generate new static coverage files for each microservice that changed, using new build IDs to describe the new builds that are to be tested.
These static coverage files will be published to DTP, which is how DTP will know what code changed between the baseline and target builds in order to calculate impacted tests.

Step 2: Attach Coverage Agents to Each Microservice

This part of the deploy process is no different from Step 2 of the Microservices Coverage Workflow. The updated microservices should get deployed to your test environment in preparation for the testing phase.

While it is technically not necessary to deploy coverage agents to run impacted tests, it is recommended to maintain a consistent deployment process where the coverage agents are part of the standard deployment to your test environment.

Step 3: Update CTP Environment with New Build Details

Similar to Step 3 of the Microservices Coverage Workflow, the same CTP environment should get updated about the new microservice builds during the deploy phase of the CI/CD process. It is important for CTP to have a reference of the new Build IDs of each microservice, where they match the Build IDs used to generate the new static coverage files from Step 1 of this workflow.

Since CTP needs to be updated every time new microservice builds are deployed into a test environment, it is recommended to use CTP's REST API to automate the updates during this deploy phase of the pipeline.

Step 4: Call CTP REST API to Retrieve Impacted Tests

At this point, both your test environment and Parasoft platform should be ready for your optimized test execution with Test Impact Analysis. Before triggering your test framework to run tests, you need to retrieve the list of impacted tests from CTP in order to parameterize your test framework with which Test IDs it should include in its execution. Send a REST request to the CTP API endpoint at

• GET /v3/environments/{environmentId}/coverage/impactedTests?baselineBuildId={baselineBuildId}

where environmentId is the ID of the environment used for both the baseline and the target (new) build. Use the baselineBuildId query parameter to specify the baseline build ID you chose in Step 4.c of the Microservices Coverage Workflow when running your baseline test build.

The endpoint will return a JSON list of tests which DTP has determined have been impacted by code changes between the two builds. In the case where there are multiple microservices with new builds, CTP will query DTP for impacted tests for each changed microservice and then eliminate duplicates so the correct set of impacted tests is returned for all changes across the distributed system that was newly deployed in the test environment. For example, here is a response consisting of a single impacted test:

[
  {
    "id": "5f6edb09-8e34-3885-aa7f-32435d3c8a8f",
    "testName": "first visit",
    "analysisType": "FT",
    "testSuiteName": "first visit",
    "toolName": "CTP"
  }
]

Troubleshooting

Takes a long time to create static coverage / process coverage

This usually happens when the system is processing more than it needs to. Check your includes and excludes and update them appropriately.