Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This guide is intended to help you collect code coverage information for microservices developed with in .NET framework or Java technologyduring functional test execution and leverage that information to optimize subsequent test executions with test impact analysis.

The primary audience for this user guide is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

...

Next, create a system to represent the architecture of your application. Each microsystem microservice should have its own component in this system representation. You will come back to this System later in this guide to add an Environment, for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at at https://docs.parasoft.com.

...

Microservices Coverage Workflow
Anchor
MicroservicesCoverageWorkflow
MicroservicesCoverageWorkflow


Image RemovedImage Added

Build Phase

Step 1: Generate static coverage for each microservice.

...

    • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.
    • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

    • (Optional) Create baseline build in CTP

Review Phase

Step 5: Review test reports and application coverage in DTP.

Step 1: Generate Static Coverage for Each Microservice
Anchor
GenerateStaticCoverage
GenerateStaticCoverage

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

...

The jtestcov tool can be found in the downloaded java_agent_coverage zip, under the jtestcov directory. You will need to create a properties file to add DTP Properties for coverage information. See DTP Properties for coverage section for details.

...

Code Block
java -jar <PATH_TO_jtestcov.jar> -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license TO .PROPERTIES>/jtestcov.properties
Info

The -ctp flag is used by the coverage tools to pass your CTP license, which is needed when you run the coverage tool independently of CTP as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched. You can see all available commands and options by running: java -jar jtestcov.jar help

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using dottestcov.bat with a command like the example below:

Code Block
dottestcov.bat -ctp -include "C:\Devel\FooSolution\src\QuxProject***.cs" -exclude "C:\Devel\FooSolution\src\QuxProject*\tests**.cs" -app <DIR>\FooSolution.sln -publish -settings <PATH TO .PROPERTIES>\dottestcov.properties
Info

The -ctp flag is used by the coverage tools to pass your CTP license, which is needed when you run the coverage tool independently of CTP as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched.

Code Block
titleSample License Properties File
# === LICENSE ===
Code Block
titleSample License Properties File
# === LICENSE ===
 
# === END USER LICENSE AGREEMENT ===
# Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
#parasoft.eula.accepted=false
 
# === END LOCALUSER LICENSE AGREEMENT ===
# Enables local license - be sure to specify password.
#jtest.license.use_network=false
 
# Specifies password for the local license.
#jtest.license.local.password=[LICENSE PASSWORD]
 Set to true to accept the Parasoft End User License Agreement (EULA).
# Please review the EULA.txt file included in the product installation directory.
parasoft.eula.accepted=true

# === NETWORK LICENSE ===
# Enables network license - be sure to configure DTP server settings.
#jtestctp.license.use_network=true
 ctp.license.network.edition=custom_edition
# SpecifiesNote: typecustomize ofthe theCTP networkcoverage license (edition).
# Supported editions: desktop_edition, desktop_compliance_edition, server_edition, server_compliance_edition
#jtest.license.network.edition=server_edition
 
# Enables specific list of license features
#jtest.license.network.edition=custom_edition
#jtestagent tier based on what your CTP license has enabled
ctp.license.custom_edition_features=JtestCTP, Automation, Desktop Command Line,Coverage Tier 5 

# === DTP Publish,SERVER Coverage, Test Impact Analysis
 
 
# === DTP SERVER SETTINGS ===SETTINGS ===
 
# Specifies URL of the DTP server in the form https://host[:port][/context-path]
#dtp.url=https://localhost:8443
 
# Specifies user name for DTP server authentication.
#dtp.user=admin
 
# Specifies password for DTP server authentication - use jtestcli -encodepass <PASSWORD> to encode the password, if needed.
#dtp.password=admin
 
# Specifies name of the DTP project - this settings is optional.
#dtp.project=[DTP Project Name]
 
# === DTP REPORTING ===
 
# Enables reporting test results to DTP server - be sure to configure DTP server settings.
#report.dtp.publish=true
 
# Specifies a build identifier used to label results. It may be unique for each build
# but may also label more than one test sessions that were executed during a specified build.
#build.id=${dtp_project}-yyyy-MM-dd
 
# Specifies a tag which represents an unique identifier for the run, used to distinguish it from similar runs.
# It could be constructed as minimal combination of following variables that will make it unique or specified manually.
# e.g. ${config_name}-${project_module}-${scontrol_branch}-${exec_env}
#session.tag=[tag]
 
# Specifies a set of tags that will be used to create coverage images in DTP server.
# Coverage images allow you to track different types of coverage, such as coverage for unit, functional, manual tests and others.
# There is a set of predefined tags that will be automatically recognized by DTP, see the examples below.
# You can also specify other tags that will be used to create coverage images.
#report.coverage.images=${dtp_project}
#report.coverage.images=${dtp_project};${dtp_project}_Unit Test
#report.coverage.images=${dtp_project};${dtp_project}_Functional Test
#report.coverage.images=${dtp_project};${dtp_project}_Manual Test
 
# === CONSOLE VERBOSITY LEVEL ===
# Increases console verbosity level to high.
#console.verbosity.level=high

Generating static coverage with dottestcov

The dottestcov tool is used similarly to the jtestcov tool detailed above. It can be found in the downloaded dotnet_agent_coverage zip, under the dottestcov directory.

Invoke dottestcov using either dottestcov.sh or dottestcov.bash, depending on your operating system, with a command like the example below running against Parabank:

Code Block
dottestcov(.sh/.bash) coverage -ctp -app c:/<PATH_TO_APPLICATION>/parabank.war -include com/parasoft/parabank/parasoft/** -settings c:/<PATH_TO_LICENSE_FILE>/license.properties
Info

The -ctp flag is used by the coverage tools to pass your CTP license, which is needed when you run the coverage tool independently of CTP as in the example above. When you run soatestcli with application coverage settings, the license is automatically passed when the coverage tool is launched.

...

Anchor
NoteIncludesExcludes
NoteIncludesExcludes

Note
titleA Note About Includes and Excludes

Defining includes and excludes is an important part of controlling how much your system processes, which can greatly affect how long it takes to generate your static coverage files. Includes and excludes should be expressed as comma-separated lists of patterns that specify classes to be instrumented. The following wildcards are supported:

* matches zero or more characters
** matches multiple directory levels

In the following example, all classes from the com.myapp.data package and all classes from package and subpackages that start with com.myapp.common will be instrumented:

-include com/myapp/data/*,com/myapp/common/**

While in this example, all classes from the com.myapp.transport package and all classes from package and subpackages that start with com.myapp.autogen will be excluded from instrumentation:

-exclude com/myapp/transport/*,com/myapp/autogen/**

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

An image of a DTP dashboard showing the coverage and build administration widgets for four different projects corresponding to four different microservices. Note that each coverage widget displays 0 out of some variable number of lines covered.Image Added

Anchor
AttachCoverageAgents
AttachCoverageAgents
Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

Code Block
 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

  1. Run runCamAgent.bat to start the coverage agent.
  2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow

...

Note
titleA Note About Includes and Excludes

Defining includes and excludes is an important part of controlling how much your system processes, which can greatly affect how long it takes to generate your static coverage files. Includes and excludes should be expressed as comma-separated lists of patterns that specify classes to be instrumented. The following wildcards are supported:

* matches zero or more characters
** matches multiple directory levels

In the following example, all classes from the com.myapp.data package and all classes from package and subpackages that start with com.myapp.common will be instrumented:

-include com/myapp/data/*,com/myapp/common/**

While in this example, all classes from the com.myapp.transport package and all classes from package and subpackages that start with com.myapp.autogen will be excluded from instrumentation:

-exclude com/myapp/transport/*,com/myapp/autogen/**

Verifying static coverage has been uploaded to DTP

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

An image of a DTP dashboard showing the coverage and build administration widgets for four different projects corresponding to four different microservices. Note that each coverage widget displays 0 out of some variable number of lines covered.Image Removed

Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

Attaching the Coverage Agent for Java

The agent.jar file for the agent as well as the agent.properties file for configuring it can be found in the downloaded java_agent_coverage zip, under the jtest_agent directory.

An argument of the form:

Code Block
 -javaagent:"<PATH_TO_AGENT_DIR>\agent.jar"=settings="<PATH_TO_AGENT_PROPERTIES_FILE>\agent.properties",runtimeData="<PATH_TO_RUNTIME_DIR>\runtime_dir"

needs to be added to the microservice's invocation or startup script in order to attach the agent to the microservice.

Once started, you can check that the agent is running by going to the agent's status endpoint at at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Attaching a Coverage Agent for .NET

You can use the Coverage Wizard tool found in the dottest_agent folder of the downloaded dotnet_agent_coverage zip to set up coverage agents. It can be invoked directly to provide agent configuration via a GUI, or alternatively run using the command line; see the documentation here for more details: Application Coverage for Standalone Applications (dotTEST) or Application Coverage for Web Applications (dotTEST).

The Coverage Wizard will generate a new folder containing the scripts necessary to run and attach the coverage agent to the microservice it is configured for.

  1. Run runCamAgent.bat to start the coverage agent.
  2. Run monitorcoverage.bat to start the microservice.

Do not stop the coverage agent process until you are completely done with any coverage workflow.

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Step 3: Update CTP Environment with Coverage Agent Details

Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

An image of the coverage tab of the component instance manager for a component of the same system as the previous image. The Agent URL, DTP Project, Filter, Build ID, and Coverage Image fields are filled out.Image Removed

If desired, you can additionally fill out other details for component instances. For example, filling out the real endpoints section of the components tab in the component instance manager will allow you to see the status of that endpoint from the environment diagram.

After environment configuration is complete, you can check to make sure all necessary components have their coverage agents configured and connected; components with coverage agents are denoted with a blue C symbol, and a green checkmark signifies a good status for the connection.

An image of the configured environment for the same system as the previous images. Four of the components have coverage agents configured in their component instances, denoted by a blue C symbol, with good connections denoted by a green checkmark.Image Removed

Now that the Environment in CTP has been fully setup via the GUI, the CTP Components or the entire Environment can be automatically synchronized from your deployment pipeline via REST API. For example, when a new version of your microservice is built and deployed, it will have a new Build ID that CTP needs to have a reference to. The microservice's deployment endpoint, and thus coverage agent endpoint, may also change, and synchronizing CTP with the current state of your deployed microservices is essential in correctly measuring code coverage and associating it with the appropriate builds that have been deployed. If automation drives the deployment of your applications into a test environment, it is recommended to automate updating CTP at the same time.

You can use the following APIs to update CTP:

  • GET - /v3/environments/{environmentId}/config
  • PUT - /v3/environments/{environmentId}/config 

Step 4: Trigger Parasoft CTP APIs from Test Framework

To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the /agents group of REST endpoints.

...

titleMultiuser Mode

You can collect coverage information for multiple users that are simultaneously accessing the same microservices. There are some additional steps necessary for this:

...

Code Block
dottest_iismanager.exe -multiuser

...

Code Block
jtest.agent.enableMultiuserCoverage=true
jtest.agent.autoloadMultiuserLibs=true

Anchor
UpdateCTPEnvironment
UpdateCTPEnvironment
Step 3: Update CTP Environment with Coverage Agent Details

Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

An image of the coverage tab of the component instance manager for a component of the same system as the previous image. The Agent URL, DTP Project, Filter, Build ID, and Coverage Image fields are filled out.Image Added

If desired, you can additionally fill out other details for component instances. For example, filling out the real endpoints section of the components tab in the component instance manager will allow you to see the status of that endpoint from the environment diagram.

After environment configuration is complete, you can check to make sure all necessary components have their coverage agents configured and connected; components with coverage agents are denoted with a blue C symbol, and a green checkmark signifies a good status for the connection.

An image of the configured environment for the same system as the previous images. Four of the components have coverage agents configured in their component instances, denoted by a blue C symbol, with good connections denoted by a green checkmark.Image Added

Now that the Environment in CTP has been fully setup via the GUI, the CTP Components or the entire Environment can be automatically synchronized from your deployment pipeline via REST API. For example, when a new version of your microservice is built and deployed, it will have a new Build ID that CTP needs to have a reference to. The microservice's deployment endpoint, and thus coverage agent endpoint, may also change, and synchronizing CTP with the current state of your deployed microservices is essential in correctly measuring code coverage and associating it with the appropriate builds that have been deployed. If automation drives the deployment of your applications into a test environment, it is recommended to automate updating CTP at the same time.

You can use the following APIs to update CTP:

  • GET - /v3/environments/{environmentId}/config
  • PUT - /v3/environments/{environmentId}/config 

Step 4: Trigger Parasoft CTP APIs from Test Framework

4a: Record Runtime Coverage

To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the /agents group of REST endpoints.

Info
titleMultiuser Mode

You can collect coverage information for multiple users that are simultaneously accessing the same microservices. There are some additional steps necessary for this:

  • For .NET, launch the coverage agent with the -multiuser switch. Note that multiuser mode is only supported for web applications; standalone applications are not supported.
    Code Block
    dottest_iismanager.exe -multiuser
  • For Java, enable the coverage agent's multiuser mode by modifying the options in the agent.properties file. Enable both of the multiuser options as shown below.
    Code Block
    jtest.agent.enableMultiuserCoverage=true
    jtest.agent.autoloadMultiuserLibs=true

  • If multiuser mode is enabled and you are using SOAtest, CTP, dotTEST, or Jtest 2024.1 or later, your tests need to inject the following HTTP request header: 
    Code Block
    baggage: test-operator-id=<USER>

The workflow goes as follows:

  1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/session/start
    • GET /v3/environments/{environmentId}/agents/session/start

    where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

  2. Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/test/start
    • GET /v3/environments/{environmentId}/agents/test/start
    • GET /v3/environments/{environmentId}/agents/test/start/{testId}

    specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases. Optionally, you may include an array of work items with the POST request in order to leverage requirements traceability in DTP.

  3. Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

  4. After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/test/stop
    • GET /v3/environments/{environmentId}/agents/test/stop
    • GET /v3/environments/{environmentId}/agents/test/stop/{testId}

    specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

  5. After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

    • POST /v3/

...

Code Block
baggage: test-operator-id=<USER>

The workflow goes as follows:

  1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/session/startstop
    • GET /v3/environments/{environmentId}/agents/session/start

    where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

    Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API
    • stop

4b: Upload Runtime Coverage to DTP

Info
titleIntegrating a Third-party Test Framework

For a better understanding of how to effectively integrate into a third-party test framework, see this example of the Spring Petclinic with Selenium tests on GitHub

To upload the runtime coverage session generated by the previous steps to DTP, we will again use the CTP REST API. Make a REST request to the endpoint:

  • POST /v3/environments/{environmentId}/

...

specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases.

...

Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

...

After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

  • POST /v3/environments/{environmentId}/agents/test/stop
  • GET /v3/environments/{environmentId}/agents/test/stop
  • GET /v3/environments/{environmentId}/agents/test/stop/{testId}

specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

...

After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

  • POST /v3/environments/{environmentId}/agents/session/stop
  • GET /v3/environments/{environmentId}/agents/session/stop
  • coverage/{id}

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image). The sessionTag in the POST request body must be unique each time results are published for a given build ID in order for DTP to aggregate the test results. This is especially useful when leveraging multi-user mode to collect coverage during parallel test execution where each parallel thread is represented as a different user executing a different set of tests.

Note

It is critical to ensure the DTP Project, Build ID, and Coverage Image Tags in the environment match the properties used to generate your static coverage. Otherwise, DTP may fail to associate the runtime and static coverage.

4c: Create baseline build in CTP (Optional)
Anchor
SetCTPBaseline
SetCTPBaseline

This step is only required if you are implementing the Test Impact Analysis workflow outlined in the following section. Test Impact Analysis requires a baseline build with coverage from a full regression test run to use for comparison when calculating impacted tests from a subsequent build. To mark a build as a baseline build to be compared against, send an additional REST request at the end of your test session to the CTP API endpoint at

  • POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

Note

It is important to note that DTP has a rolling log of how much build data it retains in its database and file system (see Data Retention Settings in DTP for details). When baselining a build to be referenced by many builds in the future, it is important that the build be archived in DTP, so the data is available when CTP queries DTP to retrieve impacted tests. If a build important to your baseline is not archived and new builds get published to the DTP project exceeding the project's rolling log settings, build data will be lost preventing the test impact analysis workflow from functioning.

If you are using CTP 2024.1 or earlier, see Locking and Archiving Builds in DTP for more information about archiving builds in DTP. DTP also has a REST endpoint that allows you to dynamically archive and unarchive builds as appropriate to your process:

Code Block
PUT /v12/builds/{id}/persistence

Starting with version 2024.2, CTP automatically archives DTP builds when creating a baseline, tagging these builds with the description "Archived by CTP" in DTP. Be aware that when someone calls

Code Block
DELETE 
Info
titleIntegrating a Third-party Test Framework

For a better understanding of how to effectively integrate into a third-party test framework, see this example of the Spring Petclinic with Selenium tests on GitHub

To upload the runtime coverage session generated by the previous steps to DTP, we will again use the CTP REST API. Make a REST request to the endpoint:

...

/v3/environments/{environmentId}/coverage

...

/baselines/{id}

and passes ?unarchive=true to delete a baseline, CTP will automatically unarchive associated builds tagged with the "Archived by CTP" description. To protect against a build accidentally getting unarchived this way, change the description for the build in DTP

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image).

Note

It is critical to ensure the DTP Project, Build ID, Coverage Image Tags, and Session Tag in the POST body match the properties used to generate your static coverage. Otherwise, DTP may fail to associate the runtime and static coverage.

Step 5: Review Application Coverage on DTP

...

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow
Anchor
TIAWorkflow
TIAWorkflow

You can use CTP in conjunction with and DTP to not only gather coverage information for your microservices for a particular buildmeasure code coverage of microservices during functional testing, but also use them to conduct enable Test Impact Analysis: examining the coverage data of two different builds to determine which tests have been impacted by code changes between the two buildsan automated process for optimizing which tests are included for execution based on code changes in an application. Code coverage from test executions against a baseline build is used in conjunction with a code diff between that baseline build and a target build to calculate which tests are impacted for testing the target build. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes they made in a build.

Step 1: Use the microservices coverage workflow to establish a baseline.

    • A baseline build in DTP will be used to compare against a future build.

Step 2: Modify your microservice(s).

    • Make code or other changes to your microservices.

Step 3: Use the microservices coverage workflow to generate new static coverage with your changes.

    • This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

Step 4: Use the CTP REST API to connect with DTP and get a list of impacted tests.

...

tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes made in a build.

Image Added

Prerequisite: Modify your microservice(s).

    • Make code or other changes to your microservices.

Step 1: Generate New Static Coverage for Each Microservice.

    • This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

Step 2: Attach Coverage Agents to Each Microservice.

Step 3: Update CTP Environment with New Build Details.

Step 4: Call CTP REST API to Retrieve Impacted Tests.

Anchor
Sect4Step1
Sect4Step1
Prerequisite: Modify your Microservice(s)

To conduct Test Impact Analysis, it is necessary to have a baseline build with coverage information from a full regression test run to compare against. Follow the Microservices Coverage Workflow defined in the previous section, including Step 4.c where CTP is configured with a baseline build id that will be used as a reference point for comparing with a target build in this workflow.

After changes have been made to your microservices that trigger a new target build, the following steps define how to integrate Test Impact Analysis into your test execution.

Step 1: Generate New Static Coverage for Each Microservice

Use the Step 1 of Microservices Coverage Workflow and generate new static coverage files for each microservice that changed, using new build IDs to describe the new builds that are to be tested.
These static coverage files will be published to DTP, which is how DTP will know what code changed between the baseline and target builds in order to calculate impacted tests.

Step 2: Attach Coverage Agents to Each Microservice

This part of the deploy process is no different from Step 2 of the Microservices Coverage Workflow. The updated microservices should get deployed to your test environment in preparation for the testing phase.

While it is technically not necessary to deploy coverage agents to run impacted tests, it is recommended to maintain a consistent deployment process where the coverage agents are part of the standard deployment to your test environment.

Step 3: Update CTP Environment with New Build Details

Similar to Step 3 of the Microservices Coverage Workflow, the same CTP environment should get updated about the new microservice builds during the deploy phase of the CI/CD process. It is important for CTP to have a reference of the new Build IDs of each microservice, where they match the Build IDs used to generate the new static coverage files from Step 1 of this workflow.

Since CTP needs to be updated every time new microservice builds are deployed into a test environment, it is recommended to use CTP's REST API to automate the updates during this deploy phase of the pipeline

To conduct Test Impact Analysis, it is necessary to have a baseline build in DTP with coverage information to compare against. Use the Microservices Coverage Workflow outlined above to first generate coverage. Then, send a REST request to the CTP API endpoint at

  • POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

Note

It is important to note that DTP has a rolling log of how much build data it retains in its database and file system (see Data Retention Settings in DTP for details). When baselining a build to be referenced by many builds in the future, it is important that you archive the build in DTP, so the data is available when CTP queries DTP to retrieve impacted tests. If a build important to your baseline is not archived and new builds get published to the DTP project exceeding the project's rolling log settings, build data will be lost preventing the test impact analysis workflow from functioning. See Locking and Archiving Builds in DTP for more information. DTP has a REST endpoint that allows you to dynamically archive and unarchive builds as appropriate to your process.

  • PUT /v12/builds/{id}/persistence

Step 2: Modify your Microservice(s)

Make code or other changes to your microservices.

Step 3: Use the Microservices Coverage Workflow to Generate New Static Coverage with Your Changes

Use the Microservices Coverage Workflow and generate a new static coverage using new build ID to compare against the baseline build.

Be sure to use the same environment as in Step 1 and update the DTP Build IDs field in the coverage tab of your component instances configured for coverage agents to new Build IDs. Since updating the coverage agent information in the CTP Environment is typically done during the deploy phase, your pipeline for build + deploy should run prior to asking for impacted tests so that DTP has static coverage for the new builds and CTP has updated build.id references for the new builds as well.

Note

The Test Impact Analysis workflow in CTP requires configuring the optional Filter setting in CTP's coverage settings for a Component Instance. DTP projects can have a number of Filters that DTP uses are used to segment the data it is reporting. By default, each DTP project starts with one filter that auto-includes all reports published to that project. If you are using additional DTP filtersincludes all reports published to that project, and the CTP component instance UI will initialize to this filter by default. If updating CTP Environments via REST API, be sure the Filter setting is retained. For advanced users that use multiple DTP filters per project, be sure to configure reference a filter that contains the coverage reporting that CTP publishes to the corresponding DTP projects for each microservice.

Step 4: Use the CTP REST API to Connect with DTP and Get a List of Impacted Tests

CTP publishes to the corresponding DTP projects for each microservice.

Step 4: Call CTP REST API to Retrieve Impacted Tests

At this point, both your test environment and Parasoft platform should be ready for your optimized test execution with Test Impact Analysis. Before triggering your test framework to run tests, you need to retrieve the list of impacted tests from CTP in order to parameterize your test framework with which Test IDs it should include in its execution. Send Once DTP receives static coverage for a new build with your changes, send a REST request to the CTP API endpoint at

...

where environmentId is the ID of the environment used for both the baseline and the target (new) build. Use the baselineBuildId query parameter to specify the baseline build ID you chose in Step 14.c of the Microservices Coverage Workflow when running your baseline test build.

The endpoint will return a JSON list of tests which DTP has determined have been impacted by code changes between the two builds. In the case where there are multiple microservices with new builds, CTP will query DTP for impacted tests for each changed microservice and then eliminate duplicates so the correct set of impacted tests is returned for all changes across the distributed system that was newly deployed in the test environment. For example, here is a response consisting of a single impacted test:

...