Versions Compared

Old Version 4

changes.mady.by.user Robert Andelin

Saved on

compared with

New Version 5

changes.mady.by.user Robert Andelin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This guide is intended to help you collect code coverage information for microservices developed with in .NET framework or Java technologyduring functional test execution and leverage that information to optimize subsequent test executions with test impact analysis.

The primary audience for this user guide is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

...

Next, create a system to represent the architecture of your application. Each microsystem microservice should have its own component in this system representation. You will come back to this System later in this guide to add an Environment, for now the System diagram is sufficient. For more details on creating systems in CTP, refer to the CTP user guide at at https://docs.parasoft.com.

...

Microservices Coverage Workflow
Anchor
MicroservicesCoverageWorkflow
MicroservicesCoverageWorkflow


Image RemovedImage Added

Build Phase

Step 1: Generate static coverage for each microservice.

...

    • CTP exposes a REST API to integrate with your test framework, allowing your test framework to notify CTP when tests are starting and stopping so that the appropriate code coverage can be collected for each individual test case that gets executed.

    • CTP exposes a REST API that can be triggered at the end of a testing session so that the runtime coverage data as well as test pass/fail results can be published to DTP.

    • (Optional) Set CTP baseline

Review Phase

Step 5: Review test reports and application coverage in DTP.

Step 1: Generate Static Coverage for Each Microservice
Anchor
GenerateStaticCoverage
GenerateStaticCoverage

Static coverage files are used to calculate the denominator of a code coverage metric. This tells us the total number of coverable lines, so that a percentage can be calculated when runtime coverage is measured during testing.

...

After generating static coverage, it should be visible on DTP. This can be seen as having some number of lines available (with zero overall covered; this will change once runtime coverage is uploaded to DTP in the next steps) on the coverage widget.

An image of a DTP dashboard showing the coverage and build administration widgets for four different projects corresponding to four different microservices. Note that each coverage widget displays 0 out of some variable number of lines covered.

Anchor
AttachCoverageAgents
AttachCoverageAgents
Step 2: Attach Coverage Agents to Each Microservice

Attaching the Coverage Agent to a microservice allows you to enable collecting dynamic (runtime) coverage for it. Typically, this is done as part of an automated deployment process coming from a CI/CD pipeline. 

...

Once started, you can check that the agent is running by going to the agent's status endpoint at http://<HOST>:<PORT>/status, where host is where the microservice is running and port is specified in agent.properties; by default, it is 8050.

Anchor
UpdateCTPEnvironment
UpdateCTPEnvironment
Step 3: Update CTP Environment with Coverage Agent Details

Go back to CTP Environment Manager, to the system you created during setup. Next create an environment for the system. Create component instances for at least each component representing a microservice that will have a coverage agent attached to it. Configuration for a coverage agent is done through the coverage tab of the component instance manager. Add the coverage agent's URL, as well as the DTP project, filter (optional), build ID, and coverage image settings; these DTP settings should be identical to the settings used when creating and uploading the corresponding static coverage for the microservice this component represents.

...

Step 4: Trigger Parasoft CTP APIs from Test Framework

4a: Record Runtime Coverage

To record To record runtime coverage from tests, use the CTP REST API to start or stop a coverage session, as well as record the results of individual tests. The full API definition can be viewed at http://<CTP_HOST>:<CTP_PORT>/em/apidoc, with the relevant section being the /agents group of REST endpoints.

Info
titleMultiuser Mode

You can collect coverage information for multiple users that are simultaneously accessing the same microservices. There are some additional steps necessary for this:

  • For .NET, launch the coverage agent with the -multiuser switch. Note that multiuser mode is only supported for web applications; standalone applications are not supported.
    Code Block
    dottest_iismanager.exe -multiuser
  • For Java, enable the coverage agent's multiuser mode by modifying the options in the agent.properties file. Enable both of the multiuser options as shown below.
    Code Block
    jtest.agent.enableMultiuserCoverage=true
jtest.agent.autoloadMultiuserLibs=true

  • If multiuser mode is enabled and you are using SOAtest, CTP, dotTEST, or Jtest 2024.1 or later, your tests need to inject the following HTTP request header: 
    Code Block
    baggage: test-operator-id=<USER>

...

  1. Create a coverage session by sending a REST request to either of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/session/start
    • GET /v3/environments/{environmentId}/agents/session/start

    where environmentId corresponds to the ID of the environment created and configured with coverage agents in the previous step. Make sure to keep the returned coverage session ID, since it will be used in the next step to upload coverage to DTP.

  2. Prior to starting a test, send a REST request to CTP via one of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/test/start
    • GET /v3/environments/{environmentId}/agents/test/start
    • GET /v3/environments/{environmentId}/agents/test/start/{testId}

    specifying the testId (the name) of the test that you are about to run. If you have multiple test cases per test, use the POST endpoint to specify the test case as well since it is the only endpoint that currently supports the ability to start individual test cases. Optionally, you may include an array of work items with the POST request in order to leverage requirements traceability in DTP.

  3. Run your test. The coverage agents attached to the microservices will collect coverage and associate it with the test (or test case).

  4. After stopping the test but before proceeding to the next one, send a REST request to CTP via one of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/test/stop
    • GET /v3/environments/{environmentId}/agents/test/stop
    • GET /v3/environments/{environmentId}/agents/test/stop/{testId}

    specifying the testId of the test you have finished running. If you are stopping an individual test case which is part of a larger test, use the POST endpoint to specify the test case as well as the test. This is also the REST request where you provide the test status: PASS, FAIL, or INCOMPLETE.

  5. After running all tests, stop the coverage session by sending a REST request to either of the following endpoints in the API:

    • POST /v3/environments/{environmentId}/agents/session/stop
    • GET /v3/environments/{environmentId}/agents/session/stop

4b: Upload Runtime Coverage to DTP

Info
titleIntegrating a Third-party Test Framework

For a better understanding of how to effectively integrate into a third-party test framework, see this example of the Spring Petclinic with Selenium tests on GitHub. 

...

  • POST /v3/environments/{environmentId}/coverage/{id}

where, as before, environmentId corresponds to the environment configured in previous steps and id corresponds to the coverage session generated in the previous step. This endpoint will send the runtime coverage from each coverage agent configured in the environment to DTP using the DTP fields specified in the coverage tab of the component instance manager (Project, Filter (optional), Build ID, Coverage Image).

...

The sessionTag in the POST request body must be unique each time results are published for a given build ID in order for DTP to aggregate the test results. This is especially useful when leveraging multi-user mode to collect coverage during parallel test execution where each parallel thread is represented as a different user executing a different set of tests.

Note

It is critical to ensure the DTP Project, Build ID, and Coverage Image Tags in the environment match the properties used to generate your static coverage body match the properties used to generate your static coverage. Otherwise, DTP may fail to associate the runtime and static coverage.

Step 5: Review Application Coverage on DTP

Coverage statistics should now be visible in DTP for the microservice projects.

Image Removed

By clicking on a coverage widget, we can drill down and view coverage statistics not only for the whole project but individual files and methods as well.

A screenshot of DTP's coverage explorer showing coverage percentages for individual files and folders.Image Removed

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow

You can use CTP in conjunction with DTP to not only gather coverage information for your microservices for a particular build, but also use them to conduct Test Impact Analysis: examining the coverage data of two different builds to determine which tests have been impacted by code changes between the two builds. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes they made in a build.

Step 1: Use the microservices coverage workflow to establish a baseline.

    • A baseline build in DTP will be used to compare against a future build.

Step 2: Modify your microservice(s).

    • Make code or other changes to your microservices.

Step 3: Use the microservices coverage workflow to generate new static coverage with your changes.

    • This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

Step 4: Use the CTP REST API to connect with DTP and get a list of impacted tests.

...

4c: Set CTP Baseline (Optional)
Anchor
SetCTPBaseline
SetCTPBaseline

This step is only required if you are implementing the Test Impact Analysis workflow outlined in the following section. Test Impact Analysis requires a baseline build with coverage from a full regression test run to use for comparison when calculating impacted tests from a subsequent build. To mark a build as a baseline build to be compared against, send an additional REST request at the end of your test session to the CTP API endpoint at

  • POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

Note

It is important to note that DTP has a rolling log of how much build data it retains in its database and file system (see Data Retention Settings in DTP for details). When baselining a build to be referenced by many builds in the future, it is important that the build be archived in DTP, so the data is available when CTP queries DTP to retrieve impacted tests. If a build important to your baseline is not archived and new builds get published to the DTP project exceeding the project's rolling log settings, build data will be lost preventing the test impact analysis workflow from functioning.

If you are using CTP 2024.1 or earlier, see Locking and Archiving Builds in DTP for more information about archiving builds in DTP. DTP also has a REST endpoint that allows you to dynamically archive and unarchive builds as appropriate to your process:

Code Block
PUT /v12/builds/{id}/persistence

Starting with version 2024.2, CTP automatically archives DTP builds when creating a baseline, tagging these builds with the description "Archived by CTP" in DTP. Be aware that when someone calls

Code Block
DELETE /v3/environments/{environmentId}/coverage/baselines/{id}

and passes ?unarchive=true to delete a baseline, CTP will automatically unarchive associated builds tagged with the "Archived by CTP" description. To protect against a build accidentally getting unarchived this way, change the description for the build in DTP.

Step 5: Review Application Coverage on DTP

Coverage statistics should now be visible in DTP for the microservice projects.

Image Added

By clicking on a coverage widget, we can drill down and view coverage statistics not only for the whole project but individual files and methods as well.

A screenshot of DTP's coverage explorer showing coverage percentages for individual files and folders.Image Added

Notice the screenshot shows "Source code not available." This is because the screenshot was taken with a coverage workflow using application binaries instead of source code. To see source code with red/green line coverage markers, the Jtest or dotTEST products are required as they produce the static coverage from sources instead of binaries and transmit the source code in addition to static coverage to DTP for viewing.

Microservices Test Impact Analysis Workflow
Anchor
TIAWorkflow
TIAWorkflow

You can use CTP and DTP to not only measure code coverage of microservices during functional testing, but also to enable Test Impact Analysis: an automated process for optimizing which tests are included for execution based on code changes in an application. Code coverage from test executions against a baseline build is used in conjunction with a code diff between that baseline build and a target build to calculate which tests are impacted for testing the target build. Test Impact Analysis enables a dramatic reduction of how many tests you can choose to run by showing you which tests can be skipped due to those tests not covering any of the changed code from your baseline build. It is a valuable technique when grappling with very long testing jobs that prevent fast feedback to development about changes made in a build.

Image Added

Prerequisite: Modify your microservice(s).

    • Make code or other changes to your microservices.

Step 1: Generate New Static Coverage for Each Microservice.

    • This new coverage information will be compared against the baseline to see which tests have been impacted by your changes.

Step 2: Attach Coverage Agents to Each Microservice.

Step 3: Update CTP Environment with New Build Details.

Step 4: Call CTP REST API to Retrieve Impacted Tests.

Anchor
Sect4Step1
Sect4Step1
Prerequisite: Modify your Microservice(s)

To conduct Test Impact Analysis, it is necessary to have a baseline build with coverage information from a full regression test run to compare against. Follow the Microservices Coverage Workflow defined in the previous section, including Step 4.c where CTP is configured with a baseline build id that will be used as a reference point for comparing with a target build in this workflow.

After changes have been made to your microservices that trigger a new target build, the following steps define how to integrate Test Impact Analysis into your test execution.

Step 1: Generate New Static Coverage for Each Microservice

Use the Step 1 of Microservices Coverage Workflow and generate new static coverage files for each microservice that changed, using new build IDs to describe the new builds that are to be tested.
These static coverage files will be published to DTP, which is how DTP will know what code changed between the baseline and target builds in order to calculate impacted tests.

Step 2: Attach Coverage Agents to Each Microservice

This part of the deploy process is no different from Step 2 of the Microservices Coverage Workflow. The updated microservices should get deployed to your test environment in preparation for the testing phase.

While it is technically not necessary to deploy coverage agents to run impacted tests, it is recommended to maintain a consistent deployment process where the coverage agents are part of the standard deployment to your test environment.

Step 3: Update CTP Environment with New Build Details

Similar to Step 3 of the Microservices Coverage Workflow, the same CTP environment should get updated about the new microservice builds during the deploy phase of the CI/CD process. It is important for CTP to have a reference of the new Build IDs of each microservice, where they match the Build IDs used to generate the new static coverage files from Step 1 of this workflow.

Since CTP needs to be updated every time new microservice builds are deployed into a test environment, it is recommended to use CTP's REST API to automate the updates during this deploy phase of the pipeline

To conduct Test Impact Analysis, it is necessary to have a baseline build in DTP with coverage information to compare against. Use the Microservices Coverage Workflow outlined above to first generate coverage. Then, send a REST request to the CTP API endpoint at

  • POST /v3/environments/{environmentId}/coverage/baselines/{id}

where environmentId is the environment configured as part of your Microservices Coverage Workflow and id is your chosen name to identify this baseline.

There can be different cadences for when a baseline is set at the end of a testing session where the Parasoft coverage workflow has been integrated into it. A baseline should be set whenever a complete regression test run is performed, whether that happens nightly, weekly, or some other frequency. This is because you want a complete code coverage mapping of all your test cases to compare with when Parasoft analyzes code changes between builds. CTP and DTP can maintain multiple baselines for comparison, where it may be desirable to compare code changes from logical points in the development process. Customers typically choose to baseline from the beginning of a new release cycle, the beginning of a sprint, or as aggressively as the latest full regression test run.

Note

It is important to note that DTP has a rolling log of how much build data it retains in its database and file system (see Data Retention Settings in DTP for details). When baselining a build to be referenced by many builds in the future, it is important that you archive the build in DTP, so the data is available when CTP queries DTP to retrieve impacted tests. If a build important to your baseline is not archived and new builds get published to the DTP project exceeding the project's rolling log settings, build data will be lost preventing the test impact analysis workflow from functioning. See Locking and Archiving Builds in DTP for more information. DTP has a REST endpoint that allows you to dynamically archive and unarchive builds as appropriate to your process.

  • PUT /v12/builds/{id}/persistence

Step 2: Modify your Microservice(s)

Make code or other changes to your microservices.

Step 3: Use the Microservices Coverage Workflow to Generate New Static Coverage with Your Changes

Use the Microservices Coverage Workflow and generate a new static coverage using new build ID to compare against the baseline build.

Be sure to use the same environment as in Step 1 and update the DTP Build IDs field in the coverage tab of your component instances configured for coverage agents to new Build IDs. Since updating the coverage agent information in the CTP Environment is typically done during the deploy phase, your pipeline for build + deploy should run prior to asking for impacted tests so that DTP has static coverage for the new builds and CTP has updated build.id references for the new builds as well.

Note

The Test Impact Analysis workflow in CTP requires configuring the optional Filter setting in CTP's coverage settings for a Component Instance. DTP projects can have a number of Filters that DTP uses are used to segment the data it is reporting. By default, each DTP project starts with one filter that auto-includes all reports published to that project. If you are using additional DTP filtersincludes all reports published to that project, and the CTP component instance UI will initialize to this filter by default. If updating CTP Environments via REST API, be sure the Filter setting is retained. For advanced users that use multiple DTP filters per project, be sure to configure reference a filter that contains the coverage reporting that CTP publishes to the corresponding DTP projects for each microservice.

Step 4: Use the CTP REST API to Connect with DTP and Get a List of Impacted Tests

CTP publishes to the corresponding DTP projects for each microservice.

Step 4: Call CTP REST API to Retrieve Impacted Tests

At this point, both your test environment and Parasoft platform should be ready for your optimized test execution with Test Impact Analysis. Before triggering your test framework to run tests, you need to retrieve the list of impacted tests from CTP in order to parameterize your test framework with which Test IDs it should include in its execution. Send Once DTP receives static coverage for a new build with your changes, send a REST request to the CTP API endpoint at

...

where environmentId is the ID of the environment used for both the baseline and the target (new) build. Use the baselineBuildId query parameter to specify the baseline build ID you chose in Step 14.c of the Microservices Coverage Workflow when running your baseline test build.

The endpoint will return a JSON list of tests which DTP has determined have been impacted by code changes between the two builds. In the case where there are multiple microservices with new builds, CTP will query DTP for impacted tests for each changed microservice and then eliminate duplicates so the correct set of impacted tests is returned for all changes across the distributed system that was newly deployed in the test environment. For example, here is a response consisting of a single impacted test:

...