...
| Table of Contents | ||||
|---|---|---|---|---|
|
Introduction
The Parasoft Findings extension is designed to be used in a post-test execution build step that integrates functional and unit test reports as well as static analysis reports and code coverage reports, including support for quality gate checks, generated by Parasoft tools into your Azure DevOps project. The extension can consume the following report types:
...
Requirements
- Agent version 2.144209.0 or higher.
Installing the Parasoft Findings Extension
...
| Anchor | ||||
|---|---|---|---|---|
|
Configuring the Publish Parasoft Results Task
- Edit an existing pipeline or create a new pipeline and click Add Task.
- Search for "Publish Parasoft Results" using the search field.
- Select the extension and click Add.
- Review the Results Files field. The predefined minimatch pattern is configured to read the test results, static analysis, and code coverage reports generated by Parasoft tools. You can modify or remove parts of the pattern to suit your needs. See Microsoft's documentation on file matching patterns for more information on working with matching patterns.
- Static Analysis reports require you to install an extension to Azure DevOps to view; see Viewing Static Analysis Reports for more information.
- (Optional but strongly recommended) Configure the Root path to Parasoft tool/Java installation field. You can enter the absolute location of the root path of a Parasoft tool (Parasoft tools contain Jtest, dotTEST or C/C++test contains a Java installation) or a standalone Java installation to significantly optimize the efficiency of report processing and avoid potential memory issues.
- (Optional) Configure the Reference pipeline field. If you want to use specific pipeline as a reference, specify the reference pipeline name. To use current pipeline, leave this field empty.
(Optional) Configure the Reference build field. If you want to use specific build as a reference, specify the reference build number. To use the last successful build, leave this field empty.
Info title About Reference Pipelines and Builds Reference pipeline and Reference build are not supported in a release pipeline.
If you are not using quality gates, the reference pipeline and build can be used to filter results by Baseline on the Scans tab.
If you are using quality gates, they can be used as a baseline build when the quality gate needs one.
- Choose Save & Queue > Save.
| Anchor | ||||
|---|---|---|---|---|
|
You can set multiple quality gates that evaluate after issues are reported. These gates adjust the build status based on desired product quality, setting builds to 'unstable' or 'failed'. Each gate measures against a specific metric: the number of issues of a certain type and severity. This
...
feature is not supported in a Release pipeline.
- Edit an existing pipeline or create a new one and click Add Task.
- Search for "Parasoft Static Analysis Quality Gate" using the search field.
- Select the extension and click Add.
- Configure the following properties as needed.
- Type: Type of issues to be evaluated for the threshold.
- Severity: Severity of issues to be evaluated for the threshold.
- Threshold: The threshold defines the minimum number of issues that will fail a build. If the actual number of issues is greater than or equal to this threshold, then a build is considered unstable or failed, respectively. Default value is 0.
- Build status if quality gate fails: Determines if the build should be set to unstable or failed if the quality gate is not passed.
- Choose Save & Queue > Save.
| Note |
|---|
You need to set a baseline by running the Publish Parasoft Results task first in order to perform quality gate checks. |
| Anchor | ||||
|---|---|---|---|---|
|
You can set multiple quality gates that evaluate after coverage results are reported. These gates adjust the build status based on desired product quality, setting builds to 'unstable' or 'failed'. Each gate measures against the value of code coverage of a certain type. This feature is not supported in a Release pipeline.
Edit an existing pipeline or create a new pipeline and click Add Task.
Search for "Parasoft Code Coverage Quality Gate" using the search field.
Select the extension and click Add.
- Configure the following properties as needed:
- Type: Type of code coverage to be evaluated for the threshold.
- Threshold: The threshold defines the actual value of code coverage that will fail a build. If the actual value of code coverage is less than this threshold, then a build is considered unstable or failed, respectively. Default value is 0.0.
- Build status if quality gate fails: Determines if the build should be set to unstable or failed if the quality gate is not passed.
- Choose Save & Queue > Save.
| Note |
|---|
You need to set a baseline by running the Publish Parasoft Results task first in order to perform quality gate checks. |
| Anchor | ||||
|---|---|---|---|---|
|
You can set multiple quality gates that evaluate after test results are reported. These gates adjust the build status based on desired product quality, setting builds to 'unstable' or 'failed'. Each gate measures against the number of test results of a certain type. Please note that the "Newly failed tests" type is not supported in a Release pipeline.
- Edit an existing pipeline or create a new pipeline and click Add Task.
- Search for "Parasoft Test Results Quality Gate" using the search field.
- Select the extension and click Add.
- Configure the following three properties as needed:
- Type: Type of test results to be evaluated for the threshold.
- Threshold: The threshold defines the number of test results that will fail a build. The default value is 0. A build is considered unstable or failed when either of the following conditions are met:
- The number of total passed/executed tests is less than threshold.
- The number of total/newly failed tests is greater than threshold.
- Build status if quality gate fails: Determines if the build should be set to unstable or failed if the quality gate is not passed.
- Choose Save & Queue >Save.
| Note |
|---|
You need to set a baseline by running the Publish Parasoft Results task first in order to perform quality gate checks |
...
. |
Running the Build
A typical build configuration would consist of the following steps:
- Clean the build checkout directory. In this step you will clean the Parasoft XML report files left over from the previous build runs. You can configure the following options:
- Set the Clean option to true under the Get sources step.
- Configure a Command Line runner to clean the Parasoft XML report files left over from the previous runs. For instance, you could run the following command to remove all XML files from the build Checkout Directory: rm $(System.DefaultWorkingDirectory)/*.xml
- Run SOAtest, C/C++test, Jtest, or dotTEST. This step will generate report files in XML format. The XML report files must be saved in the Azure DevOps build project Checkout Directory.
- Run the Parasoft Findings extension. You can have multiple Parasoft tool runs that would generate multiple XML report files prior to invoking this build step. If the build steps prior to the Parasoft Findings extension build step generate multiple XML report files, then all test results from all these files will be added to the Azure DevOps build results.
Viewing Test Results
...
Test results reports are collected from the Parasoft report XML files and the number of passed/failed tests and result details are shown. Build results can also be displayed as trending data in a graph for a simple visualization.
...
The build results overview page provides a histogram of the number of failing tests for each run. The dashboard also provides widgets for overall test result trends.
To view historical details of a particular Parasoft test, click on a test and click View History. A page with the detailed test history will open.
C/C++test Professional Unit Test Results
C/C++test reports for unit test results must be generated with the Overview of checked files and executed tests option option enabled. You can enable this option on the command line by using the option -property report.contexts_details=true or by setting the report.contexts_details=true property in the settings file.
Starting with version 2024.1, you can also use the command line option -property report.additional.report.dir=<REPORT_DIR> when generating the reports and use reports generated in this directory.
| Anchor | ||||
|---|---|---|---|---|
|
Static analysis reports are collected from the Parasoft report XML files and display static analysis data and rule documentation. This feature is not supported in a Release pipeline.
Prerequisites
Viewing static analysis reports in Azure DevOps requires the Azure DevOps extension "SARIF SAST Scans Tab" from Microsoft DevLabs. You can download it for free from the Visual Studio Marketplace. Once this extension is installed, you will have a Scans tab on build details pages.
...
C/C++test Professional Report Settings
Reports for If you are generating static analysis must be generated with the reports with C/C++test Professional 2023.1 or earlier, make sure the Add absolute file paths to XML data option enabled if you are using C/C++test Professional option is enabled to link the source code if it is stored in the Azure Repos Git. You can enable this option on the command line by setting the report.location_details=true property in the settings file, which allows you to navigate to the source code if it is stored in the Azure Repos Git.
| Anchor | ||||
|---|---|---|---|---|
|
Code coverage reports are collected from the Parasoft report XML files and display summary information and class-specific coverage data. Only data for the last matched coverage report is shown. This feature is not supported in a Release pipeline.
Viewing Results of the Last Build
...
Code coverage reports for C/C++test Professional are not supported for versions prior to 2024.1.
Starting with version 2024.1, you can also use the command line option -property report.additional.report.dir=<REPORT_DIR> when generating the reports and use reports generated in this directory.
Viewing Quality Gate Results
Static Analysis
To configure the extension for static analysis quality gates, see Configuring the Parasoft Static Analysis Quality Gates Task. The summary of static analysis quality gates results will be shown on the Extensions tab.
Code Coverage
To configure the extension for code coverage quality gates, see Configuring the Parasoft Code Coverage Quality Gate Task. The summary of code coverage quality gates results will be shown on the Extensions tab.
Test Results
To configure the extension for quality gates, see Configuring the Parasoft Test Results Quality Gate Task. The summary of quality gates results will be shown on the Extensions tab.
Troubleshooting
"JavaScript heap out of memory" error
This issue can occur when Parasoft Findings processes large reports. Configure the Root path to Parasoft tool/Java installation field to use Java for report processing instead of JavaScript, which is default. You can check the debug logs to see whether Java is being used to process the report or not.
"JAXP00010004" error when using Java to process large reports
This issue can occur when Parasoft Findings processes large reports, even if the Root path to Parasoft tool/Java installation field is set to use Java. It is a limitation of JAXP. Refer to the Using the jaxp.properties File section for informationg regarding changing the value of the jdk.xml.totalEntitySizeLimit property to a large value.
Alternatively, you can add a system environment variable that sets -DtotalEntitySizeLimit and -Djdk.xml.totalEntitySizeLimit to a large number. For example:
JAVA_TOOL_OPTIONS=-DtotalEntitySizeLimit=2147480000 -Djdk.xml.totalEntitySizeLimit=2147480000
Third-party Acknowledgements
The Parasoft Findings Plugin for Azure DevOps uses the following third-party software:
...
adm-zip
This software is used under an MIT license.
agent-base
This software is used under an MIT license.
asynckit
This software is used under an MIT license.
axios
This software is used under an MIT license.
azure-devops-node-api
This software is used under an MIT license.
azure-pipelines-task-lib
This software is used under an MIT license.
...
This software is used under an MIT license.
...
bindings
This software is used under an MIT license.
...
brace-
...
expansion
This software is used under an MIT license.
call-bind
This software is used under an MIT license.
...
combined-stream
This software is used under an Apache 2.0 MIT license.
...
concat-
...
map
This software is used under an MIT license.
...
core-util-
...
is
This software is used under an MIT license.
...
deasync
This software is used under an MIT license.
...
debug
This software is used under an MIT license.
delayed-stream
This software is used under an MIT license.
dot-properties
This software is used under an MIT license.
file-uri-to-path
This software is used under an MIT license.
follow-redirects
This software is used under an MIT license.
...
This software is used under an MIT license.
...
glob
This software is used under an MIT ISC license.
...
has-symbols
This software is used under an ISC MIT license.
has
...
This software is used under an MIT license.
has-proto
This software is used under an MIT license.
...
https-proxy-
...
agent
This software is used under an MIT license.
...
immediate
This software is used under an MIT license.
inflight
This software is used under an ISC license.
...
This software is used under a BSD 3-Clause license.
jszip
This software is used under an MIT & GPL-3.0 license.
lie
This software is used under an MIT license.
mime-db
This software is used under an MIT license.
...
This software is used under an ISC license.
mockery
This software is used under an MIT license.
ms
This software is used under an MIT license.
node-addon-api
This software is used under an MIT license.
node-fetch
This software is used under an MIT license.
nodejs-file-downloader
This software is used under an ISC license.
object-inspect
This software is used under an MIT license.
...
This software is used under an ISC license.
...
pako
This software is used under a BSD 3-Clause an MIT license and a Zlib license.
path-is-absolute
This software is used under an MIT license.
...
This software is used under an MIT license.
proxy-from-env
This software is used under an MIT license.
q
This software is used under an MIT license.
...
This software is used under an MIT license.
sanitize-filename
This software is used under WFTPL and ISC licences.
sax
This software is used under an ISC license.
...
This software is used under an ISC license.
setimmediate
This software is used under an MIT license.
shelljs
This software is used under a BSD 3-Clause license.
...
This software is used under an MIT license.
...
tr46
This software is used under an MIT license.
truncate-utf8-bytes
This software is used under a WTFPL license and an MIT license.
tunnel
This software is used under an MIT license.
...
typed-rest-
...
client
This software is used under an MIT license.
...
underscore
This software is used under an MIT license.
...
utf8-byte-length
This software is used under a WTFPL license and an MIT license.
util-deprecate
This software is used under an MIT license.
uuid
This software is used under an MIT license.
webidl-conversions
This software is used under a BSD 2-Clause license.
whatwg-url
This software is used under an MIT license.
wrappy
This software is used under an ISC license.
...
This software is used under an Apache 2.0 license.
xmlresolver-data
This software is used under an W3C license.
xslt3
This software is used under the following license.