The Test Stability Index (TSI) slice calculates a time-based score for failing test cases and updates each test case with the appropriate action. In this section:
Introduction
For projects that build constantly, whether hourly or nightly, frequent test state changes suggest instability of the project. This slice finds test cases that change frequently and prompts the user to take action. This slice is designed to function via a REST API that users can execute with a scheduled cURL command.
The score calculated by the TSI slice is based on the pass or failure status of each test case in a given period of run times. Recent failures are weighted more than older failures. If the calculated score exceeds the user-specified threshold, then the test case action will be changed to “Stabilize”. Each calculated score will be stored as a metric in DTP.
The logic behind the slice is to find all existing test cases in a given filter. For each test found, the slice will get the run time details and calculate the score accordingly. Finally, it will update the test case metadata.
The Test Stabilization Widget uses the results of TSI to perform stabilization analysis on passing and failing tests and visualizes the percentage of tests that fall into various stabilization categories. This helps managers and developers to identify what test cases they need to focus to address to improve stability of the test suite.
Comparing builds from different branches is not currently supported. Tests from the development branch, for example, are considered different tests from master branch.
Requirements
- DTP Enterprise Pack 5.3.2
- Parasoft DTP 5.3.2
Added Components
No new components are installed with this artifact.
Installation/Upgrading
See Downloading and Installing Artifacts for instructions.
About the Custom Processor
This artifact includes a custom processor for DTP that must be installed into DTP after the artifact is installed. If DTP and Extension Designer are on the same machine, the custom processor should automatically install into DTP. If they are on different machines, then you need to manually install the custom processor. Review the following sections for additional information:
This custom processor expects the following parameters:
| filterId | This value is taken from the slice parameter and does not need to be configured (see Invoking the Slice). |
|---|---|
| days | This value is taken from the slice parameter and does not need to be configured (see Invoking the Slice). |
| database | This value can be set to |
Invoking the Slice
Send a REST request to the slice endpoint to invoke the slice. You can access the endpoint by clicking on the Process Intelligence category in the Services tab and expanding to the path:
The endpoint takes the following parameters:
filterId | The slice will use the filter ID to find all associated test cases to calculate the TSI score. |
|---|---|
runs | The slice will calculate all tests based on this number of runs. |
threshold | The slice will warn the user if the calculated TSI score exceeds the given threshold. |
days | The slice will use this value to determine how many days of data to gather. |
The days parameter is used for gathering data, but runs are used for calculation. Otherwise, test cases that run multiple times a day would have a much higher score than test cases that run every other day.
Use HTTP(S)-compliance Parameters
When adding your parameters, be sure to properly encode the string parameter values if they contain spaces, +, /, or any other characters that are not allowed in the HTTP(S) protocol.
The following parameter, for example, is not allowed and will prevent the artifact from functioning correctly: buildId=c++test. The encoded parameter would be buildId=C%2D%2Dtest.
You can use an encoding tool (e.g., http://www.url-encode-decode.com) to help you properly encode parameters to be compliant with the standard.
Reviewing Results
Information about particular test cases is presented in the Test Explorer.
If a test has a TSI score higher than the specified threshold, the action in the Prioritization tab is set to Stabilize.
The calculated TSI score can be found in the DTP metric explorer.
Widget Configuration
- Import and deploy the slice
- Import and configure the Test Stability Index Widget and deploy to DTP.
- Add the Test Stabilization - Pie widget or the Test Stabilization - Top 5 Table widget from the Process Intelligence category and configure the settings (see Adding Widgets).
You can view the widgets on the dashboard.
These widgets use the term "Unstable" to refer to tests marked with the "stabilize" test case action (either by running the Test Stability Index artifact, or manually), and the term "Stable" to refer tests with any other action.
| Fail – Stable | The test failed. The TSI slice did not mark the test with the "stabilize" action since it always failed in the past. |
|---|---|
| Fail – Unstable | The test failed. The TSI slice marked the test with the "stabilize" action because it sometimes passed in the past, but is currently failing. |
| Pass – Stable | The test passed. The TSI slice did not mark the test with the "stabilize" action since it always passed in the past. |
| Pass – Unstable | The test passed. The TSI slice marked the test with the "stabilize" action because it sometimes failed in the past, but is currently passing. |
| Incomplete – Stable | The test is incomplete. The TSI slice did not mark the test with the "stabilize" action since it was always incomplete in the past. |
| Incomplete – Unstable | The test is incomplete. The TSI slice marked the test with the "stabilize" action because it sometimes passed or failed in the past, but is currently incomplete. |
To remove data from one of the displayed categories (e.g., Pass – Stable), click that label. When a label is grayed out, this means that the associated category of data is hidden. To add the associated data back to the chart, click the grayed-out label.
You can click on the widget to see more data associated with the test in the Test Explorer.
