The Test Stability Report (TSR) Process Intelligence flow monitors test case results over a specified number of builds and identifies tests that frequently change statuses as unstable. The flow includes widgets that show the current state of your tests, as well as an interactive drill-down report that shows the test execution history and test status changes.
In this section:
| Table of Contents | ||
|---|---|---|
|
...
This workflow collects all test statuses for the last 10 ten builds and calculates a stability score for each test using an algorithm. The algorithm is based on the number of test status changes and how recent the changes occurred relative to the latest build. If the score exceeds the determined threshold, the test case is considered unstable. You can customize the algorithm, threshold, and number of builds used to determine the stability score.
...
- If you have not already done so, install the Process Intelligence Pack.
- Open Extension Designer and click the Services tab.
- Expand the Process Intelligence Engine service category. You can deploy assets under any service category you wish, but we recommend using the Process Intelligence Engine category to match how Parasoft categorizes the assets. You can also click Add Category to create your own service category (see Working with Services for additional information).
- You can deploy the artifact to an existing service or add a new service. The number of artifacts deployed to a service affects the overall performance. See Extension Designer Best Practices for additional information. Choose an existing service and continue to step 6 or click Add Service.
- Specify a name for the service and click Confirm.
- The tabbed interface helps you keep artifacts organized within the service. Organizing your artifacts across one or more tabs does not affect the performance of the system. Click on a tab (or click the + button icon to add a new tab) and choose Import from the vertical ellipses menu.
- Choose Local> Flows> Workflows> Process Intelligence> Local > Flows > Workflows > Process Intelligence > Test Stability Report and click Import.
- Click anywhere in the open area to drop the the artifact into the service.
- Click Deploy to finish deploying the artifact to your DTP environment.
- Return to DTP and refresh your dashboard. You will now be able to add the related widgets.
...
| Title | Enter a new title to replace the default title that appears on the dashboard. |
|---|---|
| Filter | Choose Dashboard Settings to use the dashboard filter or choose a filter from the drop-down menu (see see Configuring Filters for more information about filters in DTP). |
| Last Run | The widget shows data for the last run based on test status. You can choose:
You can add multiple instances of the widget configured with different last run settings to create a more complete view of your test stability. |
...
- Mouse over a segment to view details.
- Click on the widget to view the unstable tests for the selected run in the Test the Test Execution Stability Report.
Viewing the Test Stability - Statistics Widget
...
This report shows the test results in the filter for the last 10 ten builds (default). You can filter the report according to test stability (All, Stable, or Unstable) and last run status (All, Passed, Failed, Incomplete, No Data).
...
This column shows the name of the test file containing the executed test case. Manual tests do not have associated test file names. You can click in a cell to highlight a test file name and use your keyboard to copy it to your operating system clipboard (Ctl CTRL+C or or Command+C). This makes searching for the file in your test automation tool easier.
...
This column shows the name of the test case that was executed. You You can click in a cell to highlight a test case and use your keyboard to copy it to your operating system clipboard (Ctl CTRL+C or or Command+C). This makes searching for the test case in your test automation tool easier.
...
This column shows a grid of builds and test cases. Each row represents a test case, and each cell represents the results for one build. The results are color-coded:
...
| Info | ||
|---|---|---|
| ||
By default, the report loads Unstable results followed by Stable results. The second-tier order within each stability category is Failed, Incomplete, Passed, and No Data. |
...
This column enables you to define an action to take for the test case. Choose an action from the drop-down menu and the information about the test will be updated in the the Test Explorer.