This topic explains how to configure your functional test suite for regression testing, which helps you determine whether functionality unexpectedly changes over time. Sections include:
Understanding Regression Testing
The purpose of regression testing is to detect unexpected faults—especially those faults that occur because a developer did not fully understand the internal code correlations when he or she modified or extended code that previously functioned correctly. Regression testing is the only reliable way to ensure that modifications did not introduce new errors into code or to check whether modifications successfully eliminated existing errors.
During regression testing, SOAtest runs specified test cases, compares the current outcomes with the previously-recorded outcomes, then alerts you to differences between the current responses and control responses. Subsequent regression test runs will remind you of this discrepancy until the application returns the expected result. We recommend that you perform functional testing to verify that your service functions correctly, then begin regression testing to ensure that system changes do not introduce errors or unexpected changes.
Creating Regression Tests
You can automatically create regression controls for an entire test suite, or you can record regression controls for individual test cases. If you want to run regression testing on all or most test cases, we recommend that you automatically add regression controls to the complete test suite, then remove the controls you do not want to use. If you only want to perform regression testing on a limited number of test cases, it is more efficient to add the controls individually.
Automatically Creating Regression Controls
To have SOAtest automatically create regression controls for an entire test suite or for specific test cases:
- Right-click the appropriate test suite or test nodes, then choose Create/Update Regression Control from the shortcut menu.
- In the Response Validation wizard that opens, expand Create Regression Control, then specify what type of control you want create and click Finish. A Diff control will be added to the selected test(s)
- Single vs. Multiple Regression Controls: This option is only available for tests that use a parameterized value from a data source. It determines whether SOAtest creates regression controls using only one row from the data source, or all rows from the data source.
- Internal vs. External Regression Controls: External regression controls allow you to manage the regression controls outside of SOAtest. If you are working with large messages, external controls are recommended because 1) they reduce the size of the .tst file and 2) they enable you to use the ExamXML Diff option, which can process very large files. Regression content in external files can also be searched, updated, scripted, and managed since the content is readable text.
All subsequent times SOAtest runs a test case with a Diff control, it will compare the actual outcomes with the outcome specified in the Diff controls.
If a single regression control was created, a single file is saved in the same directory as the current project file. This file will contain the regression content.
If multiple regression controls created, the response associated with each data source row will be saved into a separate file. A directory with a random name will also be created and it will contain multiple files with names such as DS_Row_001.xml. The number in this file name correlates to the row number. For example, if the data source has five rows, and the test ran five times when creating regression controls, five files will be generated.
Manually Creating a Regression Control
If you want to perform regression testing on a test case, but do not want to use the current outcome as the comparison value, we recommend that you define your own Diff control for that test case.
To define your own Diff control for a test case:
- Use the procedure described in Adding Test Outputs to add a Diff tool.
- Configure the Diff tool as described in Diff.
All subsequent times SOAtest runs a test case with a Diff control, it will compare the actual outcomes with the outcome specified in the Diff controls.
Modifying Regression Options
To automatically update a regression control:
- Right-click the appropriate test suite or test nodes, then choose Create/Update Regression Control from the shortcut menu.
To manually update a regression control:
- Double-click the Diff tool and modify the available options.
To customize the logic and data source usage for all regression controls in a test suite:
- Double-click the test suite’s Test Case Explorer node and modify the available options, which are described in Regression Options.
