Many SOAtest users create thousands of test cases as they evolve their test suites with Parasoft SOAtest. It is critical to be able to maintain these tests efficiently and effectively as the underlying systems that are being tested change.
This section describes a few capabilities and recommendations that help save time when maintaining and updating large numbers of tests.
Managing Service Definition and Schema Changes
If you are testing web services described with service definition or schemas, we recommend the following best practices:
- Use Form Input and constrain your tests to the WSDL/WADL/Schema/OpenAPI/Swagger/RAML definition. SOAP Client and Messaging Client tools include a “constrain” option in the WSDL tab (SOAP Client) and Schema tab (Messaging Client). The REST Client also contains a similar option for WADLs and RAML/OpenAPI/Swagger definitions. When this option is selected, the following happens when the test is executed in a session (for example, when you execute the test from the command line during nightly testing). First, the definition files will be parsed from the given location. Second, the generated request from Form Input/Form JSON view will automatically conform to the recently parsed definition files. More details on this feature are provided below.
- For endpoint URLs, use Environment variables to manage updates globally, or select the “WSDL” option in SOAP Client so the URL gets updated automatically based on the WSDL.
- Use Change Advisor to identify tests that require updates, then update the applicable tests by applying change templates.
- Use Parasoft search and replace to update your tool values such as XPath values in assertion tools, diff tools, data bank tools.
- Create a regression test for the WSDL, WADL, and/or schemas that are associated with your tests. SOAtest's Test Suite wizard includes an option to create WSDL/WADL tests in addition to functional tests. One of these WSDL/WADL tests is a regression test that captures the current version of the WSDL/WADL and serves as a change indicator; whenever the definition file gets modified on the server, the test will fail and therefore alert you to the change.
- Use regression control test suite level update capabilities.
Automatic Form Input Updates
When the "constrain" option is selected, Form Input views in SOAP Clients, Message Stubs, and Messaging Clients can be updated automatically. The same applies to REST Clients that are constrained to a WADL, RAML, or OpenAPI/Swagger definition. This operation is performed by "refreshing" the definition files (by clicking on the Refresh button in one of these tools). Parasoft SOAtest attempts to preserve existing Form Input settings so that only minimal manual changes are required. Parasoft SOAtest uses sophisticated heuristics that map existing test data.
For example, new elements appear and removed elements are removed without impacting existing ones. Also, changes to the type of an element attempt to carry the current values whenever possible. The remapping of test data according to the new definition file may not always be successful. Typically, this process is more likely to succeed for changes in data types as defined by schema files because these are relatively minor changes. However, this process is less likely to succeed for operation changes as defined by the definition file document because these are relatively major changes.
Updating Messages Specified in Form Input with Change Advisor, Change Impact Analysis
With Change Advisor, you can perform a change impact analysis to identify how changes to a service definition (for example, renamed operations, added elements or types, modified namespaces, and so on) impact the messages that your messaging tools send.
The analysis can either:
Compare the original service definition that was accessed when test was created with the current state of that same service definition.
Compare the original service definition against a different service definition file.
The result of this analysis helps you assess the scope and nature of changes that will be required to update your tests. Once a change template defining how the new service maps to the previous one has been defined (by you or another team member, such as an architect), you can apply that change template to automatically update the tests, individually or in bulk.
For details, see Updating Messages with Change Advisor.
Automatically Updating Diff Tool Regression Controls
Use of an XML Diff Tool is a popular and an effective way to apply validation on the service responses. It provides the convenience of capturing the entire response message to validate it in its entirety and it allows various batch update mechanisms to be executed at the test or test suite level by right-clicking and choosing Update/Create Regression Controls, then following the wizard. If you wish to manage the regression control updates more selectively, then open an editor for the test suite and go to the Regression Options section under the Execution Options tab. There you can select which tests should be updated when updating regression controls at the test suite level.
Updating Other Tool Values Using Search and Replace
The Parasoft search and replace functionality helps you identify and update values that need to be changed as services and environments evolve. For example, if an element or namespace changed, you can use search and replace to update the impacted XPath values in assertion tools, diff tools, data bank tools, and so on.
For details, see Updating Tool Values with Search and Replace.
Workflow for Updating Tests
With the above capabilities a tester would follow these steps:
- Notice a WSDL/WADL/Schema Regression test failure.
- If the difference indicates a change in the request message structure, test updates may be needed.
- If the difference indicates changes in the endpoint or messages/operations that are not tested by this test suite, then update the regression control and ensure that tests are succeeding properly.
- Inspect some of the response errors.
- Do they indicate that the request message was rejected by the server due to a mismatch in expected content? If so, run a Change Impact Analysis.
- Otherwise, the failure may be related to changes in the data expected by the service or an actual regression/bug introduced into the service.
- Update the impacted tools as needed. You can:
- Enable automatic form input updates for simple changes.
- Apply a change template to cover more complex changes (for example, addressing renamed operations, added elements or types, modified namespaces, and so on)
- Update other tool values (for example, assertions, XPaths) using search and replace.
- Automatically update regression controls in Diff tools as needed.
Other Recommendations and Practices to Manage Change
- When testing services during cycles of high volatility, use an XML Assertor or JSON Assertor instead of Diff regression controls. This allows you to configure validation on the specific parts of the message that are relevant to the test case and therefore minimize the update effort as the services evolve.
- SOAP Headers can be defined and managed at the test suite level. Click Add Property on the tool bar and choose the Global Property > SOAP Headers option. Notice the various other artifacts and configurations that can be configured in that dialog and reused at the test suite level to reduce duplication and better manage update activities.
- Performing a mass update on test assets not covered by functionality such as the Change Advisor and Parasoft search and replace is relatively simple, but if it needs to be performed on a large number of tests, advanced users may consider performing a Find/Replace All operation directly on .tst file content. SOAtest's .tst files are saved in an XML format, soa .tst file can be opened in a text editor to perform such updates. The best way to do this successfully is to search the file for the current value which (so you can locate it in the file), then design a replace instruction in the editor that would perform the needed change. Please be sure to backup your files before editing them directly.
- Follow Test Creation Best Practices.
