This topic covers the Diff tool, which compares saved data with incoming data and reports differences in SOAtest and Virtualize.
Sections include:
Understanding Diff
With support for Binary, Text, XML, and JSON, the Diff tool is the current backbone of a best-practices regression testing solution.The Diff tool compares the data in its editor against the data that it receives from another tool's output or from another input that you specify and reports a failure when it finds a difference.
The Diff Project Configuration panel is divided into two main sections. The upper section contains general options. The bottom section contains the Regression Control, Ignored Differences, and Options tabs in which you can configure the options for Text, Binary, and XML comparisons.
Binary Diff Mode Options
When Binary is selected from the Diff Mode drop-down menu of the Diff tool, the following options are available:
Data Source: Specifies the Data Source to be used for providing control values. This menu is only available if a Data Source was added to the project.For more information on Data Sources, see Parameterizing Tests with Data Sources, Variables, or Values from Other Tests.
Regression Control Source: Determines what data source values, file, or text is used as the "control" value (the value against which it will compare all subsequent results).
Editor: Choose this option from the drop-down menu and enter text you want to use as a regression control in the related field.
File: Choose this option from the drop-down menu and specify a file to use as a regression control. If you want to ensure that this file's path is always relative to your project file, enable the Persist as Relative Path option.
Data Source Column: Choose this option from the drop-down menu and specify a value from the designated data source column as regression controls. This option is only available if your project includes a data source.
Text Diff Mode Options
When Text is selected from the Diff Mode drop-down menu of the Diff tool, the Regression Controls, Ignored Differences, and Options tabs are available:
Text Regression Controls
The following options are available in the Regression Control tab of the Diff tool for Text mode.
Data Source: Specifies the Data Source to be used for providing control values. This menu is only available if a Data Source was added to the project. For more information on Data Sources, see Parameterizing Tests with Data Sources, Variables, or Values from Other Tests.
Regression Control Source: Determines what data source values, file, or text is usedas the "control" value (the value against which it will compare all subsequent results).
Editor: Choose this option from the drop-down menu and enter text you want to use as a regression control in the related field.
File: Choose this option from the drop-down menu and specify a file to use as a regression control. If you want to ensure that this file's path is always relative to your project file, enable the Persist as Relative Path option.
Data Source Column: Choose this option from the drop-down menu and specify a value from the designated data source column as regression controls. This option is only available if your project includes a data source.
Text Ignored Differences
From the Ignored Differences tab of the Diff tool for Text mode, you can delete any ignored text differences by clicking the Delete button
Text Options
The following options are available in the Options tab of the Diff tool for Text mode.
- Regular Expression: Determines whether the expected value will be parsed as a regular expression. If this box is unselected, the control value will not be parsed as a regular expression. If this box is selected, the control value will be parsed as a regular expression.
For example, the following regular expression may be entered:Java[a-zA-Z ]+\Q(\E[4-9]+th Edition\Q)\E
Where the actual string extracted may be the following: Java How to Program (4th Edition)
The regular expression will parse through the above string to check if "Java" appears as the first substring, followed by one or more occurrences (indicated by the + operator after the open and closed brackets) whereas the occurrences are limited to only letters ranging between a-z and A-Z and whitespace. Notice we escaped ( and ) by adding \Q and \E around ( and ). This needs to be done because open and closed parentheses can be part of a regular expression, therefore they need to be escaped if they are represented to be substrings. We also have a digit appear after the first open parenthesis which is indicated by [4-9]+. This specifies at least one occurrence of a digit between 4 and 9 that should appear and be followed by "th Edition". Lastly we have the closed parenthesis which we escape with \Q\E. - Ignore Whitespace: Determines whether empty lines and whitespaces at the end and beginning of input lines and diff control lines are ignored. If this box is unselected, empty lines and leading/trailing whitespaces will cause the regression test to fail. If this box is selected, empty lines and leading/trailing whitespaces will be ignored.
A whitespace is any of the following: horizontal tabulation, new line, form feed, carriage return, space. An empty line is a line that contains one or more of these whitespaces. - Output results as UNIX-style diff: Determines the diff output format. If this box is unselected, the output will display in table form. If this box is selected, the output will display in UNIX-style
XML Diff Mode Options
When XML is selected from the Diff Mode drop-down menu of the Diff tool, the Regression Controls, Ignored Differences, and Options tabs are available.
XML Mode parses XML files, then compares them element by element and attribute by attribute (ignor-ing “ignorable” whitespace). The result of the comparison is expressed as an XML document. If you want to use this mode, make sure that both inputs are well-formed XML documents which have the same type of document element. The XML Mode provides three different graphical representations to configure XML messages:
- Literal XML
- Form XML
- SOAP Response
Literal XML Regression Controls
The following options are available in the Regression Control tab of the Diff tool for Literal XML mode.
Data Source: Specifies the Data Source to be used for providing control values. This menu is only available if a Data Source was added to the project.For more information on Data Sources, see Parameterizing Tests with Data Sources, Variables, or Values from Other Tests.
Regression Control Source: Determines what data source values, file, or text is used as the "control" value (the value against which it will compare all subsequent results).
Editor: Choose this option from the drop-down menu and enter text you want to use as a regression control in the related field.
File: Choose this option from the drop-down menu and specify a file to use as a regression control. If you want to ensure that this file's path is always relative to your project file, enable the Persist as Relative Path option.
Data Source Column: Choose this option from the drop-down menu and specify a value from the designated data source column as regression controls. This option is only available if your project includes a data source.
- Set From WSDL: Initializes the Form XML content with the expected response based on the WSDL. This button is only available if a WSDL document exists for the particular SOAP client.
Form XML Regression Controls
When Form XML is selected as the Diff mode, the options in the Regression Control tab are divided into two sections: the XML View Tree and the XML Configuration Tabs.
The XML View Tree displays the literal XML as a tree, with each tree node representing an element. The options for the XML View Tree in the Diff panel can be configured in the same fashion as the Form XML SOAP Envelope options of the SOAP Client tool. For more information, see Manipulating the XML View Tree.
The XML Configuration Tabs allow you to add, remove, and rename XML components. The options for the XML Configuration Tabs can be configured in the same fashion as the Form XML SOAP Envelope options of the SOAP Client tool. For more information, see Manipulating the XML Configuration Tabs.
SOAP Response Regression Controls
The SOAP Response mode compares values that you define to the actual outcome of the SOAP response. It allows you to compare multiple elements and attributes of the most complex XML structures.
Depending on the method, a number of different options may be available. The options for the SOAP Response mode can be configured in the same fashion as the Literal Value options of the SOAP Client tool. For more information, see Specifying Literal Values.
XML Ignored Differences
From the Ignored Differences tab of the Diff tool for XML mode, you can Add and Modify XPath settings by clicking the appropriate buttons. For more information on configuring Ignored Differences, see Understanding XPaths.
XML Options
The following options are available in the Options tab of the Diff tool for XML mode.
- Diff engine: Determines what diff engine is used. XMLUnit was introduced as the default in version 9.8.2; it provides support for large messages, comparisons that generate a large num-ber of differences, and the ability to ignore changes in element order. If desired, you can switch to VMTools or ExamXML MDCXML.
- Ignore element order: Configures the tool to ignore element order throughout all areas of every processed document. Note that if you’d rather ignore order for only a specific part of the document, use the XML Sort tool to sort that part before passing the document to the Diff tool. See XML Sort for details.
- SOAP mode: If this box is selected, the following actions are enabled:
- SOAP multi-references are resolved before diffing and are not reported as errors. For example, some services (such as Axis) rearrange the XML responses in an unpredictable way when using SOAP multi-refs (i.e., giving different id numbers to the references). When such responses are diffed in text mode, failures occur even though the SOAP messages are logically equivalent, but their references are different.
- Namespace prefix changes in type and arrayType attributes are ignored. This is needed when the control is generated automatically from the WSDL because prefixes cannot be determined in advance.
- Numerical values are compared as numbers. For example, the difference between 1 and 1.0 would not be reported as an error.
Using ExamXML MDCXML
ExamXML is a 3rd party tool that is licensed and purchased separately from A7Soft. If you select ExamXML MDCXML, be sure to specify the location of the executable and options file.
Notes:
- Versions tested with this integration include 4.74 and 4.38.
- The MDCXML.exe options file contains various settings for mdcxml, including a list of elementsto ignore. An example of an options file is provided on the A7Soft Web site at http://www.a7soft.com/mdcxml.html.
The options file needs to contain the following minimum settings (using different values for the following parameters is not supported):
- ValidationErrorAsFatal=0
- InsertPI=1
- InsertTags=0
- InsertComments=0
- In order to use the ExamXML mode, the Diff tool’s Control content must be provided in a file.
It is possible to create external regression controls automatically (save the expected content into files instead of embedding it in the Diff tool) as described in Configuring Regression Testing.
Using VMTools
VMTools is built in to the product. No additional license is needed. If you select VMTools, you can specify the following options:
Output results as XML: Determines the diff output format. If this box is unselected, the output will display in table form. If this box is selected, the output will display in XML. Differences are only reflected when another tool is chained to the message output of the Diff tool.
- SOAP mode: If this box is selected, the following actions are enabled:
- SOAP multi-references are resolved before diffing and are not reported as errors. For example, some services (such as Axis) rearrange the XML responses in an unpredictable way when using SOAP multirefs (i.e., giving different id numbers to the references). When such responses are diffed in text mode, failures occur even though the SOAP messages are logically equivalent, but their references are different.
- Namespace prefix changes in type and arrayType attributes are ignored. This is needed when the control is generated automatically from the WSDL because prefixes cannot be determined in advance
- Numerical values are compared as numbers. For example, the difference between 1 and 1.0 would not be reported as an error.
Under certain conditions, VMTools will fall back to a text-based diff. This can happen if the XML is very complex and produces an excessive number of differences or would take an excessive amount of time to complete.
If this occurs, one of the following message will be printed to the Console view to explain what happened.
Did not run XML Diff, there are too many differences. Performing text mode diffing instead.
Did not run XML Diff, it did not finish within 30.0 secs. Performing text mode diffing instead.
The maximum time allowed to perform an XML diff before switching to text mode is the default timeout value specified in the preference panel’s Misc tab (Connection Settings> Default timeout)—but no higher than 120 seconds.
JSON Diff Mode Options
When JSON is selected from the Diff Mode drop-down menu of the Diff tool, the Regression Controls, and Ignored Differences are available.
JSON Regression Controls
The following options are available in the Regression Control tab of the Diff tool for JSON mode.
- Name: Specifies the name of the Diff tool.
Data Source: Specifies the Data Source to be used for providing control values. This menu is only available if a Data Source was added to the project. For more information on Data Sources, see Parameterizing Tests with Data Sources, Variables, or Values from Other Tests.
Regression Control Source: Determines what data source values, file, or text is used as the "control" value (the value against which it will compare all subsequent results).
Editor: Choose this option from the drop-down menu and enter text you want to use as a regression control in the related field.
File: Choose this option from the drop-down menu and specify a file to use as a regression control. If you want to ensure that this file's path is always relative to your project file, enable the Persist as Relative Path option.
Data Source Column: Choose this option from the drop-down menu and specify a value from the designated data source column as regression controls. This option is only available if your project includes a data source.
JSON Ignored Differences
From the Ignored Differences tab of the Diff tool for JSON mode, you can Add and Delete XPath settings by clicking the appropriate buttons.
Understanding XPaths
When the Diff tool is configured in the XML mode, any differences found between the actual value and the expected value are expressed in XPath. XPaths represent the position of an XML element and specifies where a difference occurred, the type of difference that occurred, and whether that difference was caused by a modification, an insertion, or a deletion.
You can determine which XPaths to ignore when running the Diff tool. You can choose to either ignore an entire XPath, or you can ignore a specific XPath operation such as a Content: Insert operation.
Ignoring XPaths is useful for ignoring transient values that would normally cause the regression test to fail. For example, the actual outcome may contain a time stamp value that is constantly changing. Since this value is never the same, it will more than likely not match the value you configure in the Diff tool, causing the test to fail. Therefore, you can specify the XPath of this value to be ignored so that the test will not fail.
There are three different ways you can set XPaths to Ignore:
- Ignore XPaths from the Quality Tasks view: This is the easiest way to configure the Diff tool to ignore XPaths. After a failed regression test, you can simply right-click the Quality Tasks view node to select an XPath to ignore.
- Manually entering an XPath in the Diff GUI: You can manually enter an XPath by clicking the Configure Ignored Differences button in the Diff GUI.
- Ignore XPaths from the Form XML tree: You can right-click an element node from the Form XML tree and configure an XPath to ignore based on the selected element.
When entering XPaths to ignore in the Diff tool, it is easiest to right-click in the Quality Tasks view rather than manually entering the XPath position into the Diff tool GUI. XPath positions are displayed in tree form in the Quality Tasks view after a failed regression test: To ignore an XPath from the Quality Tasks view after a failed regression test:Ignoring XPaths from the Quality Tasks View\
The XPath you specified will now be ignored for any future test runs. In addition, the XPath you specified now appears in the Ignored Differences tab of the XML Mode in the Diff tool. To modify the XPath, see Modifying XPath Settings.
Manually Entering XPaths to Ignore
You can also manually type or paste an XPath to ignore into the Diff tool configuration panel.
To manually enter an XPath:
- Select the Ignored Differences tab within the Text, XML, or JSON mode of the Diff tool.
- Click the Add button. An empty field displays in the XPath column of the Ignored XPaths list. By default, the Settings column is automatically filled in with all XPath operations specified, meaning that the entire XPath you add will be ignored. To specify a single XPath operation to ignore, see Ignoring XPaths from the Form XML Tree.
- Either enter an XPath position in the empty XPath field.
- To add additional XPaths, repeat steps 1 through 3.
- Click the OK button.
All XPaths added will be ignored in future runs of the modified regression test. For information on modifying ignored XPaths, see Modifying XPath Settings.
Ignoring XPaths from the Form XML Tree
You may also ignore XPaths directly from the Form XML tree within the Form XML tab by completing the following:
- Right-click an element node from the tree and choose Setup Ignored XPaths from the shortcut menu.
- If the element you selected from the Form XML tree is not a repeated element, the Ignored XPaths Settings dialog box displays with the selected XPath automatically entered.
- If the element you selected from the Form XML tree is a repeated element that has siblings with the same local name, a Repeated XPath Settings dialog displays.
You can either select to Ignore All of the repeated elements, or Ignore Selected Element Only. After making your selection and clicking OK, the Ignored XPaths Settings dialog box displays with the selected XPath automatically entered.
- If the element you selected from the Form XML tree is not a repeated element, the Ignored XPaths Settings dialog box displays with the selected XPath automatically entered.
- Select the appropriate check boxes of the XPath operations you would like to ignore. The following options are available:
- XPath: Specifies the XPath position that you selected.
- Recursive: Select to apply the Ignored XPath settings to child elements.
- Text Content (Modify/Insert/Delete): Select the content operation you want to ignore.
- Element/Subtree (Insert/Delete): Select the element or subtree operation you want to ignore.
Attribute (Modify/Insert/Delete): Select the attribute operation you want to ignore. If this field is selected, only the specified attribute name within the XPath will be ignored. To ignore a specific attribute, enter the attribute name in the field next to the Attribute check box. If you want to ignore more than one attribute at an element’s XPath location, leave the attribute name empty or use the wild card * (e.g. myAttribute*).
- Element Name and Namespace (Modify/Insert/Delete): Select the element name operation you want to ignore.
- Click OK.
The selected element node now displays in gray in the Form XML tree. All XPath operations specified will be ignored in future runs of the modified regression test.
To reset and no longer Ignore an XPath, right-click the Ignored XPath node and select Remove Ignored XPath from the shortcut menu. The node will then display in black and will no longer be ignored.
Creating a Shared Ignored XPaths List
Diff tools can either have their own local, exclusive Ignored XPath settings, or they can reference a list of XPaths that can be used by other Diff tools.
Modifying XPath Settings
By default, when you add an XPath to the Ignored Differences tab, all of the possible operations for that XPath display in the Settings column. Since all of the possible operations for the XPath are specified by default, all of these operations will be ignored the next time the regression test is run. However, you can specify which operations are ignored, rather than having all of them ignored, by modifying the XPath settings.
To modify XPath settings:
- In the Ignored Differences tab, select the XPath you would like to modify and click the Modify button. The Ignored XPaths Settings dialog box displays.
- Select the appropriate check boxes of the XPath operations you would like to ignore. The following options are available:
- Recursive: Select to apply the Ignored XPath settings to child elements.
- XPath: Specifies the XPath position that you selected.
- Text Content (Modify/Insert/Delete): Select the content operation you want to ignore.
- Element/Subtree (Insert/Delete): Select the element or subtree operation you want to ignore.
Attribute (Modify/Insert/Delete): Select the attribute operation you want to ignore. If this field is selected, only the specified attribute name within the XPath will be ignored. To ignore a specific attribute, enter the attribute name in the field next to the Attribute check box. If you want to ignore more than one attribute at an element’s XPath location, leave the attribute name empty or use the wild card * (e.g. myAttribute*).
- Element Name and Namespace (Modify/Insert/Delete): Select the element name operation you want to ignore.
- Click OK.
All XPath operations specified will be ignored in future runs of the modified regression test.
Video Tutorial
In this video, you'll learn how to generate and update regression controls.
