This topic describes the reports produced after Load Test performs a load test and explains how to access, customize, save, and reopen these reports. In this section:

Preselecting Detailed Report Options

If you want Load Test to create a detailed report, you must configure this before load test execution. By default, Load Test does not gather the information for this report (in order to streamline the testing process).

Preselecting the Detailed Report option prompts Load Test to create detailed scenario reports in addition to its regular load testing reports. Detailed scenario reports plot each "hit" that results from a scenario run. This is the most detailed report that Load Test produces for virtual user tests. Because this report plots every hit, you can see where most hits occur, as well as how many hits fall outside of the normal distribution. 

To preselect Detailed Report options that will apply to all scenarios in a project:

1. Select the Scenarios node in the Load Tests tab. The Report Settings displays in the right GUI panel.

   If the Load Tests tab is not active, right-click anywhere in the Projects tab and select Create Load Tests from the shortcut menu.

2. In the Report Settings panel, select the Record graph data and error details checkbox. A detailed scenario report will be created after the completion of a load test. For more information on detailed reports, see Detailed Reports.
   After selecting the Record graph data and error details checkbox, the following options are available:
   
   • Record first: Select this radio button and enter a number in the text field to record the first number of error details you specify. The Record first option is useful in preventing the detailed report files to grow to unmanageable sizes if a large number of errors are produced. By default, the first 300 error details are recorded.
   • Record all: Select this radio button to record all error details in the detailed report.
   • Record individual hits: Select to record individual hits and to have Load Test create a histogram as part of the detailed report after completion of a load test. You will also have the option to display individual hits in graphical and table forms within the Detailed report.

Viewing a Report

Load test results are available in the right GUI panel after a load test completes. To open this window at a later time: 

• Select Window> <Scenario Name>.

Load test results display in the right GUI panel with the following menu options:

• Views: Determines what type of report is displayed.
  • To view a summary of test information choose Test Information.
  • To view detailed test statistics, choose Statistics.
  • To view test execution messages reported by tools involved in the test, choose Messages. The Messages view is only visible if tools generated an error message. It reports only one instance of each error message reported--even if a certain message was reported multiple times. To view details about an error message, double-click the related entry in the Messages view.
  • To view the detailed report, choose Detailed Report.
• Machines: Determines which machine results are displayed. Select from any available instances to display machine results. The default is All Machines.
• Profiles: Determines which profile results are displayed. You may select any of the profiles that you created. The default is All Profiles.
• Tests: Hides or shows the Tests tree selection panel. The selection in that panel determines which test results are displayed. You may select any combination of the tests that is available in the listed test suite. By default, all Tests tree nodes are selected.
  
  

Test Tree Selection Panel Navigation and Options

From the Test Tree selection panel, you can select All Tests, All Test Suites, All Test Requests (if there are requests in the report) All, or None test tree nodes by choosing the appropriate item in the dropdown menu of the Select button below the Tests tree selection panel:

The same dropdown menu is available for each tree node in the Test tree selection panel:

The expandable Options panel shows additional controls for the Test tree selection panel. The Single Selection on Path option (selected by default) ensures that no more than one tree node is selected on each tree path. This prevents undesirable report accumulation of parent and child node data in the views controlled by the Test tree selection panel. 

The Propagate Selection option (unselected by default) allows you to apply the selection to an entire sub-tree. For example if you select/unselect a tree node that has child nodes, all the child nodes will be selected/unselected as well. 

Understanding Available Reports

This section describes the following report types:

• Test Information
• Statistics
• Detailed Report
• Quality of Service Report

Test Information

Test Information report lists basic test parameters and results in text format. A Quality of Service Report summary will also be displayed for the metrics that have been configured.

Statistics

The Statistics report lists detailed statistics about the load test, such as run times and failure count.

The report displays the statistics of the individual tests as well as the test suites to which they belong (if such data is available). Both test and test suite reports are generated during the load test after the completion of each individual test or test suite. These individual reports are later used to calculate aggregate values that are presented in the Statistics view.

The Statistics report can be viewed in a Hierarchical or a Flat layout. 

• The Hierarchical layout displays test results in a tree-table format that matches the order of tests in the functional project (i.e. the SOAtest project) that was used to run the load test. To preserve the report hierarchy in Hierarchical layout, the sorting of the tree-table nodes is performed within the scope of the parent node. If you need to sort all Statistics results, you can switch to the Flat table layout.
• The Flat layout allows you to sort all test results by a parameter of your choice. It displays the test path of each test in Test Path column. It also displays the path of the selected test in the Report Path sub-panel at the bottom of the table view.

This is the Hierarchical layout:

This is the Flat layout:

Output Types

When viewing the Statistics report, an Output Types menu is available. The Output Types menu determines what type of output report is displayed. Two types of reports are available from this menu, each of which displays different columns of information. The differences between report types are:

• Generic Reports: Contains Test Suite name, Test Index, Test Name, Min/Max/Avg Time (ms), Std. Deviation (ms), Run Count, and Failure Count.  Also contains 99th, 98th, 95th, 90th, 85th, 80th, 70th, 60th, 50th, 25th, and 10th percentiles when individual hits have been collected.
• Network Client Report: Contains all of the information in Generic Reports plus Min/Max/Avg Ping (ms), Min/Max/Avg Request Size (bytes), Min/Max/Avg Response Size (bytes) Min/Max/Avg Total Size (bytes).

If some of the machines were configured to run in the High Throughput Mode the available report types will be:

• Unverified Report Mode: Contains Run and Failure counts for Tests in Verified Mode, Run counts and Failure Estimate intervals for Tests in Unverified Mode as well as the Total Run count and Failure Estimate. The user can choose the Confidence Level percent value that will be applied to the Estimated error intervals in the report.
• Verified Mode Generic Reports: Same as Generic Reports in non High Throughput Mode.
• Verified Mode Network Client Report: Same as Network Client Report in non High Throughput Mode

Customizing Available Statistics Report Columns

You can remove the selected columns that display in the panel for Generic and Network Client Report and/or add additional columns for each report type.

To customize the available report columns:

1. Right-click the table area of the Statistics Report.
2. Choose Select Columns from the shortcut menu.
3. Use the controls in the Columns Configuration dialog box to specify which columns you want shown.
4. (Optional) If you want Load Test to apply this same configuration to all subsequent load test reports for the current project, click Save Configuration before closing this dialog box.

Viewing Statistics Reports Test Failures

To access details about a specific test failure, double-click the related report row. You can determine whether a given test failed by locating the row that represents that test, then checking the number in that row’s Failure Count column. Load Test will show the details of the last error of the test. You can view all errors of  a specific test in the Detailed Report section, which is described in Viewing Detailed Report Table Content.

Detailed Reports

Detailed reports save all samples of data collected during load testing, and details of all errors that occur. This is the most detailed report that Load Test produces for load tests. Because this report can plot every hit, you can see where most hits occur, as well as how many hits fall outside of the normal distribution. Results for all server machines are displayed in the same graph; the results from different servers are color coded.

The Detailed Report contains the following tabs:

• Graph: Displays all possible graphs for the individual load test.
• Histogram: Displays the distribution of values for the selected variable (execution time, request size, response size).
• Table: Displays either the errors or individual hits that occurred for the load test.

In addition, the results in each tab of a Detailed Report can be filtered through the drop-down menus located at the top of the right GUI panel. Load test details that display are filterable by Machines, Profiles, and Tests.

Note that the Tests tree filter allows you to have graphs include test suites, tests, and—and for browser tests – browser request reports. When evaluating the load that is applied to the system under test, be aware of this and select the appropriate subset of reports that applies to your testing scenarios. For example if you run a load test based on SOAtest SOAP Clients, you may want to select only Test nodes (see screenshot below) to view the graph of SOAP Client requests to the system under test. You would select Test Suite nodes only if you wanted to see the graph of transactions that include multiple tests.

Viewing Detailed Report Graph Content

The following options display in the Detailed Report Graph tab legend:

• Tests Completion Rate: Select to display the rate of the tests completed over time in the graph.
• Tests Start Rate: Select to display the rate of the tests started over time in the graph.
• Test Error Rate: Select to display the rate of errors over time in the graph.
• Virtual Users: Select to display the number of virtual users over time in the graph.
• Avg. Execution Time: Select to display the average execution time of the load test.
• Max. Execution Time: Select to display the maximum execution time of the load test.
• Min. Execution Time: Select to display the minimum execution time of the load test.
• Monitor Parameters: Select to display data collected from any SNMP, Windows, or JMX Monitors you added to the Monitors node of the Load Test tree. The names of the monitor parameters are based on the monitors’ Host and Graph Title values. For example, if you added a monitor and entered "ox" for the Host, and "SNMP TCP Established" for a Graph Title, you will see a check box labeled "ox SNMP TCP Established" in the Graph tab during load test progress. For more information on monitor parameters, see Using Monitors.

Customizing Detailed Report Graph Content

There are a variety of ways to customize the report content of the Detailed Reports Graph tab. To filter the graphical data displayed:

• Select from the drop-down menus located at the top of the right GUI panel. Load test details that display in the Graph tab are filterable by Machines, Profiles, and Tests.

To customize the report parameters in the Graph tab:

• Choose values from the X axis box. The graph will change accordingly. To see more options in the X axis box, check the More X-axis options check box.

To change the report’s Y axis from the default linear scale to a logarithmic scale (or vice versa):

• If you want to use a logarithmic scale, check the Log Scale check box. A logarithmic scale allows you to see the shape of multiple curves on the same graph (even if the displayed values are very far from one another). For example, if the total loading time is between 1000 ms and 200 ms and the number of users is between 10 and 20, you will not be able to see the users values if the graph uses the linear scale (the graph will be a flat line close to 0). If you use a logarithmic scale, you will be able to see loading time values as well as the users values.
• If you want to use the linear scale, clear the Log Scale check box.

To fill in a graph area with its designated color, check the Fill graph areas check box.

To display the individual hits that occurred during the load test as individual points:

• Select the Show Individual Hits check box. All of the individual hits that occurred during the load test displays in the graph tab.
  Hovering over a hit with the mouse will display a tool tip with hit statistics.

  Individual hits can only be displayed in a Detailed Report if the Record individual hits option was preselected from the Report Settings tab of the Scenarios configuration panel. For more information, see Preselecting Detailed Report Options.

To reduce the number of graph points, in cases when there large reports for long running load tests (12 hours or more), click the arrow button to expand the Reduce Graph Points box. The following options are available:

• Point average: Specifies the point average applied to the graph and corresponds to the increments on the slider bar.
• Scale Limit: Enter the scale limit to be applied to the slider bar. The max value is 500.
• Interactive: Select this checkbox to make the slider bar interactive. If this is selected, moving the slider bar will simultaneously alter the graph. If this is not selected, you must move the slider bar and then click the Apply button to alter the graph.

To zoom in on a specific area of the graph:

1. Click and drag to place a gray box around the area you want to focus on.
2. Right-click the newly-created box, then choose Zoom to selection from the shortcut menu.

To zoom in and out on the graph:

• Move the mouse while pressing the left mouse button and the Ctrl keyboard button.
• Rotate the mouse wheel to zoom in and out of the view.

To shift the graph:

• Move the mouse while pressing the left mouse button and the Shift keyboard button.

To return to the original graph size and/or position:

• Right-click any area of the graph, then choose Original Size from the shortcut menu.

To export and save individual hits in CSV format:

1. Filter the individual hits that display by making the appropriate selections from the Machines, Profiles, and Tests menus.

2. Select the Show Individual Hits check box. All of the individual hits that occurred during the load test displays in the graph tab.

   Individual hits can only be displayed in a Detailed Report if the Record individual hits option was preselected from the Report Settings tab of the Scenarios configuration panel. For more information, see Preselecting Detailed Report Options.

3. Complete one of the following:
   • Right-click any area of the graph, then choose Export Individual Hits from the shortcut menu. This will export all individual hits that occurred during the load test.
   • Specify a time interval by moving the mouse while pressing the left mouse button and the Alt keyboard button to place a gray box over the desired time interval on the graph, then right-click the gray box and choose Export Individual Hits from the shortcut menu. This will export all individual hits that occurred during the specified time interval.
   • Click and drag with your mouse over any area of the graph and create a gray rectangular box to specify an (x,y) region, then choose Export Individual Hits from the shortcut menu. This will export all individual hits that occurred within the specified (x,y) region.
4. In the dialog box that opens, choose the location to where you would like to save the CSV file.

To export and save graph points in CSV format:

1. Filter the graph points that display by making the appropriate selections from the Machines, Profiles, and Tests menus.
2. Complete one of the following:
   • Right-click any area of the graph, then choose Export Graph Points from the shortcut menu. This will export all graph points that occurred during the load test.
   • Specify a time interval by moving the mouse while pressing the left mouse button and the Alt keyboard button to place a gray box over the desired time interval on the graph, then right-click the gray box and choose Export Graph Points from the shortcut menu. This will export all individual hits that occurred during the specified time interval.

3. In the dialog box that opens, choose the location to where you would like to save the CSV file.

   The Export Graph Points option is only available if the Record individual hits option was preselected from the Report Settings tab of the Scenarios configuration panel. For more information, see Preselecting Detailed Report Options.

To customize Detailed Report graph colors:

1. Right-click the desired bar in the Graph tab legend and select Change Color from the shortcut menu. Within the Change Color shortcut menu are two sub-menus:
   • In Report Only: Select to customize the color for the current Detailed Report only.
   • In Report and Configuration: Select to customize the color for the current Detailed Report, and all Detailed Reports to follow.
   The Select color dialog box opens.
   
   
   
   
2. Manipulate the color scheme as needed in the dialog box and click OK.

Viewing Detailed Report Histograms

Histograms graph the distribution of the values obtained for the selected variable (execution time, request size, response size).

To control the histogram’s focus content:

• Choose a variable from the X axis drop-down menu at the top of the histogram results panel.
• If you want to use a logarithmic scale, check the Log Scale check box. A logarithmic scale allows you to see the shape of multiple bars on the same graph (even if the displayed values are very far from one another).
• To shift the histogram, move the mouse while pressing the left mouse button and the Shift keyboard button.

Viewing Detailed Report Table Content

Before you can view any errors or individual hits in the Table tab of a Detailed Report, you must first select a time interval from the Graph tab by completing one of the following:

To view errors in the Table tab, complete the following in the Graph tab:

1. Select the appropriate parameter values from the X axis box, and the appropriate check boxes from the bottom of the graph to determine the graph report.
2. Move the mouse while pressing the left mouse button and the Alt keyboard button to place a gray box over the desired time interval on the graph. To unselect the gray box, press the Alt keyboard button and mouse click the gray box.
   
   
   
   
3. Right-click the gray box and select Show Recorded Details from the shortcut menu. The Table tab opens and displays all of the recorded details collected during the selected time interval.
   If you do not select a time interval, the Table tab opens and displays all available details after you right-click the graph and select Show Recorded Details from the shortcut menu.
   
   
   
   The Details column shows either the error string of the test (if it failed) or the [Traffic] tag (if the test succeeded). Double-click the row to see either error details or the traffic of the test.
   The Table tab displays the first 300 detail results by default. If more than 300 details are present (if you selected to record all details or selected to record more than the first 300 details) a drop-down menu will appear at the top of the Table tab, along with navigation buttons that allow you to browse through groupings of details.
4. To access details about a specific test results, double-click the related report row.

To view individual hits in the Table tab, complete the following in the Graph tab:

1. Select the Show Individual Hits check box.
2. Select the appropriate parameter values from the Y axis box.
3. Move the mouse while pressing the left mouse button and the Alt keyboard button to place a gray box over the desired time interval on the graph. To unselect the gray box, press the Alt keyboard button and mouse click the gray box.
   
   
   
   
4. Right-click the gray box and select Show Individual Hits from the shortcut menu. The Table tab opens and displays all of the individual hits that occurred during the selected time interval.
   
   
   
   Rather than selecting a time interval, you can select a few points in the Graph tab by clicking and dragging your mouse to place a gray box around the points you want to focus on, and then right-clicking and selecting Show Individual Hits from the shortcut menu.
   You can also double-click a single point in the Graph tab and the Table tab will open and display information for the single hit you selected.
   If details information has been recorded for the hit, the Details column of the Individual Hits table will contain the error details of the test (if the test failed) or the [Traffic] tag (if the test succeeded). Double-click the row to see either error details or the traffic of the test for which the details were recorded.
   
   

Viewing Hits of a Virtual User

You can see all hits of a Virtual User by right-clicking a row in the Individual Hits table, then choosing Show All Hits of This VU from the shortcut menu.

This opens a table with all hits of the selected Virtual User. 

To return to the Individual Hits table, click the Show Previous Table << button.

Customizing Available Report Columns

To customize the visible columns in the Table tab view:

1. Right-click the table area of the Table tab.
2. Choose Select Columns from the shortcut menu.
3. Use the controls in the Columns Configuration dialog box to specify which columns you want shown.
4. (Optional) If you want Load Test to apply this same configuration to all subsequent load test reports for the current project, click Save Configuration before closing this dialog box.

Quality of Service Reports

The Quality of Service (QoS) Report shows the metrics that you have configured in the Load Test, and displays a Summary tab and a Details tab.

The Summary tab displays a Metric, Description, and Status column. Each row in the QoS report corresponds to one QoS metric.

The Details tab displays the results of the available metrics that have been configured for the selected scenario. The Details tab also displays the name, parameters, and notes for the metric selected from the Metric List.

You can further customize the metrics in the reports by configuring the parameters of any metric from the Metric List and clicking the Apply to Report button. You can also select multiple metrics from the Metric List (CTRL-click) and then click the Apply to Report button. This will update the QoS Results that are seen in the QoS Report and Test Information views, but will not reflect in the QoS node of the Scenario View.

For more information on using and configuring Quality of Service Metrics, see Customizing QoS Metrics for Scenarios.

Load Test Configuration Report

You can also right-click anywhere in the Load Tests tab and select View Structure Report> HTML/XML. A Load Test Configuration Report displays in your browser. This report contains the following information:

• Load Test Scenarios: Displays any load test scenarios that were created and run for the selected project. You may click on the QoS metric name within the report, and additional information will display such as the Scenario, Group, and Description of the metric.

Special Reporting for Web (Browser) Tests

During Load Test execution, a web functional test with the Browser Testing tool can send multiple requests (the requests that you see in the test's editor in SOAtest’s Load Test perspective). You can see the reports for these requests as child nodes of the Browser Test report in the Statistics, Detailed Report and QoS Report sections of the Load Test report (see screenshots below).

For each Browser Test and the associated requests, Load Test will record network-related statistics. These statistics will be available in the Network Client Reports table within the Statistics section of the Load Test report (see Statistics for details).

The Statistics section of the Load Test report shows browser requests as follows:

The Detailed Report section of the Load Test report shows browser requests as follows:

The QoS section of the Load Test report shows browser requests as follows:

Saving and Reopening Reports

You can save individual load testing reports in binary, HTML, XML, GIF, and CSV formats. Not all reports can be saved in all formats. Only binary-format reports can be reopened in Load Test.

You can also save a comprehensive load testing report that includes test information, histograms, and tables. This option is also available from the command line.

The following table describes which formats apply to which reports and how to save reports in the available formats.

To open a binary-format report, you choose File> Open, then use the file chooser to specify which .rpt file you want to open. 

Saving Part of a Report

To save a part of a load test report:

1. Open the Detailed Report view of the load test report.
2. Select an interval that you would like to save. To select an interval, move the mouse while pressing the left mouse button and the Alt keyboard button to place a gray selection box over the desired time interval on the graph.
3. Release the Alt button, right-click in the graph and choose the Save As> Binary> Selected Interval command.
   
   
   
   
4. Enter the report file name in the file selection dialog.

The statistical data and the QoS metrics of the interval report will be recalculated based on the selection.

Configuring HTML Report Options

In addition to saving and viewing load test reports in HTML format, you can also customize the graph views that are included in these HTML reports according to various parameters you choose.

To configure the graph views that display in an HTML report:

1. Open the HTML Report Options dialog box by completing one of the following:
   • Right-click in the active area of any report view (i.e. Test Information, Statistics, Detailed Report) and select View Report Configuration from the shortcut menu. An HTML Report Options dialog box displays with three graph entries by default.
   • Right-click in the active area of either the Graph tab or Histogram tab of the Detailed Report view and select Add to Report from the shortcut menu. An HTML Report Options dialog box displays with the current Graph or Histogram view options added to the three default graph entries.
     
     
     
     Each entry in the HTML Report Options dialog box reflects options of a graph or histogram view that will display in the HTML report. You can add and/or remove graph entries by right-clicking in the dialog box and selecting Add Graph/Remove Graph from the shortcut menu.
2. Configure the graph entries by customizing the following options for each entry:
   
   • Graph Title: Enter a title for the graph as it will display in the HTML report.
   • View Type: Select to display either a Graph or a Histogram in the HTML report.
   • Filters: Select the desired choices from the Machines and Profiles drop-down menus and in the Tests tree selection view to filter HTML report graphs accordingly.
   • X Axis: (Only available for Graph View Types) Select the parameter to display as the X Axis of the graph.
   • Y Axis: Select the parameters to display on the Y Axis of the graph. You can select multiple parameters by pressing the Ctrl keyboard button while clicking your left mouse button. These parameters are the same parameters that display in the Graph tab legend of a Detailed Report view.

   • Log Scale: If you want to use a logarithmic scale, check the Log Scale check box. A logarithmic scale allows you to see the shape of multiple bars on the same graph (even if the displayed values are very far from one another).

   • Save in Project: Click to save the report to the current project. The next load test report generated for that project will have the saved HTML report options by default. However, if the report is a standalone report (loaded from disk rather than part of a current project) the Save in Project button will not be available.

   • View Report: Click to view the HTML report in your web browser.
   • Close: Click to close the HTML Report Options dialog box.
3. Click the View Report button. The HTML Report opens in your Web browser and displays the customized graph views.

Comparing Load Test Reports

After saving load tests as Binary reports, you can then load these reports into Load Test and compare and contrast various detailed report parameters. For example, after running and saving a load test, you may notice that the execution time for a particular operation is slow. After making the necessary changes to improve upon the execution time, you can run a load test for the new and improved service/application, and then compare its detailed report to that of the previously saved report, to see if the changes to the operation cause any new errors.

Within the Load Tests tab is a Reports node in which you can add report composites. Report composites contain references to the Load Test reports that were saved. Detailed report parameters from these different reports can be viewed in an overlaid graph, allowing you to compare any differences that may have occurred.

Loading Composite Reports

Before being able to view and compare multiple reports, you must first load the reports into Load Test. To load composite reports, complete the following:

1. Right-click the Reports node in the Load Tests tab and select New Composite from the shortcut menu. A Report Composite node displays beneath the Reports node and the Report Composite controls display in the right GUI panel.

2. Click the New button at the bottom of the right GUI panel and browse to and open a saved report file (.rpt) in the dialog box that displays. A node corresponding to the file you opened displays in italics beneath the Report Composite node in the Load Tests tree.
   
   

   Report composite entries are displayed in italics to emphasize that they are references to reports. By default, when a project is reopened, or when a new reference is created, the report composite is not loaded and displays in gray. After a report composite is loaded, it displays in black.

3. Repeat Step 2 until you have added all of the report composites you would like to load.

4. Right-click the Report Composite node and select Load All from the shortcut menu. Load Test will load each of the report composites you added.

   You may also load individual report composites by selecting the appropriate node from beneath the Report Composite node and clicking the Load button in the right GUI panel.

5. (Optional) You may access the Test Information, Statistics, and Detailed Report views of each individual report composite by selecting the desired report composite node in the Load Test tree and then selecting from the Views drop-down menu in the right GUI panel.
   
   

Working with and Comparing Composite Reports

After loading multiple load test reports, you can select the Report Composite node beneath the Reports node of the Load Tests tree to configure and compare the composite reports. The following options are available  in the Report Composite controls

• Name: Enter a name for the Report Composites folder that will contain the report composites that you add and load.
• Persist as Relative Path: Select to ensure that the path of the composite reports you add are always relative to your project file.
• Views: The following views are available in the Report Composite controls.
  • Information: Contains a table of report sources and titles. The Report Source is a file path to the report. By default, the Report Title is also a file path to the report, but can be changed by double-clicking a table row and changing the Report Title in the dialog that opens.
  • Graph: Contains an overlaid graph that displays various detailed report parameters from each of the composite reports that you loaded.
    To compare detailed report parameters within the Graph view, select the desired parameter from the Y-axis drop-down menu and then select the desired composite reports from the legend at the bottom of the right GUI panel.  Now you can graphically compare the multiple load test reports to visualize performance changes between load test runs.
    
    
    
    If you want to use a logarithmic scale, check the Log Scale check box. A logarithmic scale allows you to see the shape of multiple bars on the same graph (even if the displayed values are very far from one another).

Viewing the Active Load Test Report 

The Active Report gives you access to all the data collected for a running load test scenario—during test execution, without having to stop the load test.

The Active Report contains the load test data collected up to the moment when the Active Report was opened; it includes all the data of a standard Load Test report, including the statistical data, the individual hit data, the traffic, and the error details (if those were configured to be recorded). The Active Report can be refreshed at any time while the load test scenario is running.

The Active Report should be used only when you need to see load test details that are not available in the Load Test Progress view—for example, if you want to see error details while the load test is running.

To access the Active Report:

1. Launch a new instance of the Load Test application.
   • Accessing the Active Report from the Load Test process that is running a load test scenario is not allowed. This is because opening and manipulating large reports may consume considerable memory and CPU resources, which may adversely affect the accuracy of the running load test scenario.
   • The Load Test process that is used to access the Active Report must be launched from the same installation as the process that is running the Load Test scenario you want visibility into.
2. Choose File> Active Report.

You can manipulate and save the Active Report in the same way as the standard Load Test report.

To refresh the Active Report with the most up to date data:

• Click the Refresh button in the report view header.
  
  

CPU Utilization Notes

• Use discretion in how often you update the Active Report view; reloading large reports can take considerable CPU resources from the machine and affect the accuracy of the Load Test scenario execution. 
• Make sure that the average CPU utilization of the machine running the load test does not exceed 80% (on average).
• If you need to free the CPU resources on the load test controller machine for opening and manipulating the Active Report while the load test is running, redistribute the load to the remote machines.

Viewing the Active Report of a Load Test Running in a Command Line Mode

The Active Report can be used to periodically assess the progress of load tests that are running in command line mode. This is especially valuable for getting feedback on load tests that are configured to run for extended periods of time. 

When tests are running in command line mode, you can access the Active Report in the same way as you would for tests running in the GUI mode.

Recovering Load Test Reports

You can typically use the Load Test report recovery feature to restore the load test data after a system crash or an application shutdown. To view recoverable reports:

1. Choose File> Recover Report. The Recover Report dialog will show the start date/time(s) of recoverable load test reports. Recovery data is deleted 5 days after creation. Report data that is more than 5 days old cannot be recovered.
   
   
   
   
2. Select a report and click OK.

After a load test report is restored, you can save it in the format of your choice. 

Report Terms Defined

The following are definitions for terms used in load test reports:

• Inter-Invocation Time: Time between subsequent requests "on the wire". The requests may be coming from different VUs.
• Server Time: The time from the last byte sent to the first byte received.
• Total Execution Time (Execution Time): The time from the first byte sent to the last byte received.
• Request Send Time: The time from the first byte sent to the last byte sent.
• Response Receive Time: The time from the first byte received to the last byte received.
• Ping Time: Time to establish an HTTP connection to the SOAP endpoint under test. This time is used as an estimate of the network layer latency.
• Total Size: Total size of the request and the response.

The following diagrams illustrate how these metrics correspond to actual test actions.

The first two diagrams show the request send time:

Now the message is in transit to and from the server, including application "think time":

The next two diagrams show the response receive time:

The final diagram shows how the execution time is the total of the above operations:

Sending QoS Results to DTP

You can configure Load Test to send QoS metric results to Parasoft Development Testing Platform so that it can be associated with tasks, requirements, and other artifacts. The success/failure of the metrics will indicate the success/failure of the testing task. If multiple metrics are associated with a single testing task, a failure of any one metric will cause the associated testing task to fail.

To configure this reporting, you need to:

• Correlate Load Test scenarios to items in DTP (tasks, requirements, bugs).
• Configure the Load Test installation (the one that will be running the correlated tests) to report to DTP.

Configuring Reporting to DTP

1. Choose File> Customize Preferences to open the Preferences panel.
2. Open the Report Center / Project Center page.
3. Enable Send results to Report Center and Project Center.
4. In the Host name field, enter your team’s Parasoft Development Testing Platform (DTP) server host (either a name or an IP address).
5. In the Data Collector port field, enter your team’s DTP Data Collector port (default is 32323)
6. In the General project field, enter the name of the DTP project that this testing task is associated with.
7. Click Test Connection to verify the connection.
8. (Optional) Add custom  attributes as directed by your team leader.

Correlating Load Test Scenarios to DTP Artifacts

To configure correlations:

1. Select the load test tree’s Scenarios node.
   
   
   
   
2. In the configuration panel on the right, open the Requirements Tracking tab.
3. Click the Add button. A new entry will be added to the table.
4. In the Type box, select a requirement type. DTP will use this information to associate the test suite’s test cases to the specified element type. Custom tags can be added as described in Indicating Code and Test Correlations. Default tags are:
   • @pr and @bugzilla: for bugs.
   • @fr: for feature requests.
   • @req: for requirements.
   • @task: for tasks.
5. Enter an ID and a URL for the requirement.
   
   
   
   
6. Save the load test project.

Troubleshooting: Connection Reset Errors

When reviewing Parasoft Load Test results, you may occasionally see "Connection Reset" errors. 

Basically, a "Connection Reset" error is typically generated by overloaded servers, in which case it is caused by the OS TCP stack(s) and propagated up to the application layer as a Java net exception.

In more detail, "Connection Reset" is not an error message unique to Parasoft Load Test; it is generated by Java and it corresponds to specific low-level events. Connection Reset will be reported by Java (in this case the JVM running Load Test) when it uses an RST (short for reset) message to abort the connection. This will happen if the JVM encounters any prematurely closed connections on the server side. A prematurely closed connection indicates that while Load Test is receiving the response, the server is closing the connection—thus causing the Load Test JVM to forcibly issue a "reset" that aborts the connection. This is not a feature of Load Test; rather, it is a behavior of the JVM in order to comply with the TCP protocol specifications.

It is obvious that the Connection Reset is generated by Load Test because a reset indicates that the connection has already been closed on one side (in this case the server side). If a server application were issuing an RST, Load Test would not receive any information of this since the connection would have already been terminated.

The Connection Reset can be difficult to understand. It is easiest to think of it as a symptom of a problem that has already happened. In other words, the reset connection is not the problem—the problem is the prematurely closed connection that occurred on the server which caused the Connection Reset in Load Test.

From the java.net.Socket JavaDoc:

"Under abnormal conditions the underlying connection may be broken by the remote host or the network software (for example a connection reset in the case of TCP connections)."

To learn more about Connection Reset errors, see http://download.oracle.com/javase/6/docs/technotes/guides/net/articles/connection_release.html .