This topic provides a general introduction to the reports that SOAtest produces for GUI and CLI tests. Report details will vary based on report settings, the Test Configuration used, and the errors found. Not all of the report elements described below will be present in all reports.
Sections include:
Report Types
Two types of reports can be produced from the command line interface:
- Comprehensive reports: Reports that contain all tasks generated for a single test run.
- Individual reports: Reports that contain only tasks assigned to the specified team member.
For example, if a test generated five tasks for Tom and ten tasks for Joe, the comprehensive report would contain all 15 tasks, Tom’s report would contain five tasks, and Joe’s report would contain ten tasks.
Report Contents
Reports may contain the following sections:
Header
In addition to the report name, the header shows the tool used for the analysis.
Session Summary
The Session Summary section includes high-level information about the report and may include the following:
- Build ID
- Test configuration
- Time stamp of the analysis
- Machine name and username
- Session tag
- Project name
- Number of findings with the highest severity
- Number of failed tests
- Number of API security issues
Summary - Static Analysis
The Summary - Static Analysis summary section appears in the report whenever you perform static analysis on your project. It shows an overview of findings as a donut chart with various colors representing severities. The same information is shown in a table next to the chart and the total number of findings is shown below.
Summary - Functional Tests
The Summary - Functional Tests section appears in the report whenever a functional test is run. It will show one or more donut charts representing test coverage, API coverage, or impacted assets, based upon the type of tests you are running.
Details - Static Analysis
The Static Analysis details section appears in the report whenever you perform static analysis on your project and shows a table that summarizes findings as well as detail reports of those findings by type and author.
The Static Analysis table contains the following information:
- Module name
- Number of suppressed rules
- Total number of findings
- Average number of findings per 10,000 lines
- Number of analyzed files
- Total number of files in the module
- Number of code lines analyzed
- Total number of code lines in the module
Below this table, findings are listed by rule with the number of violations for each shown in brackets. You can sort these findings by category or severity by clicking the Category or Severity link to the right.
Below the All Findings list is a Findings by Author list showing findings by author associated with the analyzed code. Click an author to view their findings details.
Details - Functional Tests
The Details - Functional Tests section appears in the report whenever a functional test is run. It shows a table that summarizes test results as well as detail reports of findings by author.
The Test Suite Summary table contains the following information:
- Test name
- Tests within the test suite that failed
- Tests within the test suite that succeeded
- Total tests within the test suite
- Success rate as a percentage
Below this table, findings are listed by author. Click an author to view their finding details. In addition, you can click the error message summary for a finding to view the detailed error report and click View Traffic for any finding to see its associated request and response.
Details - API Coverage
The Details - API Coverage section appears in the report whenever you run tests with 'Calculate API coverage' enabled in the test configuration. It shows a table that summarizes how well your tests covered the related resources.
The API Coverage table contains the following information:
- Coverable resource (generally, a service definition like a RAML, OpenAPI/Swagger, WADL, or WSDL)
- Number of times a test invoked the coverable resources as well as how many succeeded
- Coverage expressed as the number of resources that were invoked vs. the total number of resources
The table is hierarchical. Coverage is calculated for a service as a whole, as well as each resource/method and operation and the tree branches can be collapsed and expanded.
If you want to see what .tsts and specific tests covered a certain method, drill down into the tree, then click the Show tests link. For quick expansion and collapsing, you can use the Expand All / Collapse All links at the top left of the report.
Details - Change Impact
The Details - Change Impact section appears in the report whenever you run change impact analysis against your services. It shows a table that summarizes assets affected by recent changes.
The Asset Summary table contains the following information:
- Affected test
- Number of test assets affected
- Number of test assets unaffected
- Total test assets
- Change impact as a percent
Below this table, findings are listed by author with total changes and changes per asset shown. Click an author to view their finding details and click a findings link to view a more detailed report.
API Security Issues
The API Security Issues section appears in the report whenever you run penetration tests against your project.
The API Security Issues table gives an overview of issues found, either by CWE or OWASP 2021 top 10 (as determined by your preference setting in Parasoft > Preferences > Reports > API Security), in a matrix sorted by risk level and confidence. Below that, security findings are sorted by CWE number or OWASP 2021 Top 10 number. Click either the CWE or OWASP number or description to jump to that section of the report for more details about those findings.
In addition to a brief description of the issue, you will also find instance URLs with their associated tests and other information, proposed solutions, and links to additional resources. You can also click View Traffic for any instance URL to see its associated request and response.
