Versions Compared

Old Version 47

changes.mady.by.user Robert Andelin

Saved on

compared with

New Version Current

changes.mady.by.user Robert Andelin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The primary audience for this user guide are is people responsible for ensuring compliance with your organization's policy regarding the application coverage level, including QA Engineers, developers, and build masters.

...

We assume that you are familiar with Parasoft technologies and have already deployed and licensed the following products:

  • dotTEST or Jtest
  • Coverage Agent Manager (CAM) or SOAtest
  • Parasoft DTP

The following options must be configured in the .properties file where you configure your dotTEST or Jtest settings to ensure that coverage data is correctly displayed on DTP:

  • report.coverage.images -  – Specifies a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. DTP supports up to three coverage images per report.
  • session.tag - – Specifies a unique identifier for the test run and is used to distinguish different runs on the same build.
  • build.id - – Specifies a build identifier used to label results. It may be unique for each build, but it may also label several test sessions executed during a specified build.

...

  1. Generating the static coverage file.
    The static coverage file is generated by Jtest or dotTEST in the .xml or .data format and contains metadata about user classes, methods, and lines.
  2. Attaching the coverage agent to the application under test (AUT).
    The coverage agent ships with Jtest and dotTEST, and allows you to monitor the code being executed when the AUT is running.
  3. Configuring the Parasoft product that you use for application testing: SOAtest.
         SOAtest must be configured to connect to the coverage agent and collect runtime coverage, merge it with static coverage, and upload the data to DTP.Coverage Agent Manager (CAM) must be configured to connect to the coverage agent.
  4. Performing tests on the AUT with your Parasoft product:SOAtest.
    SOAtest will collect runtime coverage, merge it with static coverage, and upload the merged data to DTP.
  5. CAM will collect runtime coverage and test results, which must be manually downloaded and sent to DTP (see steps 5-7).
  6. (CAM only) Downloading test results and the dynamic (runtime) coverage.
    The data can be downloaded when a test session is finished and stopped in CAM.
  7. (CAM only) Uploading the test results to DTP.
    Test results are uploaded via the DTP interface.
  8. (CAM only) Merging the dynamic and static coverage data and sending it to DTP.
    Static and dynamic coverage is merged and uploaded to DTP by Jtest or dotTEST.
  9. Reviewing the application coverage on DTP.
    Application coverage is displayed in the DTP's Coverage Explorer.

...

  1. Reviewing the application coverage on DTP.
    Application coverage is displayed in the DTP's Coverage Explorer.

Anchor
step_1
step_1
Step 1: Generate the Static Coverage File

The static coverage file must be generated on the build machine that contains the source code. The name of the file is static_coverage.xml.

Generating Static Coverage with dotTEST

Run the following test configuration on the solution:

Code Block
dottestcli.exe -config "builtin://Collect Static Coverage" -solution SOLUTION_PATH

The location of the static_coverage.xml file will be printed to the console.

Anchor
static_jtest
static_jtest
Generating Static Coverage with Jtest

The static coverage file is included in the monitor.zip

The static coverage file must be generated on the build machine that contains the source code. The name of the file is static_coverage.xml.

Generating Static Coverage with dotTEST

Run the following test configuration on the solution:

Code Block
dottestcli.exe -config "builtin://Collect Static Coverage" -solution SOLUTION_PATH

The location of the static_coverage.xml file will be printed to the console.

...

The static coverage file is included in the monitor.zip package generated by the Jtest Plugin for Maven, Gradle, or Ant during the build process.

  1. Add the monitor task or goal to your build command and execute the command in the AUT's main directory to generate the monitor.zip package. The location of the package will be printed to the console.

    Maven

    Code Block
    mvn package jtest:monitor

    Gradle

    Code Block
    gradle assemble jtest-monitor -I [INSTALL]/integration/gradle/init.gradle

    Ant

    Code Block
    ant -lib [INSTALL]/integration/ant/jtest-ant-plugin.jar -listener com.parasoft.Listener jtest-monitor
    Info

    Ant requires all classes to be compiled before the monitor task is executed. Modify your project prior to the build and configure the task to ensure the correct sequence. The following example shows how the target can be configured:

    <target name="jtest-monitor" depends="compile">
                 <jtest:monitor/>
    </target>

  2. Extract the contents of the monitor.zip package to the server machine.

...

  1. Deploy the AUT to the application server.
  2. Copy the [DOTTEST<DOTTEST_INSTALLATION_DIR]DIR>\integration\IIS directory to the machine where IIS is installed and the AUT is deployed.
  3. Run a console as Administrator.

  4. Invoke the following command to launch the IIS Manager tool:

    Code Block
    dottest_iismanager.exe

  5. Go to the following address to check the status of the coverage agent: http://host:8050/status. If the Coverage Agent is attached, you should receive the following response:

    Code Block
    {"session":null,"test":null}

...

  1. Deploy the AUT to the application server.
  2. Extract the contents of the monitor.zip package you generated (see Generating Static Coverage with Jtest) to the server machine. The package contains the agent.sh/agent.bat script.

  3. Run the agent.sh/agent.bat script to print the Jtest Java agent VM argument to the console:

    Code Block
    Jtest Agent VM argument:
-javaagent:"[path to agent dir]\agent.jar"=settings="[path to agent properties file]\agent.properties",runtimeData="[path to monitor dir]\monitor\ runtime_coverage"
  4. Add the -javaagent argument to the application server’s startup script.
  5. Restart the server.

  6. Go to the following address to check the status of the coverage agent: http://host:8050/status. If the Coverage Agent is attached, you should receive the following response:

    Code Block
    {"test":null,"session":null,"testCase":null}

Anchor
connect_cam_agent
connect_cam_agent
Step 3:

...

Configuring SOAtest 

You need to configure the Parasoft product that you use for application testing. You can use either Coverage Agent Manager (CAM) or SOAtest. CAM provides an interface for starting and stopping test sessions during manual test execution. Connecting SOAtest enables you to collect application coverage during automated functional test execution.  configure SOAtest, which SOAtest enables you to collect application coverage during automated functional test execution.  

  • You need to configure SOAtest to:If you use CAM, you need to configure CAM to
      • connect to the Coverage Agent.
      If you use SOAtest, you need to configure SOAtest to:
      • connect to the Coverage Agent.
      • collect runtime coverage.collect runtime coverage.
      • merge runtime coverage with static coverage (see Step 1).
      • upload the merged data to DTP.

    Anchor
    connect_SOA
    connect_SOA
    Connecting

    ...

    SOAtest to the Coverage Agent

    ...

    1. Open CAM in a browser:

      Code Block
      http://[your-Tomcat-host:port]/cam.
    2. Click Agents.
      Image Removed
    3. Click Add Connection to configure the connection with the Coverage Agent.
      Agent Name: Any user-friendly string that you want to display in CAM (for example, the name of the application).
      Agent Address: The URL of the server where the Coverage Agent is attached to the AUT (see Step 2). The default port number for the Coverage Agent is 8050.
    4. Click Test to verify the connection.
      Image Removed
    5. Click Connect after successfully testing the connection.

    ...

    and Enabling Coverage Upload to DTP

    1. Open SOAtest and go to Preferences > Reports > Coverage, then specify the installation directory of dotTEST or Jtest.
    2. Configure a SOAtest test configuration to collect, merge, and upload application coverage data. You can configure an existing test configuration or create and configure a new one.
      1. Open the test configuration that you want to configure to collect application coverage data.
      2. Open the Execution > Application Coverage tab.
      3. Enable the Collect application coverage option and specify the host name or IP address where the application under test and coverage agent are hosted and the port number of the agent. The port number should match the value of the jtest.agent.port setting in the agent.properties file (default is 8050). You can click Test Connection to verify that SOAtest can communicate with the coverage agent. 
      4. (Optional) Under Coverage agent user ID, you can specify a user ID so that coverage results can be associated with a specific user. A user ID should only be specified when application coverage for multiple users is enabled for the coverage agent.
      5. Enable the Upload coverage report to DTP option to enable merging static and runtime coverage data and sending the merged coverage results to DTP.
      6. Configure the Static coverage file location option to specify the path to the static coverage file generated with dotTEST or Jtest in the .xml or .data format (see Step 1).
      7. Configure the Coverage image tags option to specify a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.
      8. Enable the Report coverage agent connection failures as test failures option for test failures to be reported when the coverage agent connection fails, or when the static coverage file location is misconfigured. Otherwise, the problems will be reported to the console, but will not cause the test to fail.
      9. Click Apply to save your changes.
      10. (Optional) Save the test configuration to your workspace. This step makes running the configuration on the SOAtest command line more convenient:
        1. Right-click the new test configuration and choose Export.
        2. Save the test configuration to the SOAtest workspace. 

    In the next step, you will execute your tests using the new test configuration from the command line. But in order for DTP to be associated the test results with the coverage information

    1. Open SOAtest and go to Preferences> Reports> Coverage and specify the installation directory of dotTEST or Jtest.
    2. Configure a SOAtest test configuration to collect, merge, and upload application coverage data. You can configure an existing test configuration or create and configure a new one.
      1. Open the test configuration that you want to configure to collect application coverage data.
      2. Open the Execution> Application Coverage tab.
      3. Enable the Collect application coverage option and specify the host name or IP address where the application under test and coverage agent are hosted and the port number of the agent. The port number should match the value of the jtest.agent.port setting in the agent.properties file (default is  8050). You can click Test Connection to verify that SOAtest can communicate with the coverage agent. 
      4. (Optional) Under Coverage agent user ID, you can specify a user ID so that coverage results can be associated with a specific user. A user ID should only be specified when application coverage for multiple users is enabled for the coverage agent.
      5. Enable the Upload coverage report to DTP option to enable merging static and runtime coverage data and sending the merged coverage results to DTP.
      6. Configure the Static coverage file location option to specify the path to the static coverage file generated with dotTEST or Jtest in the .xml or .data format (see Step 1).
      7. Configure the Coverage image tags option to specify a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. For details, see the Parasoft DTP user guide at https://docs.parasoft.com.
      8. Enable the Report coverage agent connection failures as test failures option for test failures to be reported when the coverage agent connection fails, or when the static coverage file location is misconfigured. Otherwise, the problems will be reported to the console, but will not cause the test to fail.
      9. Click Apply to save your changes.
      10. (Optional) Save the test configuration to your workspace. This step makes running the configuration on the SOAtest command line more convenient:
        1. Right-click the new test configuration and choose Export.
        2. Save the test configuration to the SOAtest workspace. 

    In the next step, you will execute your tests using the new test configuration from the command line. But in order for DTP to associated the test results with the coverage information, you will need to create a properties file and specify the settings required for DTP to properly match and display application coverage data. The simplest way is to export the properties referenced by the SOAtest desktop.

    1. Choose Parasoft> Parasoft > Preferences from the SOAtest menu.
    2. Click the share link on the Parasoft page.
    3. Specify a name and location to export the settings file.
    4. Enable any settings you want to include in the file and click OK.
    5. Open the exported file and add the following properties:
      1. report.coverage.images -  – Specifies a set of tags that are used to create coverage images in DTP. A coverage image is a unique identifier for aggregating coverage data from runs with the same build ID. DTP supports up to three coverage images per report.
      2. session.tag - – Specifies a unique identifier for the test run and is used to distinguish different runs on the same build.
      3. build.id - – Specifies a build identifier used to label results. It may be unique for each build, but it may also label several test sessions executed during a specified build. If you want to merge SOAtest coverage results with results from other types of testing—such as unit testing and manual testing of the same application—they must all use the same build ID. The build ID set in this file must match the build ID set for dotTEST or Jtest when collecting static coverage.
      4. dtp.project - – Specifies the project on DTP with which you want to associate the coverage results. The DTP project set in this file must match the DTP project set for set for dotTEST or Jtest when collecting static coverage.
    6. Save the changes.

    Step 4: Perform Testing Sessions

    Parasoft provides several ways to execute tests. You can execute tests manually or automate test scenario execution using Run SOAtest in the command line interfaces. In this guide, we will demonstrate executing manual test scenarios with CAM. We will also describe automated test execution with SOAtest. 

    Execute Test Scenarios with CAM

    To collect coverage data, you need to start a session. Only one tests session can be in progress at a time.

    1. Click the New Session button, enter the following information when prompted:
      • Session Name: the name of the session
      • Project: the name of the project in DTP that the results will be associated with
      • Build ID: A build identifier for label results. It may be unique for each build but may also label more than one test sessions that were executed during a specified build.
      This information is critical for the data to be properly merged and correlated in DTP.
      Image Removed
    2. Click the Start Session button to begin a new test session.
    3. Click the New Test button.
      Image Removed
    4. Enter a name and click Start Test to continue.
    5. Perform your manual test on the AUT and enter the data associated with the test.
      (info) Ensure that you select the correct Test result option (Pass/Fail/Incomplete).
      Image Removed
    6. Click Stop Test when finished.
    7. Continue adding and performing your tests until your scenario is complete.
      (info) Ensure that you stop and start tests in between interacting with the AUT.
    8. Click Stop Session when you’ve completed your tests.

    Execute Test Scenarios with SOAtest

    Run SOAtest in the command line mode using the test configuration and the .properties file created in the previous step. You will also need to include the workspace and test assets (.tst files).

    No Format
    ./soatestcli -config "/path/to/your/test-configuration/app-cov.properties" -data "/path/to/your/workspace" -resource "/path/to/your/tests/your_tests.tst" -localsettings "/path/to/your/localsettings/file/soatest-app-cov.properties"

    Step 5: (CAM only) Download Coverage and Test Results

    1. Click Download Coverage to download the runtime_coverage_[timestamp].data file and note the location.
    2. Click Download Test Results to download the report.xml file and note the location.
      Image Removed

    Step 6: (CAM only) Upload the Test Results to DTP

    Upload the report.xml file you downloaded from CAM to DTP. If you used SOAtest to execute your tests, the report will be saved in the location specified with the -report  flag. If you did not specify a location for the report, the file will be saved to the working directory.  Uploading test results to DTP allows you to associate coverage data with individual tests and view the information on DTP.  

    1. Go to Report Center in the DTP interface.
    2. Click the gear icon and choose Report Center Settings> Additional Settings> Data Collector Upload Form (requires admin permissions).
    3. Click Choose File and browse for the report.xml file you downloaded from CAM.
    4. Click the Upload button to upload the file to DTP.

    Step 7: (CAM only) Merge the Static and Dynamic Coverage Data and Send to DTP

    The static and dynamic coverage data must be merged into one coverage.xml file. To merge the coverage data and send the merged information to DTP, run dotTEST or Jtest with the following arguments:

    • -staticcoverage - Provides the path to the static_coverage.xml generated by dotTEST or Jtest
    • -runtimeCoverage - Provides the path to the runtime_coverage_[timestamp].data downloaded via CAM or generated by SOAtest. You can provide a path to a folder that contains many .data files from multiple testing sessions.
    • -publish - sends the merged coverage data to DTP
    • (Jtest only) -config "builtin://Calculate Application Coverage" - specifies the built-in test configuration required to merge the coverage data

    Your command line may resemble the following:

    dotTEST

    Code Block
    dottestcli.exe -staticcoverage [path] -runtimecoverage [path]  -publish

    Jtest

    Code Block
    jtestcli -staticcoverage [path] -runtimecoverage [path/dir] -config "builtin://Calculate Application Coverage" -publish

    (info) The -publish option is required to send the merged coverage data to DTP. Alternatively, you can configure the report.dtp.publish=true option in the .properties file where you configure dotTEST or Jtest.

    Step 8: Reviewing Application Coverage Results in DTP

    In order to view coverage in the DTP dashboard, you will need to associate the coverage data sent to DTP with the filter.

    1. Choose Report Center Settings from the DTP settings (gear icon) menu.
    2. Click Filters and locate your DTP project. By default, a filter is created in DTP for each project. The filter has the same name of the project.
    3. Click on the filter and verify that the appropriate run configurations have been added to the filter. A run configuration is a set of metadata about test and analysis executions that enable DTP to make the correct correlations for accurate reporting. If Jtest or dotTEST is configured to report to the project associated with the filter, then the run configurations will automatically be added to the filter by default.
    4. In the Coverage Images section, rename one of the image indexes to Application Coverage  and choose the coverage image tag you specified from the Coverage Image drop-down menu. 
      Image Removed
      You can include additional coverage images if you want to view all of your coverage data in one interface. 
      Changes are automatically saved.
    5. Return to the DTP dashboard and add a coverage widget configured to show the Application Coverage coverage image.
      Image Removed
    6. Specify the filter and build ID in the dashboard settings and the widget will show your coverage data. 
      Image Removed
    7. You can click on the widget to view details in the Coverage Explorer view.
      Image Removed

    Collecting Application Coverage from Multiple Users

    You can collect coverage information for multiple users that are simultaneously accessing the same Java or IIS web application server. This allows QA engineers to perform parallel testing sessions and associate coverage with individual users.

    Collecting coverage data from multiple users requires additional configuration steps before you start your testing session. You need to:

    1. Configure the Coverage Agent to enable the multiple-user mode; see Configuring the Coverage Agent to Run in the Multiple-User Mode.
    2. Specify your User ID for the connection with the Coverage Agent attached to the AUT. This will enable the Coverage Agent to identify and assign coverage information to individual users who are simultaneously interacting with the AUT.
      • If you use SOAtest, provide your User ID on the Application Coverage tab of your test configuration; see Connecting SOAtest to the Coverage Agent.
      • If you use CAM, add your User ID to the HTTP request header of the browser you will use to interact with the tested web application and then provide the same User ID when connecting CAM to the Coverage Agent; see Specifying User ID for Testing with CAM.

    ...

    mode using the test configuration and the .properties file created in the previous step. You will also need to include the workspace and test assets (.tst files).

    No Format
    ./soatestcli -config "/path/to/your/test-configuration/app-cov.properties" -data "/path/to/your/workspace" -resource "/path/to/your/tests/your_tests.tst" -localsettings "/path/to/your/localsettings/file/soatest-app-cov.properties"

    Step 5: Reviewing Application Coverage Results in DTP

    In order to view coverage in the DTP dashboard, you will need to associate the coverage data sent to DTP with the filter.

    1. Choose Report Center Settings from the DTP settings (gear icon) menu.
    2. Click Filters and locate your DTP project. By default, a filter is created in DTP for each project. The filter has the same name of the project.
    3. Click on the filter and verify that the appropriate run configurations have been added to the filter. A run configuration is a set of metadata about test and analysis executions that enable DTP to make the correct correlations for accurate reporting. If Jtest or dotTEST is configured to report to the project associated with the filter, then the run configurations will automatically be added to the filter by default.
    4. In the Coverage Images section, rename one of the image indexes to Application Coverage and choose the coverage image tag you specified from the Coverage Image menu. 
      Image Added
      You can include additional coverage images if you want to view all of your coverage data in one interface. 
      Changes are automatically saved.
    5. Return to the DTP dashboard and add a coverage widget configured to show the Application Coverage coverage image.
      Image Added
    6. Specify the filter and build ID in the dashboard settings and the widget will show your coverage data. 
      Image Added
    7. You can click on the widget to view details in the Coverage Explorer view.
      Image Added

    Collecting Application Coverage from Multiple Users

    You can collect coverage information for functional tests even when multiple users are simultaneously accessing the same Java or IIS web application server. This allows QA engineers to perform testing sessions in parallel with automation and associate coverage with functional tests.

    Collecting coverage data from multiple users requires additional configuration steps before you start your testing session. You need to:

    1. Configure the Coverage Agent to enable the multiple-user mode; see Configuring the Coverage Agent to Run in the Multiple-User Mode.
    2. Specify your User ID for the connection with the Coverage Agent attached to the AUT. This will enable the Coverage Agent to identify and assign coverage information to individual users who are simultaneously interacting with the AUT. In SOAtest, provide your User ID on the Application Coverage tab of your test configuration; see Connecting SOAtest to the Coverage Agent.

    Anchor
    enable_multiuser
    enable_multiuser
    Configuring the Coverage Agent to Run in the Multiple-User Mode

    Enabling the Multiple-user Mode with dotTEST

    You can enable the multi-user mode by modifying the invocation of the dotTEST IIS Manager tool (see Step 2: Attach the Coverage Agent to the Application Under Test (AUT)). Launch the tool with the -multiuser switch:

    Code Block
    dottest_iismanager.exe -multiuser

    Enabling the Multiple-user Mode with Jtest

    You can enable the multiple-user mode by modifying the options in the agent.properties file, which is included in the monitor.zip package (see Generating Static Coverage with Jtest). Open the file and configure the following option:

    Code Block
    jtest.agent.enableMultiuserCoverage=true

    Enabling the Multiple-user Mode with dotTEST

    You can enable the multi-user mode by modifying the invocation of the dotTEST IIS Manager tool (see Step 2: Attach the Coverage Agent to the Application Under Test (AUT)). Launch the tool with the -multiuser switch:

    Code Block
    dottest_iismanager.exe -multiuser

    Enabling the Multiple-user Mode with Jtest

    You can enable the multiple-user mode by modifying the options in the agent.properties file, which is included in the monitor.zip package (see Generating Static Coverage with Jtest). Open the file and configure the following option:

    Code Block
    jtest.agent.enableMultiuserCoverage=true

    ...

    Adding the User ID to the HTTP header

    Modify the the HTTP request header of your browser by configuring the Test-Operator-ID option with a unique User ID.

    A convenient way to add a Test-Operator-ID is to install one of the browser plugins that allow you to easily modify HTTP headers. The following example shows a Test-Operator-ID specified in Chrome with the ModHeader plugin.

    Image Removed

    Connecting CAM to the Coverage Agent in the Multiple-user Mode

    Provide your User ID when connecting CAM to the Coverage Agent (see Step 3: Connect CAM or SOAtest to the Coverage Agent). The User ID must be identical to the Test-Operator-ID provided to your browser.

    Image Removed

    Testing in the Multiple-user Mode

    Perform testing sessions, download the results, and merge the coverage data, as described in steps 4-7 above. When downloading the results from CAM, ensure that the Session Tag field is completed with a tag unique to each user. We recommend that the session tag include the User ID.

    ...