In this section

Introduction

A test job is a set of one or more test scenarios that are associated with a specific environment configuration. When a test job is executed, the designated environment will be provisioned before the tests are executed. You can execute jobs on demand or with an automation tool, such as Jenkins (see Environment Manager Plugin for Jenkins). Test jobs are configured and run from the Test Scenarios page.

Reviewing Available Test Scenarios and Jobs

Test Scenarios

The test scenarios panel lists the .tsts in each connected SOAtest Server’s TestAssets folder. The panel is also updated automatically (see Integrating Virtualize Server and/or SOAtest Server with CTP for details). You can extend and modify the test suites directly from CTP. See Building Scenarios and Tests for details.

Click on a a scenario in the panel to view details.

You can use the search bars in test scenarios and jobs panels to locate specific assets. Metadata and names are included in the search.

Jobs

The Jobs panel allows you to create, search filter, review, execute, and remove jobs, as well as review/remove job execution results.

The Jobs panel is automatically populated with jobs for the available SOAtest Test Executor component instances. In addition, the jobs are synchronized with the associated SOAtest Test Executor component instances, so changes are automatically applied if .tsts are added, removed, or reconfigured in the SOAtest Test Executor.

You can filter the available jobs to focus on specific systems and environments of interest.

Adding Test Jobs

1. You can either click the add new job button (+) in the jobs list or click Create Job in the scenario details view to begin adding a job. 
    
2. Specify a name for the test job. The job will automatically be populated with the selected test file.
3. Specify a test configuration to run when the job executes. If no test configuration is specified, then the default configuration on the server will be used.  
4. CTP automatically saves a history of job executions. If you want to limit the saved history, specify the maximum days and/or runs you want to allow. 
5. Enable the Batch (fastest) option to execute tests on each designated server or enable the Sequential option to control the execution order. The Batch (fastest) option optimizes performance, but the Sequential option enables you to drag and drop tests into a specific order. The Sequential option also generates a separate report for each test scenario. 
    
6. Click Add Test Scenarios and click Add to specify which test scenarios you want to run in the job. By default, each scenario will be added with the variables from the active variable set (also known as a SOAtest environment). You can hover over the test name and choose which variable set to add with each scenario. The green ball icon marks the environment that is currently active for the test scenario on the SOAtest server.
   

7. Click on a scenario to view the environment variables and data sources available for each scenario.
    

8. If multiple environments are available for the scenario, you can choose which environment to enable from the Variable Set drop-down menu. Variable sets defined in the test scenario will automatically be configured to use the existing values, but you can choose Literal from the drop-down menu and specify specific values. Literal fields are pre-populated with local variables if available. See Configuring Variable Mappings for additional information
    
9. If the .tst contains references to data sources, the job will run the rows of data specified in each data source from the .tst by default. You can enter all to configure the job to run all rows.
   
   You can choose a data source from the drop-menu and specify which rows to run.
    
10. (Optional) You can specify a context to be provisioned upon test execution by specifying the appropriate system, environment, and set of component instances.
    
    1. Choose a system, environment, and an instance from the drop-down menus. You can also choose Custom from the Select an Instance drop-down menu to manually configure the active component instance settings.

    2. Verify configuration of the component instances to be provisioned and make any adjustments.
        

11.  Click Save.

Executing Jobs

Jobs can be executed directly from the UI, or as part of an automated Jenkins job.

Executing Test Jobs from the UI

To use the UI to kick off a test job and provision any associated environments:

1. From the Jobs panel, open the job you want to modify.
2. Click Execute.

The specified test environment will be provisioned, then the tests will be executed. Progress and results will be indicated in the Jobs panel.

Executing a Job without Affecting Team Members

You may might want to quickly run an existing job with new environment and/or variable settings without saving changes to that job and potentially impacting other team members who are also working with that job. In this case, just select the job, configure the environment context and/or variables, and click Execute without first saving the job. Be sure to select the top-level job node rather than a time-stamped job history node. Other changes, such as renaming the job, setting a different history limit, test configuration, requires saving the updates. Users with the provision role can make adjustments in order to customize their job executions, but cannot save changes.

Executing Test Jobs Automatically

The Parasoft Environment Manager Plugin for Jenkins (contact your Parasoft representative for download information) can automatically run test jobs as part of a Jenkins job. This plugin is designed to help you rapidly configure various actions needed for automated, continuous testing across your software delivery pipeline. For more details, see the description and documentation available on Marketplace.

Executing a Subset of Tests in a Job

You can disable tests inside of a job so that they do not run when executed. 

This simplifies ad-hoc testing by enabling you to add a full suite of tests that can be disabled as necessary.  

Reviewing Test Job Results

To review a test job’s execution results:

1. From the Jobs panel, expand the associated test job, then click the test run you want to review.
2. Click View report to open the execution report.

Icons in the Jobs panel indicate test outcomes.

Managing Test Jobs

Modifying Test Jobs

To modify a test job’s .tsts or execution settings:

1. From the Jobs panel, open the job you want to modify.
2. Make the desired modifications.
3. Click Save.

Cloning Test Jobs

To clone an existing test job:

1. From the Jobs panel, open the job you want to duplicate.
2. Click the Clone Job icon.

Clearing a Test Job’s History

To clear a job’s history:

1. From the Jobs panel, open the job whose history you want to clear.
2. Click Clear Job History.

Removing Test Jobs or Test Runs

To remove a test job or test run:

• Click the Delete icon in the Jobs panel.

Replacing Tests

You can replace .tst files within a test scenario that's part of job without affecting the job. See Replacing Test Files.

Distributing Test Job Execution Across a Cluster of Servers (Execution Group)

If you want to distribute test job execution across an "execution group" (a cluster of SOAtest servers grouped under the same server name), ensure that those servers all have:

• The same name (this is specified in the SOAtest Preferences> Environment Manager panel).
• All of the .tst files you want executed in this manner.

The first of these servers to connect to CTP will be treated as the primary server in the execution group; the others will be considered alternates. The SOAtest server page shows only the primary servers (one server for each unique server name).

The page for a specific SOAtest server contains a table of the other servers in this same "execution group," along with their current status (online or offline). When the primary server is refreshed, all servers in that execution group will be refreshed.

The list of servers in the execution group is also shown when you select the primary server in the Test Scenarios page.

To run the distributed tests, simply ensure that all servers are running, then configure and execute the desired job to run on the primary server. If that primary server is busy executing another job, CTP will execute it on one of the other servers within that cluster.