This topic explains how to define a custom test execution flow for cross-platform testing (e.g., for building tests using the cross compiler and then performing platform-specific operations for deployment, test execution, and result collection).
Section includes:
When you run a Test Configuration set to execute tests, C++test performs a series of actions that usually lead to the unit testing results being loaded into the C++test UI. These actions are specified in a test flow definition, which is stored in XML format and saved as part of the Test Configuration. All built-in Test Configurations that perform test execution have corresponding preconfigured test flow definitions.
In addition to providing preconfigured execution flows designed specifically for host-based testing, C++test also allows you to customize test flows to provide an easy way of working with non-standard environments.
You can configure test execution flow parameters to suit your specific needs by using the editor available in the Execution> General> Execution Details section of the Test Configuration manager.
In most cases, there is no need to create a custom version of the execution flow because built-in Test Configurations allow easy editing of the most critical and commonly-modified flow properties.
To define a custom execution flow for a user-defined Test Configuration:
Find that property definition in the test flow. It will look like:
<SetProperty key="property_key" value="property_default_value" |
Add two additional attributes — uiEditable="true" and displayName="user_friendly_name" — so that the property definition looks as follows
<SetProperty key="property_key" value="property_default_value" uiEditable="true" displayName="This is a customizable property" /> |
Since the uiEditable="true" attribute is added to the SetProperty flow step, this property will be customizable in the properties table. It will be listed here using the value of the displayName attribute. The value attribute will be used as a default value.
The customizations will then be saved with the Test Configuration.
We recommend that you try to modify the properties list (as described above) first, then—if additional flow customization is required—you define a custom test flow.
Custom test execution flow definitions are stored in XML format and saved as part of the Test Configuration (which makes it possible to share it across the team). Typically, such definitions describe the sequence of steps necessary to perform unit testing in a non-standard environment. They can also be used to start any command line utility. In fact, a custom test flow definition can include C++test internal actions or external utilities started as processes in the operating system. The default test flow can be modified by:
In embedded environments where TCP/IP sockets are available, it is common to use them to automate the test flow. Test execution using a socket communication channel typically involves:
If necessary, modify this Test Configuration’s test flow to suit your specific needs. In most cases, the only difference is the extension of the generated executable file and the settings of specific network-related values (ports, IP number etc.).
The first part (TestRunnerWithSocketsGenerationStep) prepares the test framework to run using specific network values (IP, ports – taken from property file). The second part compiles the test executable. The last part (LinkStep) is responsible for calling the linker in order to obtain the final test exec (called $ {project_name}Test.foo here; you will need to change the extension to match your system's specifics). Note that the TestRunnerWithSocketsGenerationStep should be used instead of the default TestRunnerGenerationStep. |
<CustomStep
id="run_socket_listeners"
label="Running Socket Listeners..."
commandLine=""java" -cp "${cpptest:cfg_dir}/../lib/source/socket_listener"
SocketListener --channel "${cpptestproperty:results_port}@$
{cpptest:testware_loc}/cpptest_results.tlog" --channel
"${cpptestproperty:coverage_port}@$
{cpptest:testware_loc}/cpptest_results.clog" -sf
"${cpptest:testware_loc}/sync_file" -to 60"
workingDir="${cpptest:testware_loc}"
result="${cpptest:testware_loc}/cpptest_results.res"
runInBackground="true"
/>
<CustomStep
id="run_synchronization"
label="Running Synchronization..."
commandLine=""java" -cp "${cpptest:cfg_dir}/../lib/source/socket_listener"
Synchronize -sf "${cpptest:testware_loc}/sync_file.init" -to 60"
workingDir="${cpptest:testware_loc}"
result="${cpptest:testware_loc}/cpptest_results.res"
runInBackground="false"
/>
<CustomStep
id="run_test_exec"
label="Running Tests..."
commandLine="${cpptest:testware_loc}/${project_name}Test.foo" workingDir="${cpptest:testware_loc}"
result="${cpptest:testware_loc}/cpptest_results.res"
runInBackground="false"
/>
<CustomStep
id="run_synchronization"
label="Running Synchronization..."
commandLine=""java" -cp "${cpptest:cfg_dir}/../lib/source/socket_listener"
Synchronize -sf "${cpptest:testware_loc}/sync_file.final" -to 60"
workingDir="${cpptest:testware_loc}"
result="${cpptest:testware_loc}/cpptest_results.res"
runInBackground="false"
/> |
The above steps perform the following actions (in the listed order):
Run the Synchronize Java class. Note that the first step has the runInBackground="true" attribute; this means that the next step can run without waiting for that one to complete. This is necessary because SocketListener does not complete until the listening stops. When SocketListener initializes and is ready to receive results, it creates the synchronization file (sync_file.init, in this case) so that the run_synchronization step can complete and test_exec can be run. This prevents test execution from starting without proper listening agent initialization, which would cause the test results to be lost.
Selecting External Embedded debugging mode can only be done by direct modification to the Test Flow recipe. Insert the following line into the Test Flow recipe somewhere near the start of the first <RunnableExecution> section; among other unpublished properties:
<SetProperty key="emb_dbg" value="true" />
For examples, please review the built-in Test Configurations for environments for which the External Embedded mode is supported (see appropriate chapters).
For information about External Embedded debugging mode see Debugging Test Cases.
Entry point macros control the testing entry point. By default, main() is called. However, sometimes the tests can only be called from a different point in the program you are testing, or need to be called in a special manner. You can control this using the following macros:
| Name | Description |
|---|---|
| CPPTEST_EPT_main | If defined, the 'int main(int, char*[])' is used as an entry point. |
| CPPTEST_EPT_wmain | If defined, the 'int wmain(int, wchar_t*[])' is used as an entry point. |
| CPPTEST_EPT_WinMain | If defined, the 'int WINAPI _tWinMain(HINSTANCE, HINSTANCE, LPSTR, int)' is used as an entry point. |
| CPPTEST_EPT_WinMain | If defined, the 'int WINAPI _tWinMain(HINSTANCE, HINSTANCE, LPWSTR, int)' is used as an entry point. |
| CPPTEST_EPT_void_main | If defined, the 'void main(int, char*[])' is used as an entry point. |
| CPPTEST_EPT_main_no_args | If defined, the 'int main(void)' is used as an entry point. |
| CPPTEST_EPT_void_main_no_args | If defined, the 'void main(void)' is used as an entry point. |
| CPPTEST_ENTRY_POINT_C_LINKAGE | If defined, the main function declaration starts with 'extern "C"' if compiled as c++ code. |
You can also define the CPPTEST_ENTRY_POINT macro. If this macro will be defined, the generated main function will look like this:
CPPTEST_ENTRY_POINT_RETURN_TYPE
CPPTEST_ENTRY_POINT(CPPTEST_ENTRY_POINT_ARGS)
{
CppTest_Main(CPPTEST_ENTRY_POINT_ARGC, CPPTEST_ENTRY_POINT_ARGV);
CPPTEST_ENTRY_POINT_RETURN_STATEMENT
} |
CPPTEST_ENTRY_POINT_RETURN_TYPE, CPPTEST_ENTRY_POINT_ARGS, CPPTEST_ENTRY_POINT_ARGC, CPPTEST_ENTRY_POINT_ARGV and CPPTEST_ENTRY_POINT_RETURN_STATEMENT have the following default values that can be redefined:
| Name | Value |
|---|---|
| CPPTEST_ENTRY_POINT_RETURN_TYPE | int |
| CPPTEST_ENTRY_POINT_ARGS | void |
| CPPTEST_ENTRY_POINT_ARGC | 0 |
| CPPTEST_ENTRY_POINT_ARGV | 0 |
| CPPTEST_ENTRY_POINT_RETURN_STATEMENT | return 0; |
You can also define the CPPTEST_ENTRY_POINT_DEFINED macro. Defining this macro prevents the main routine from generating. In these cases, you need to call a CppTest_Main(0, 0) function to execute the test cases.
You must include the cpptest.h header file in the source file that contains the CppTest_Main(0, 0) call. Do not call the CppTest_Main(0, 0) from a function that is called during unit testing. Doing so will result in a loop where CppTest_Main(0, 0) is called infinitely.
Be sure to control changes to the source file with the PARASOFT_CPPTEST macro. For example:
#ifdef PARASOFT_CPPTEST
#include "cpptest.h"
#endif
...
#ifdef PARASOFT_CPPTEST
CppTest_Main(0, 0);
#endif |
These macros can be set in the Compiler Options area of the Project Options’ Build Settings.
See Setting Project and File Options for details on project options.
These are highest level elements (besides the 'FlowRecipeTemplate' document root) of every recipe. They are containers for execution flow steps (commands): steps have to be placed inside their context and are executed sequentially in it. However, they cannot be nested. Introducing a new one requires closing its predecessor.
Unconditionally execute contained commands.
Example:
<RunnableExecution>
<Echo msg="Hello world!" />
</RunnableExecution>" |
Conditionally execute contained commands. Useful when test value contains one of C++test execution flow variables described later.
Attributes:
Example:
<ConditionalExecution value="${cpptest:os}" equals="unix">
<Echo msg="This is UNIX-like OS." />
</ConditionalExecution>" |
Execute contained commands in a loop until all Test Cases have been executed. This element is only useful/valid when contains the test execution step (a 'CustomStep' launching the Test Executable) followed by the 'ReadTestLogStep'. Inside the {cpptestproperty:test_case_start_number} variable is updated automatically and shall be passed to the '--start-after=' parameter of the Test Executable.
Example:
<ExecuteTestsExecution>
<CustomStep commandLine=""${cpptest:testware_loc}/
${project_name}Test.exe" --start-after=${cpptestprop-erty:test_case_start_number}"
...other attributes... />
<ReadTestLogStep testLogFile="${cpptest:testware_loc}/cpptest_results.tlog" />
</ExecuteTestsExecution>" |
Internal step used to analyze information about missing symbols. It can be also used to automatically generate stubs for missing symbols if the appropriate attribute is set. If (after optional stub generation) there are still some missing symbols, execution flow will be stopped if the Execution -> Symbols -> Perform early check for potential linker problems option is checked in the Test Configuration.
Attributes:
Example:
<AnalyzeMissingDefinitions generateStubs="true" />
Internal step that bundles the included test suite files with the appropriate project source files.
Example:
<AppendIncludedTestCases />
Internal step used to clear temporary static and dynamic coverage data. It should be included at the end of the execution flow if there is no need to store static coverage data for future use. If you have two execution flow recipes that are meant to be run in sequence (for example, "Build test executable" and "Read test results"), only the second one should contain ClearTempCoverageData step at the end.
Example:
<ClearTempCoverageData />
Internal step that compiles the instrumented source files to be linked into the test executable.
Example:
<CompileStep />
Stops test execution flow if given the condition is fulfilled or not fulfilled (depending on attributes).
Deprecated - This command will be removed in subsequent versions of C++test. Use combinations of 'ConditionalExecution' control element and unconditional 'Stop' instead.
Attributes:
Example:
<ConditionalStop property="test_exec_timeouted" equals="true" />
Internal step used to create stub configuration xml file. The location of the resulting file is controlled by the "stub_config_file" execution flow property.
Example:
<ConfigureStubs />
Runs external executable.
Attributes:
Example:
<CustomStep
id="ctdt_nm"
label="CTDT Generation - extracting symbols..."
commandLine=""${cpptestproperty:nm}" -g ${cpptest:test_objects_quoted}"
workingDir="${cpptest:testware_loc}"
stdOut="${cpptest:testware_loc}/ctdt_nm.out"
result="${cpptest:testware_loc}/ctdt_nm.out"
bypassConsole="true"
dependencyFiles="${cpptest:test_objects_quoted}"
/> |
Internal step used to produce a stub configuration header file to be included by the instrumented sources. The location of the resulting header is controlled by the "stub_config_header_file" execution flow property.
Example:
<CreateStubConfigHeader />
Prints given message on the C++test console or to the file.
Attributes:
Example:
<Echo msg="Hello world!" />
Internal step instruments the project source files.
Example:
<HarnessInstrumentationStep />
Links the test executable using previously prepared object files.
Attributes:
Example:
<LinkStep result="${cpptest:testware_loc}/${project_name}Test.exe" />
Internal step used to analyze information about used and available symbols.
Attributes:
libSymFile - Specifies the file with external symbols list to read. Optional.
Example:
<LsiStep />
Internal step that precompiles the project source files for the LSI analysis.
Example:
<PrecompileStep />
Converts managed data sources into the format supported by C++test test execution runtime library— either a CSV file or a source file containing a source code array to be compiled into test executable.
Attributes:
Example:
<PrepareDataSources limit="100" type="csv" />
Reads the log file containing coverage results from the test execution.
Attributes:
covLogFile -- File to be read. Can contain '*' and '?' wildcards to read multiple files at once. To read the series of files obtained by using split file communication channel (see File Communication Channel Implementation), specify the path to the first file (e.g., 'cpptest_results.clog') and make sure that all the files are placed in the same directory and that their names follow the original naming scheme ('cpptest_results.clog', 'cpptest_results.clog.0001', 'cpptest_results.clog.0002', etc.). The rest of the files in the series will be automatically merged with the first one and then removed.
Internal step used to read data from the LSI module.
Example:
<ReadLsiConfigStep />
Reads test coverage information from the native test cases imported from the C++test 2.3/6.x (if there are any in the current test unit).
Example:
<ReadNativeCoverage />
Reads test execution log from the native test cases imported from the C++test 2.3/6.x (if there are any in the current test unit).
Example:
<ReadNativeTestsLog />
Internal step that reads the static coverage information file prepared when instrumenting project source files.
Example:
<ReadStaticCoverageStep />
Internal step used to read information about symbols used / defined in processed source files.
Attributes:
Example:
<ReadSymbolsDataStep />
Reads test execution results from the test log.
Attributes:
testLogFile -- Specifies the test log file to be read. To read the series of files obtained by using split file communication channel (see File Communication Channel Implementation), specify the path to the first file (e.g., 'cpptest_results.tlog') and make sure that all the files are placed in the same directory and that their names follow the original naming scheme ('cpptest_results.tlog', 'cpptest_results.tlog.0001', 'cpptest_results.tlog.0002', etc.). The rest of the files in the series will be automatically merged with the first one and then removed.
y Year
M Month
d Day
a Am/pm marker
H Hour (0-23)
h Hour (1-12)
m Minute
s Second
S Millisecond
Z RFC 822 time zone
The minimum number of digits can be set by repeating a pattern letter. Shorter numbers are zero padded. If the year pattern has two letters (yy), a two digit year is used. If the month pattern has three or more letters (MMM), text will be used; otherwise, numbers will be used. This attribute is optional.
Example:
<ReadTestLogStep
testLogFile="${cpptest:testware_loc}/cpptest_results.tlog"
timeoutInfoProperty="test_exec_timeouted"
logTime="'Date': yy/MM/dd, 'Time': HH:mm:ss.SSS"
/> |
Removes given file/files from the file system.
Attributes:
Example:
<RemoveFileStep file="${cpptest:testware_loc}/*.clog" />
Executes the native test cases imported from the C++test 2.3/6.x (if there are any in the current test unit).
Example:
<RunNativeTests />
Internal step used to pass all read static coverage information to the coverage results system.
Example:
<SendStaticCoverageStep/>
Sets the execution flow property. You can use regular expressions with the 'search' and 'replace' attributes to search for and replace values . This is especially useful when the test value contains one of the C++test execution flow variables described later in this section.
Attributes:
Example 1:
<SetProperty key="stub_config_file" value="${cpptest:testware_loc}/stubconfig.xml" />
stores the plain value into "${cpptestproperty:stub_config_file}"
Example 2:
<SetProperty key="twl_mixed_path" value="${cpptest:testware_loc}" search="\\" replace="/" />
replaces all backslashes with forwardslashes for the provided value and stores the outcome into "${cpptestproperty:twl_mixed_path}":
Unconditionally stops the execution flow.
Internal step used to prepare an xml file with the list of the test cases to be executed. The xml file is later used by the TestRunnerGenerationStep.
Attributes:
Example:
<TestCaseFindingStep
testSuiteConfigFile="${cpptest:testware_loc}/testsuites.xml" allowNoTestCasesRun="false"
/>
Prepares test runner source code that will work as a driver of the test case execution. Depending on the specified attributes, the test executable will produce log files or will send results via socket connection.
Attributes:
Internal step that instruments the stub files.
Example:
<UserStubsInstrumentationStep />
Since the execution flow recipe is an xml document, all values used as flow step attributes need to be valid xml strings. This means that all occurrences of special characters (such as " or < ) need to be substituted with appropriate escape sequences (for example, " or < respectively).
Variables that can be used in the flow step attributes are:
This step pauses the execution flow while waiting on a specific file to either be created or for the file to be inactive for a specified amount of time. Behavior depends on the attributes used described below.
Attributes:
Examples:
The following setting will pause the execution flow for 10 seconds or until the sync_file.txt file appears at the specified location:
<FileSynchronizeStep
fileSync="${cpptest:testware_loc}/sync_file.txt"
timeout="10000"
/> |
The following setting will track the timestamp of the cpptest_results.tlog file. When C++test detects that the file is inactive for 10 seconds, then the execution flow will resume:
<FileSynchronizeStep
fileSync="${cpptest:testware_loc}/cpptest_results.tlog"
fileInactiveTimeout="10000"
/> |