General Settings

The general settings are the minimum settings you must specify in order to run a test.

Execution configuration: General tab

Data storage

Data storage setting

Directory

The data storage is the place where the data concerning the test execution is gathered. By default, the directory is set to:

${tpt.tptfile.dir}\testdata\${tpt.execconfigname.filename}

Thus a folder named testdata is created next to your TPT project file. And inside the testdata folder, a folder with the name of your execution configuration is created. You can either use ${tpt.execconfigname.filename} or ${tpt.execconfigname}. Both variables are replaced by the name of the execution configuration. When using one of these variables in path names, the execution configuration name must be unique, otherwise the content will be overwritten. The difference between both variables is explained in Predefined execution configuration variables.

TPT can add up to two new sub-folders to the data directory. Thus, you can add a path like mypath\new_folder_A\new_folder_B where new_folder_A and new_folder_B do not exist yet in your data directory. The folders are created when you execute the tests.

Variables from the ${tpt.matlab.modelfile.original} cannot be accessed in this field. A single execution configuration might represent multiple platforms, so this could lead to a conflict.

In case there are files in the data directory specified in Directory, this content will be overwritten if the automatically by TPT generated file names are identical. Use (trash bin) to delete the content of the data directory before the test execution. The trash bin icon is dimmed when the directory is empty.

Example

You run test case TC_A and TC_B on Monday, and run TC_A and TC_C on Tuesday without using Delete the content of the data directory, the data of TC_A is overwritten, while the data of TC_B is kept and the data of TC_C is added.

When you run TC_A and TC_C on Tuesday and use Delete the content of the data directory, the folder is emptied and finally only the data of TC_A and TC_C are inside the data directory.
The data directory contains a TXT file whose name is identical with the name of the executed test case. No characters are escaped to form the name.

Click the Advanced link to go to the Test Execution Configuration - Storage tab to specify the directory layout and select a reference directory.

Report

Report settings

In the Report sectionof the Execution Configuration dialog, you can select the format and the report directory.

Format

HTML

Renders report as standard webpage is generated for each test case. This report format can be displayed inside TPT in the Report view, see Views - Report.

HTML (embedded resources)

The same as the HTMLformat but the images, JavaScript and CSS used in the test report are added to HTML files. Thus, the number of created folders and files is reduced. This report format can be displayed inside TPT in the Report view, see Views - Report.

Projects using MHTML as report format are automatically changed to HTML (embedded resources).

PDF

Renders the report as PDF file that contains the report overview and an additional PDF file for each test case.

PDF (all test reports in one file)

Renders the report as PDF file that contains the report overview and the report of all test cases in a single file.

Report directory

The report is stored by default in the test data directory. You can specify a different directory. Thus a folder named report is created next to your TPT project file. And inside the report folder, a folder with the name of your execution configuration is created. When you enter no path at all, the report is generated in the test data folder. When using ${tpt.execconfigname.filename} or ${tpt.execconfigname} to specify the report directory, the execution configuration item names must be unique, see Directory.

TPT can add up to two new sub-folders to the report directory. Thus, you can add a path like mypath\new_folder_A\new_folder_B where new_folder_A and new_folder_B do not exist yet in your directory. The folders are created when you execute the tests.

Only for PDF (all test reports in one file) you can add the PDF file name to the end of the path, for example:  C:\Users\admin\Desktop\my_test.pdf. Nevertheless, if you specify a file name for the other types of report, a warning indicates that the file name must be removed, in order to leave just the path to the report directory.

Use (trash bin) to delete the content of the report directory. The trash bin icon is dimmed when the directory is empty.

Click the Advanced link to specify if and how the following items should be displayed in the test report: Overview report section, Assessment variables report section, Status Summary report section, Requirements report section, see Test Execution Configuration - Report Settings.

Execution

In the Execution section, select a platform configuration, a test set, a parameter set (optional), and set environment variables (optional).

The combination of platform configuration, test set, parameter set, environment variable is called an execution item.

For each execution configuration, you can specify if the assesslets applied to the test cases should be executed or not. You can also open the Platform Configuration dialog from within the Execution Configuration dialog by clicking Open platform configuration to set up new platforms or edit existing platforms.

To create and edit test sets, click Open test set definition, see Test Sets. At least one execution item needs to be specified to execute tests. Setting up an execution item can be done in two ways using the normal mode or using the advanced mode. The default setting is the normal mode. To switch to the advanced mode, click the button next to Normal mode.

Once the configuration mode is changed to advanced mode, you cannot switch back to the normal mode.

Normal mode

Normal mode

Cancel when <number> tests failed 

Define the number of failed tests that cause the execution to cancel all subsequent pending test cases automatically. Select a value in the range 1-100000.

Platform

The platform upon which the tests are executed, for example dSPACE XiL Platform, MATLAB/Simulink Platform, or ASCET Platform. Click Open platform configuration to edit the platform.

Test Set

Select Selected test cases to execute test cases selected in the Project view.

You can start the test execution from the Execution Configuration dialog or from the Project view, see Start Test Execution.

Select a test set specified in the Test Set Definition dialog, see Test Sets. Click Open test set definition to open the Test Set Definition dialog and to create additional test sets.

It is possible to execute tests with empty test sets. After test execution, the execution configurations with empty test sets are marked in the Build Progress dialog.

Parameter Set

Optionally select a parameter set for the execution. This can be a TPTAIF, DCM, or CDF file. To use a CDF file, the plug-in cdfsupport must be loaded. For more information about importing DCM files, see Import DCM.

Variables

Optionally configure special execution configuration variables for the test execution item. These variables can be used in the Script assesslet and in a Test Set Definition condition, see Test Sets using Conditional Expressions. Click Edit the variables available for execution to open the dialog to configure the variables. The execution configuration item variables are displayed in the report in the table Platform Information. More about environment variables, see Environment Variables.

Back-to-Back

Select a source that contains reference signals to be used for back-to-back tests where the results of two different platforms are to be compared.

<None>

No back-to-back tests are performed.

Reference directory

Select this option to execute back-to-back comparisons against previously executed test results that have been stored in a Reference directory, see Test Execution Configuration - Storage.

Reference file

Select this option to execute back-to-back comparisons against signal data stored in a reference file. TPT supports the CSV, TPTBIN, XLSX, XLS, MAT, MF4, MDF, DAT file format.

<reference platform>

Select this option to execute back-to-back comparisons against execution results from another execution items of this execution configuration or another platforms.

Click Set up comparison behavior to open the Back-to-Back Settings section to set up the back-to-back test, see Back-to-Back Settings.

Advanced mode

In the advanced mode, you can set up several configuration items for a single execution configuration and individually activate or deactivate those items. You can also deactivate or activate the assessment of an individual execution configuration.

Advanced mode

Toolbar icons of the advanced mode
creates another execution configuration item
deletes the selected execution configuration item
moves the selected item up
moves the selected item down
activates an execution configuration item
deactivatea an execution configuration item
opens the Configure Execution Item dialog for the selected execution configuration item; you can do the same settings here as in the table, but the back-to-back specification can only be done in the table.

Cancel when <number> tests failed 

See normal mode.

Platform configuration

Same as Platform in the normal mode.

Test set

See normal mode.

Variables

See normal mode.

Assess

You can deactivate the assessment for a single execution configuration item to run the assessment only on specific parts of the test execution. Since the assessment is a very time consuming part of the test execution this may save a lot of time if the assessment is not needed for all parts of the test execution. Deselect this checkbox to deactivate the assessment for a single execution configuration item. If deselected, no test case reports are created and no information for this execution configuration will be shown in the overview report.

Instead a paragraph is added to the overview report, for example: 'Default Configuration - Stand-alone Platform' was executed with 3 testcases, but not assessed. They will not be reported.

The assessment is only executed if the assess checkbox is selected in the lower part of the Execution Configuration dialog. If this checkbox is not selected, the assessment is not run, even if Assess is selected for the execution configuration item.

The "Assess" checkbox must be selected

Back-to-Back

See normal mode.

Deactivate and active execution configuration items

When you have specified more than one platform configuration item, all of them are executed from top to bottom the moment you click Run. To prevent platform configurations items from being used, select the platform configuration items you do not want to use, and click Deactivate execution item.

Deactivated execution configuration items are dimmed. To activate a deactivated platform configuration item, click Activate execution item.

Example

In this example we have two platforms (MiL_Platform and SiL_Platform). The goal is to compare the result of the MiL_Platform with the SiL_Platform. This comparison is specified in the execution configuration names Compare_MiL_with_SiL. This is shown once in the normal mode and once in the advanced mode.

Normal mode

In the Execution section, select the MiL_Platform from the Platform drop-down list, specify a test set, choose a parameter set, set variables, and select SiL_Platform from the Back-to-Back drop-down list.

Normal mode

Advanced mode

You need to create two execution configuration items. First, choose SIL_platform from the Platform configuration drop-down list, specify a test set, choose a parameter set, set variables, and leave the Back-to-Back drop-down list unchanged.

Next, create a second execution configuration item by clicking Add. Choose MiL_Platform from the Platform configuration drop-down list, specify a test set, choose a parameter set, set variables, and choose the execution configuration item you have previously created (Compare_MIL_with_SIL[0]--SIL_platform--..\..\..\tools\example\parameter_set_12.dcm) from the Back-to-Back drop-down list.

Advanced mode

Execution options

By selecting (multiple selection is possible) of the checkboxes Execute, Dashboard, Assess, and the Report in the Execution Configuration dialog, you can activate specific functions, see Test Execution Configuration - Execution Options.

Execution configuration checkboxes