Data Collector is the component that accepts reports from testing and code analysis tools. You can configure Data Collector’s default storage and upload limits to meet data requirements for your organization.
In this section:
When Data Collector receives and processes files from analyzers, a report file is processed and moved to the <DTP_DATA_DIR>/data/_DEFAULT_/dc/stored folder. This folder contains subfolders that are structured according to a YEAR/MONTH/DAY/XXX/YYY format where XXX and YYY are integer values. The values are calculated based on file name.
You can set a maximum size for the stored folder and DTP will automatically prune older entries. This configuration only applies to the XML reports stored in DTP, not the build details retention settings (see Configuring Build Details Retention Settings).
Open the DCConfig.xml file located in the <DTP_DATA_DIR>/grs/config/ directory and locate the following entries:
<stored-folder> <max-size>20</max-size> <size-to-clean>5</size-to-clean> </stored-folder> |
<max-size> property (in GB) to an acceptable maximum storage capacity; the default is 50 GB.<size-to-clean> property (in GB) to the amount of data you want pruned when the maximum capacity is reached; the default is 5 GB.If these entries do not appear in the DCConfig.xml file, then the defaults will be used.
By default, the maximum size of the XML report file that Data Collector can accept is 512M. You can change the limit by adding the following Data Collector JVM argument:
-Dcom.parasoft.sdm.api.rawstorage.datacollector.uploadMaxSize=512 |
For Linux installations, you can modify the following string located in the <DTP_INSTALL>/bin/variables file:
JAVA_DC_CONFIG_ARGS=" -Dcom.parasoft.sdm.api.rawstorage.datacollector.uploadMaxSize=512" |
By default, Data Collector stores 10 build's worth of metrics data and two build's worth of test results and test coverage data in the database (see Default Data Retention Settings). The data related to these practices is referred to as "build details" and are used to populate their respective explorer views. When a new report is sent to DTP, the details for the oldest relevant build are deleted if the limit is exceeded. Each build details type, however, is stored independently so that exceeding a build details type's limit does not affect other build details for the same build. Static analysis details are always stored in the database and are not configurable.
In the following example, two build's worth of test data have been stored and are available from the Test Trend widget drill-down report:
You can configure the number of build's worth of data retained for each practice (see Configuration). Raise the limit if you want Data Collector to retain additional build's worth of details when new reports are sent to DTP. Build details can consume a significant amount of disk space, so if disk space is an issue, consider reducing the limit. At least two build's worth of details is required for DTP to display meaningful information.
If a filter is configured to show data according to resource groups, then resource coverage details are also required in order to view data for a single build, as well as to view the build in coverage trend widgets. A resource group is a collection of files and/or folders defined by a set of one or more ANT file patterns. They enable you to view software quality information for specific parts of the code (see Adding Resource Groups to Projects).
DTP collects coverage at the method level and at the resource level. Method-level coverage data is required to view coverage in the Coverage Explorer. Unless explicitly stated, "coverage" in this documentation means method-level coverage. |
If your filter does not use resource groups, then you can ignore the settings for resource coverage details.
If neither the project, nor the Data Collector's build details settings have been configured, the limit for test and coverage details will be defined by the value set with the following Java argument:
-Dcom.parasoft.sdm.dc.build.details.to.keep |
The argument is specified in the variables file located in the <DTP_INSTALL>/bin directory. There will be no limit for the metrics or resource coverage build details. This option is deprecated and may be removed in future versions.
The amount of data stored is based on the number of historical builds, as opposed to a hard limit, such as gigabytes. By default, DTP keeps the following data:
To configure the data retention settings:
Open the <DTP_DATA_DIR>/grs/config/DCConfig.xml configuration file and locate the <details-retention-builds-count> group of settings:
<!-- <details-retention-builds-count>
<tests>2</tests>
<coverage>2</coverage>
<resource-coverage>10</resource-coverage>
<metrics>8</metrics>
</details-retention-builds-count>--> |
Uncomment the settings and specify how much data should be stored for each practice. The values refer to number of builds. Specifying 1, for example, means storing one build worth of data for the practice.
<details-retention-builds-count>
<tests>3</tests>
<coverage>4</coverage>
<resource-coverage>12</resource-coverage>
<metrics>9</metrics>
</details-retention-builds-count> |
By default, Data Collector will reject reports that contain more than 5000 reported test failures. You can change the limit by adding the following Data Collector JVM argument:
-Dcom.parasoft.sdm.rawstorage.failures.limit=5000 |
For Linux installations, you can modify the following string located in the <DTP_INSTALL>/bin/variables file:
JAVA_DC_CONFIG_ARGS=" -Dcom.parasoft.sdm.rawstorage.failures.limit=5000" |
If a negative value is provided, Data Collector will not reject reports due to number of test failures.
The variables file is overwritten during an upgrade, so you will need to reapply the configuration setting after upgrading DTP.
Reports sent to DTP are queued in a folder in the DTP_DATA directory before being saved to the database. By default, Data Collector checks the queue folder every 60 seconds for new report files. You can change the idle time between scans so that Data Collector checks the queue folder in shorter or longer intervals. Add the following Data Collector JVM argument to the <DTP_INSTALL>/bin/variables file to configure the scanning interval:
-Dcom.parasoft.sdm.dc.scanning.period=<seconds> |
The variables file is overwritten during an upgrade, so you will need to reapply the configuration setting after upgrading DTP.
Reports sent to DTP from SOAtest may include request response messages. By default, Data Collector will accept message responses that are 1 million bytes, but you can specify a limit by setting the following Data Collector JVM argument:
-Dcom.parasoft.sdm.dc.traffic.max.length=<BYTES> |
SSL is enabled by default in DTP, but you can disable SSL protocols and/or cipher suites if you experience security-related issues associated with your Parasoft installation:
Open the the <DTP_DATA_DIR>/conf/DCServerConfig.xml configuration file and specify the protocols and/or cipher suites to exclude in the appropriate elements:
<!-- comma separated list of protocols to exclude By default, TLSv1, TLSv1.1 are excluded --> <excluded-ssl-protocols>TLSv1,TLSv1.1</excluded-ssl-protocols> <!-- comma separated list of cipher suites to exclude --> <excluded-cipher-suites></excluded-cipher-suites> |
Restart Data Collector.
Reverse proxies are sometimes used to ensure high availability or strengthen network security. If your organization uses a reverse proxy, you can configure Data Collector to work in your reverse proxy environment. In addition to configuring the Data Collector configuration file, you must also configure the reverse proxy server to forward requests to the correct ports. See Reverse Proxy Support for additional details.
<dc-reverse-proxy-protocol> : The protocol (HTTP or HTTPS) on the reverse proxy server that users connect to.<dc-reverse-proxy-port>: The port on the reverse proxy server that users connect to.<dc-reverse-proxy-host>: The host name of the reverse proxy server that users connect to.<dc-reverse-proxy-path>: (Optional) Specify a path that the reverse proxy server can use to upload data to DTP through Data Collector. This option enables you to provide access to Data Collector, Report Center, and other DTP applications if they have been configured to use a single context path. You will only need to configure the path if your reverse proxy is configured to support single context paths. The path must begin with /. In the following example, Parasoft reports are sent to https://reverse-proxy123.example.com:7777/dtp/dc, which forward to http://your-dtp-server.example.com:8082:
<root-config> <!-- fields that always need to be are not part of the reverse proxy setup --> <dc-server-protocol>http</dc-server-protocol> <dc-server-port>8082</dc-server-port> <!-- reverse proxy setup --> <dc-reverse-proxy-protocol>https</dc-reverse-proxy-protocol> <dc-reverse-proxy-port>7777</dc-reverse-proxy-port> <dc-reverse-proxy-host>reverse-proxy123.example.com</dc-reverse-proxy-host> <dc-reverse-proxy-path>/dtp/dc</dc-reverse-proxy-path> ... other fields ... </root-config> |
If these fields are not set, then a Parasoft tool (Jtest, etc.) will fail when uploading a report to Data Collector.
You should also configure your reverse proxy to handle large request payloads. This is because Parasoft tools can generate reports that are megabytes in size. Refer to the documentation for your reverse proxy server for details.