Data Collector is the component that accepts reports from testing and code analysis tools. You can configure Data Collector’s default storage and upload limits to meet data requirements for your organization.

In this section:

Configuring the Report Storage Threshold

When Data Collector receives and processes files from analyzers, a report file is processed and moved to the <DTP_DATA_DIR>/data/_DEFAULT_/dc/stored folder. This folder contains subfolders that are structured according to a YEAR/MONTH/DAY/XXX/YYY format where XXX and YYY are integer values. The values are calculated based on file name.

You can set a maximum size for the stored folder and DTP will automatically prune older entries. This configuration only applies to the XML reports stored in DTP, not the build details retention settings (see Configuring Build Details Retention Settings). 

1. Open the DCConfig.xml file located in the <DTP_DATA_DIR>/grs/config/ directory and locate the following entries:

   <stored-folder>
   	<max-size>20</max-size>
   	<size-to-clean>5</size-to-clean>
   </stored-folder>

2. Change the value for the <max-size> property (in GB) to an acceptable maximum storage capacity; the default is 50 GB.
3. Change the value for the <size-to-clean> property (in GB) to the amount of data you want pruned when the maximum capacity is reached; the default is 5 GB.
   
4. Save the file.

If these entries do not appear in the DCConfig.xml file, then the defaults will be used.

Configuring the Maximum Size of Report Uploads

By default, the maximum size of the XML report file that Data Collector can accept is 512M. You can change the limit by adding the following Data Collector JVM argument:

-Dcom.parasoft.sdm.api.rawstorage.datacollector.uploadMaxSize=512

For Linux installations, you can modify the following string located in the <DTP_INSTALL>/bin/variables file:

JAVA_DC_CONFIG_ARGS=" -Dcom.parasoft.sdm.api.rawstorage.datacollector.uploadMaxSize=512"

Configuring Build Details Retention Settings

By default, Data Collector stores 10 build's worth of metrics data and two build's worth of test results and test coverage data in the database (see Default Data Retention Settings). The data related to these practices is referred to as "build details" and are used to populate their respective explorer views. When a new report is sent to DTP, the details for the oldest relevant build are deleted if the limit is exceeded. Each build details type, however, is stored independently so that exceeding a build details type's limit does not affect other build details for the same build. Static analysis details are always stored in the database and are not configurable.

In the following example, two build's worth of test data have been stored and are available from the Test Trend widget drill-down report:

You can configure the number of build's worth of data retained for each practice (see Configuration). Raise the limit if you want Data Collector to retain additional build's worth of details when new reports are sent to DTP. Build details can consume a significant amount of disk space, so if disk space is an issue, consider reducing the limit. At least two build's worth of details is required for DTP to display meaningful information. 

About Resource Coverage Details

If a filter is configured to show data according to resource groups, then resource coverage details are also required in order to view data for a single build, as well as to view the build in coverage trend widgets. A resource group is a collection of files and/or folders defined by a set of one or more ANT file patterns. They enable you to view software quality information for specific parts of the code (see Adding Resource Groups to Projects).

If your filter does not use resource groups, then you can ignore the settings for resource coverage details.

Configuration Hierarchy

1. When Data Collector receives new results from a Parasoft tool, it first checks the build details storage settings for the project (see Configuring Build Details Settings). 
2. If the project build details settings have not been configured, Data Collector uses the global build details settings (see Configuration).

3. If neither the project, nor the Data Collector's build details settings have been configured, the limit for test and coverage details will be defined by the value set with the following Java argument:

   -Dcom.parasoft.sdm.dc.build.details.to.keep

   The argument is specified in the variables file located in the <DTP_INSTALL>/bin directory. There will be no limit for the metrics or resource coverage build details. This option is deprecated and may be removed in future versions.

Default Data Retention Settings

The amount of data stored is based on the number of historical builds, as opposed to a hard limit, such as gigabytes. By default, DTP keeps the following data:

• Unit test data: two (2) historical builds
• Coverage data: two (2) historical builds
• Resource coverage data: ten (10) historical builds
• Metrics data: ten (10) historical builds.

Configuration 

To configure the data retention settings:

1. Stop the Data Collector service. See Stopping DTP Services.

2. Open the <DTP_DATA_DIR>/grs/config/DCConfig.xml configuration file and locate the <details-retention-builds-count> group of settings:

   <!-- <details-retention-builds-count>
           <tests>2</tests>
           <coverage>2</coverage>
   		<resource-coverage>10</resource-coverage>
           <metrics>8</metrics>
    </details-retention-builds-count>-->

3. Uncomment the settings and specify how much data should be stored for each practice. The values refer to number of builds. Specifying 1, for example, means storing one build worth of data for the practice. 

   <details-retention-builds-count>
   	<tests>3</tests>
       <coverage>4</coverage>
   	<resource-coverage>12</resource-coverage>
       <metrics>9</metrics>
   </details-retention-builds-count>

4. Save the file and restart Data Collector and DTP Server.

Configuring Test Failures Threshold

By default, Data Collector will reject reports that contain more than 5000 reported test failures. You can change the limit by adding the following Data Collector JVM argument:

-Dcom.parasoft.sdm.rawstorage.failures.limit=5000

For Linux installations, you can modify the following string located in the <DTP_INSTALL>/bin/variables file:

JAVA_DC_CONFIG_ARGS=" -Dcom.parasoft.sdm.rawstorage.failures.limit=5000"

If a negative value is provided, Data Collector will not reject reports due to number of test failures.

The variables file is overwritten during an upgrade, so you will need to reapply the configuration setting after upgrading DTP.

Configuring Report Scanning Intervals

Reports sent to DTP are queued in a folder in the DTP_DATA directory before being saved to the database. By default, Data Collector checks the queue folder every 60 seconds for new report files. You can change the idle time between scans so that Data Collector checks the queue folder in shorter or longer intervals. Add the following Data Collector JVM argument to the <DTP_INSTALL>/bin/variables file to configure the scanning interval:

-Dcom.parasoft.sdm.dc.scanning.period=<seconds>

The variables file is overwritten during an upgrade, so you will need to reapply the configuration setting after upgrading DTP.

Configuring Max Traffic Response from SOAtest

Reports sent to DTP from SOAtest may include request response messages. By default, Data Collector will accept message responses that are 1 million bytes, but you can specify a limit by setting the following Data Collector JVM argument:

-Dcom.parasoft.sdm.dc.traffic.max.length=<BYTES>

Disabling Security Layer Protocols and Cipher Suites

SSL is enabled by default in DTP, but you can disable SSL protocols and/or cipher suites if you experience security-related issues associated with your Parasoft installation:

1. Stop the Data Collector service. See Stopping DTP Services.

2. Open the the <DTP_DATA_DIR>/conf/DCServerConfig.xml configuration file and specify the protocols and/or cipher suites to exclude in the appropriate elements:
   

   <!-- comma separated list of protocols to exclude
    By default, TLSv1, TLSv1.1 are excluded -->
   <excluded-ssl-protocols>TLSv1,TLSv1.1</excluded-ssl-protocols>
   <!-- comma separated list of cipher suites to exclude -->
   <excluded-cipher-suites></excluded-cipher-suites>

3. Restart Data Collector.

Reverse Proxy Configuration

Reverse proxies are sometimes used to ensure high availability or strengthen network security. If your organization uses a reverse proxy, you can configure Data Collector to work in your reverse proxy environment. In addition to configuring the Data Collector configuration file, you must also configure the reverse proxy server to forward requests to the correct ports. See Reverse Proxy Support for additional details.

1. Open the DCServerConfig.xml configuration file located in the <DTP_DATA_DIR>/conf directory.
2. Specify values for the following elements:
   • <dc-reverse-proxy-protocol> : The protocol (HTTP or HTTPS) on the reverse proxy server that users connect to.
   • <dc-reverse-proxy-port>: The port on the reverse proxy server that users connect to.
   • <dc-reverse-proxy-host>: The host name of the reverse proxy server that users connect to.
   • <dc-reverse-proxy-path>: (Optional) Specify a path that the reverse proxy server can use to upload data to DTP through Data Collector. This option enables you to provide access to Data Collector, Report Center, and other DTP applications if they have been configured to use a single context path. You will only need to configure the path if your reverse proxy is configured to support single context paths. The path must begin with /. 
3. Save the file and restart Data Collector (see Starting DTP Applications).

Example Configuration

In the following example, Parasoft reports are sent to https://reverse-proxy123.example.com:7777/dtp/dc, which forward to http://your-dtp-server.example.com:8082:

<root-config>
<!-- fields that always need to be are not part of the reverse proxy setup -->
<dc-server-protocol>http</dc-server-protocol>
<dc-server-port>8082</dc-server-port>
<!-- reverse proxy setup -->
<dc-reverse-proxy-protocol>https</dc-reverse-proxy-protocol>
<dc-reverse-proxy-port>7777</dc-reverse-proxy-port>
<dc-reverse-proxy-host>reverse-proxy123.example.com</dc-reverse-proxy-host>
<dc-reverse-proxy-path>/dtp/dc</dc-reverse-proxy-path>
... other fields ...
</root-config>

If these fields are not set, then a Parasoft tool (Jtest, etc.) will fail when uploading a report to Data Collector. 

You should also configure your reverse proxy to handle large request payloads. This is because Parasoft tools can generate reports that are megabytes in size. Refer to the documentation for your reverse proxy server for details.