Adding SNMP Agent Monitors

Load Test can retrieve data based on the provided OID(s) from the SNMP agents of versions 1 and 2c.

To add an SNMP agent monitor:

1. Right-click the Monitors node in the Load Test tab and choose New Monitors. A New Monitors dialog opens.
2. Select SNMP from the New Monitors dialog, then click Finish. A configuration panel for the new monitor will open in the right panel.
3. Type the SNMP agent’s host name in the Host field.
4. Type the SNMP agent’s community in the Community field.
5. Enter a parameter by clicking New. The New Parameter dialog opens.
   
6. Complete the New Parameter dialog as follows:
   • Parameter Type: Choose OID or Composite.
   • Parameter: Depending on the Parameter Type selected, the Parameter options will vary according to the following:
     • OID: Enter the Object ID that identifies a specific parameter within the SNMP agent community. (Only available if OID is selected as the Parameter Type.)
     • Composite: Choose Processor, Memory, Network, or TCP to collect corresponding data.
     • Counters: Depending on the selected Composite parameter, displays CPU Utilization (%), Memory Utilization (%), and Network Utilization (%); these counters are calculated by querying a set of OIDs.
       If TCP is selected as the Composite parameter, you must select a counter from the Counters box.
     • Parameter Explanation: Displays an explanation for the Composite parameter selected.
   • Graph Title: Enter a title for the graph that will correspond to the Monitor you are adding. This title will display in Load Test Progress graphs, and in Detailed Report graphs.
   • Apply operation: Select to apply simple operations to values returned by an SNMP agent. Use "*" for "multiply", "/" for "divide", and "%" for "percentage of". For example, if the agent returns the value of the allocated memory in bytes but you want to see it in KB, you would apply a "/ 1024" operation by selecting "/" from the menu and entering "1024" in the adjacent text field. You could then add "Memory in kilobytes" into the Graph Title field.
7. Repeat the above step until the Parameters table shows all parameters you want this SNMP agent to monitor.
8. Verify the new monitor by right-clicking the related node in the Load Tests tree’s Monitors > SNMP branch.

Adding Windows Performance Monitors

To add a Windows performance monitor (functionality which is only available if you have a Windows installation of Load Test):

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select Windows from the New Monitors dialog, then click Finish.
3. Type the perfmon’s host name in the Host field.
4. Enter a parameter by clicking New. The Add performance counter dialog opens.
   
5. Complete the New Parameter dialog as follows:
   • Select the performance counter and instance of the parameter you want to monitor. An explanation of the parameter you select displays in the Parameter Explanation box.
   • In the Graph Title field, enter a title for the graph that will correspond to the Monitor you are adding. This title will display in Load Test Progress graphs, and in Detailed Report graphs.
   • In the Apply Operation sub-panel, select an operation and a value to be applied to the monitor data if needed. For example, by selecting the / (divide) operation and entering 1024 in the operation value field, you could convert the original monitor value from Bytes to KBytes.
6. Repeat the above step until the Parameters table shows all parameters you want this perfmon to monitor.
7. Verify the new monitor by right-clicking the related node in the Load Tests tree’s Monitors > Windows branch.

Adding WebSphere Monitors

Before adding a WebSphere monitor, go to the Security > Global security section of the WebSphere Administrative console and ensure that Enable administrative security is not enabled.

To add a WebSphere monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select WebSphere from the New Monitors dialog, then click Finish.
3. In the Project Configuration panel, click Configure. A Configure WebSphere Agent dialog opens.
   
4. Enter the Host name, change the Port number if necessary, then click Next.
5. In the Configure WebSphere Agent panel, enter or browse to the appropriate Path of the WebSphere installation. For example: C:\IBM\WebSphere.
6. Click Next. The Configure WebSphere Agent dialog displays a table of the required Jar files for the server and indicates whether or not the appropriate Jar files were found.
   
7. In the Configure WebSphere Agent dialog, click Finish. The Add Parameter dialog opens.
   
8. Select a Parameter from the left GUI panel of the Add Parameter dialog. The Name, Description, and Value for each instance of the parameter you want to monitor displays in the right GUI panel of the Add Parameter dialog.
9. In Select column from the right GUI panel, check the parameter instances that you want to monitor, then click OK. The parameters you choose will display in Load Test Progress graphs, and in Detailed Report graphs.

Adding WildFly Monitors

Load Test WildFly monitor supports WildFly application server version 27 and higher.

To add a WildFly monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select WildFly from the New Monitors dialog, then click Finish.
3. In the Project Configuration panel, click Configure. A Configure WildFly Agent wizard dialog opens.
   
   
4. Enter the Host and Port for your WildFly server.
5. If your WildFly server is configured to accept anonymous connections, you can leave the User and Password fields blank. If a login is required, enter the credentials in the User and Password fields.
6. Click Next.
   
   
7. Enter the path to your WildFly application server installation directory. Click Next.
   

   In order to connect to the WildFly application server, the Load Test WildFly monitor needs to load the jboss-cli-client.jar file from the WildFly installation directory. If your Load Test and WildFly installations are on different hosts, copy the jboss-cli-client.jar from the WildFly server installation to your Load Test machine and enter that directory. The wizard will look for the jboss-cli-client.jar directly under the path you specified (in case you copied the jboss-cli-client.jar from a remote installation) or under the bin\client directory if the path you specified is the actual WildFly installation.

   
8. Verify the required jar archives were found and click Finish. A progress dialog opens followed by the Add Parameter dialog.
   
   
9. Choose a parameter from the left panel. The Name, Description, and Value for each instance of the parameter you want to monitor will appear in the right panel.
10. In the right panel, enable the parameter instances that you want to monitor, then click OK.
11. The parameters you selected will be added to the Parameters table of the WildFly monitor configuration view. Double-click on a parameter to modify its Title. The Title is the name under which the parameter graph will appear in the Load Test progress view and the report. You can apply an operation to each parameter, if needed. For instance, you can multiply the SystemCpuLoad parameter by 100 to convert this value to percent.
    
    
12. Click the Properties icon in the upper-right corner of the monitor panel to see monitor properties. The User and Password values are editable (press ENTER after editing to apply your changes).
    
    

Adding Tomcat Monitors

set "JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7744 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

To add a Tomcat monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select Tomcat from the New Monitors dialog, then click Finish.
3. In the Project Configuration panel, click Configure. A Configure Tomcat Agent dialog opens.
   
4. In the Configure Tomcat Agent dialog, enter the host name and change the port number, if necessary, then click Finish. The Add Parameter dialog opens.
   
5. Select a Parameter from the left GUI panel of the Add Parameter dialog. The Name, Description, and Value for each instance of the parameter you want to monitor displays in the right GUI panel of the Add Parameter dialog.
6. In Select column from the right GUI panel, check the parameter instances that you want to monitor, then click OK. The parameters you choose will display in Load Test Progress graphs, and in Detailed Report graphs.

Adding rstat Monitors

To add an rstat monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select rstat from the New Monitors dialog, then click Finish.

Adding Remote JVM Monitors

To add a remote JVM Monitor:

1. Right-click the Load Tests tab and choose Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select Remote JVM from the New Monitors dialog, then click Finish.
3. In the Project Configuration panel, click Configure. A Configure Remote JVM Agent dialog opens.
   
4. In the Configure Remote JVM Agent dialog, enter host name and change port number, if necessary, then click the Finish. The Add Parameter dialog opens.
   
5. Select a Parameter from the left GUI panel of the Add Parameter dialog. The Name, Description, and Value for each instance of the parameter you want to monitor displays in the right GUI panel of the Add Parameter dialog.
6. In Select column from the right GUI panel, check the parameter instances that you want to monitor, then click OK. The parameters you choose will display in Load Test Progress graphs and in Detailed Report graphs.

Adding Custom Monitors

Custom monitors can be implemented by JavaScript or Jython scripts, or by Java class methods. This is especially useful if you would like to collect data from sources that are not listed above.

To add a custom monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select Custom from the New Monitors dialog, then click Finish.
3. Enter a name for the Custom monitor in the Name field.
4. Click New. An Add Channel dialog opens.
   
5. Enter a name for the channel in the Channel ID field. The name you entered for the Custom Monitor, together with the Channel ID, will comprise the title of the monitor. This title will display in Load Test Progress graphs, and in Detailed Report graphs.
6. (Optional) Enter a short description of the channel in the Description field.
7. Select a language from the Language menu.
8. Specify the script by doing one of the following:
   • Enable File and click Browse to select the appropriate file.
   • Enable Text and enter the script in the text field.
9. Specify the method you want invoked from the Method menu.
10. Click OK. The new custom channel is added to the Channels table in the right GUI panel.

AppDynamics

Adding AppDynamics Monitors

To add an AppDynamics monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select AppDynamicsMonitor from the New Monitors dialog’s Deployed monitor types table, then click Finish.
   
3. In the upper-right corner of the Project Configuration panel, click Properties and configure the AppDynamics properties.
   
   
   • Host – host where the AppDynamics server is installed.
   • Port – port where AppDynamics is listening for REST calls.
   • User Name – username for REST calls authentication.
   • Password – password for REST calls authentication.
   • Query Interval (Minutes) – time in minutes over which the monitor values are averaged. The minimum value is 1 minute.
4. Click OK to save the property values in the project.
5. To add channels to the AppDynamics monitor:
   1. Click New in the Deployed monitor panel of the Project Configuration tab. The monitor channel selection dialog displays. The top-level nodes in monitor channel tree are applications monitored by AppDynamics.
   2. Expand the top-level application nodes until you reach the channel leaf nodes with check boxes.
   3. Ensure that the items you want to monitor are checked.
   4. Click OK to have selected nodes added to the Parameters table of the Deployed monitor configuration panel. The parameters you choose will display in Load Test Progress graphs as well as in Detailed Report graphs.
      

Monitoring Data Available from AppDynamics Monitoring Extensions

The AppDynamics Exchange site offers a number of Monitoring Extensions. Monitoring Extension data available in AppDynamics Metric Browser will be available in the Load Test AppDynamics monitor.

The screenshots below show the data exposed by the ‘Static’ Monitoring Extension available from the Exchange in the AppDynamics Metric Browser View and the Load Test AppDynamics monitor channel selection view.

Custom FileContentMetric in AppDynamics:


Custom FileContentMetric in Load Test AppDynamics Monitor:

Dynatrace

Adding Dynatrace Monitors

To add a Dynatrace monitor:

1. Right-click the Load Test tab’s Monitors node, then choose New Monitors. A New Monitors dialog opens.
2. Select DynatraceMonitor from the New Monitors dialog’s Deployed monitor types table, then click Finish.
3. In the upper-right corner of the Project Configuration panel, click Properties and configure the Dynatrace properties.
   
   
   • Host – host where the Dynatrace server is installed.
   • Port - port where Dynatrace is listening for REST calls. Default port is 8020. Refer to the Dynatrace documentation for more details.
   • User Name - username for REST calls authentication.
   • Password - password for REST calls authentication.
4. Click OK to save the property values in the project.
5. To add channels to the Dynatrace monitor:
   1. Click New in the Deployed monitor panel of the Project Configuration tab. The monitor channel selection dialog opens.
   2. Expand the top-level application nodes until you reach the channel leaf nodes with check boxes.
   3. Ensure that the items you want to monitor are checked.
   4. Click OK to have selected nodes added to the Parameters table of the Deployed monitor configuration panel. The parameters you choose will display in Load Test Progress graphs as well as in Detailed Report graphs.
      

The monitor channel selection view tree is structured in the following way:

• Level 1: Dynatrace dashboards
  • Level 2: Chart Dashlets in a dashboard
    • Level 3: Measures in a Chart Dashlet
      • Level 4: Min/Max/Avg vales of the Measure

You can monitor either existing Chart Dashlets of Line Chart type or create new ones. You can add multiple Measures (data series) to a Chart Dashlet. See the Dynatrace vendor documentation for more details.

Dynatrace REST Query Response Size Considerations

The sizes of the Dynatrace REST responses can be quite large (100KB and more), depending on the Chart and Dashboard configurations.

To minimize the size of the Dynatrace REST query responses:

1. In the charts of the dashboards that are being monitored by Load Test, choose the following settings in the Chart control panel in the top right corner of the Chart view.
   • Select Timeframe: set to last 5 minutes.
     
   • Select Chart Resolution: set to 10s.
     
2. To apply these settings to all charts in a dashboard, choose the Apply to Dashboard command in the right-click menu shown in screenshots above.
3. To estimate the sizes of responses of REST queries of dashboards that are used by the Load Test Dynatrace monitor, open the query results in a browser using the following URL template (after substituting the host, port and the dashboard names to the ones relevant to your environment): http://dynatrace.parasoft.com:8020/rest/management/dashboard/your_dashboard_name

Monitoring Data Available from Dynatrace Plugins

The Dynatrace ‘PLUGIN CENTRAL’ site and ‘Community Plugins and Extensions’ site contain a number of plugins, many of which can be used to extend the out-of-the box Dynatrace monitoring capabilities. The Dynatrace plugin data added as a Measure to a Chart Dashlet will be available in the Load Test Dynatrace monitor.

The screenshots below show a StockQuote sample monitor plugin available from the ‘Community Plugins and Extensions’ site added as Measure in a Dynatrace Chart Dashlet and in Load Test Dynatrace monitor channel selection view.

Custom StockQuote Measure in Dynatrace:

Custom StockQuote Channel in Load Test Dynatrace monitor:

JVM Threads

Adding Remote JVM Threads Monitors

The JVM Threads Monitor allows you to observe threads statistics, as well as record individual thread details of a remote Java application using Java JMX technology. 

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=7744 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

To add a Remote JVM Threads Monitor

1. Right-click the Monitors node in the Load Test tab and choose New Monitors.
2. Choose JVMThreadsMonitor from the New Monitors dialog’s Deployed monitor types table and click Finish.
3.  In the upper-right corner of the Project Configuration panel, click Properties and configure the JVMThreadsMonitor properties.
   
   
   • Host: host name where the JVM is running
   • Port: the JMX port of the JVM (See Enable JVX Monitoring).
   • Thread Filter: Comma-separated list of strings. If a thread name contains any of the strings in the list, the thread will be processed by the monitor. Thread name matching is case insensitive. (The filter is applied after the monitor receives all available thread data from a remote JVM process.)
   • Thread Dump Mode: Comma-separated list of thread states to be dumped for further analysis. See Taking Selective Thread Dumps for a detailed description.

   • Thread Dump Directory: Directory path where threads should be dumped. See Taking Selective Thread Dumps for a detailed description.

     The JVM Threads Monitor may affect the performance of your application under test (AUT)

     The performance effects associated with adding the JVM Threads Monitor are comparable to using the jvisualvm Java performance diagnostic and monitoring tool. The effect should be negligible in most cases, however we recommend evaluating it by comparing key performance indicators of the AUT, such as CPU utilization, Average and Maximum response times, and so on, before and after applying the JVM Threads Monitor to your Load Test project.

4. Click OK to save the property values in the project.
5. To add channels to the JVM Threads monitor:
   1. Click New in the Deployed monitor panel of the Project Configuration tab. The monitor channel selection dialog opens.
   2. Enable the channels you want to monitor. Channel details are available in the Description column of the channel selection dialog.

About Channel Descriptors

All data returned by monitor channels is related to the last Load Test data collection interval.

The NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED are the states from the State enumeration of the java.lang.Thread class.

For details about JVM thread states, see the Java documentation for relevant Java classes:

• java.lang.management.ThreadInfo:
  https://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadInfo.html
   
• java.lang.managementThreadJMXBean:
  https://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadMXBean.html

Upon initialization, the JMX Threads Monitor checks if thread contention monitoring is enabled on a remote JVM. If contention monitoring is disabled, the JMX Threads Monitor attempts to enable it. If this attempt fails, the BlockedTime, BlockedRatio, WaitedTime and WaitedRatio channels will return zero values.

Taking Selective Thread Dumps

The JMX Threads Monitor allows you to take selective dumps of a remote application's threads and save them to disk for further analysis. This feature helps you catch undesired thread states that appear sporadically or record thread states during the automated load test runs.

The JMX Threads Monitor will only dump threads when a load test is running. Thread dumps will not be taken when clicking Send Request in the monitor configuration view.

Set the following monitor properties to configure selective thread dumps.

Examining Thread Dumps

The path provided in the Thread Dump Directory setting will be used as a thread dump root. The JVM Threads Monitor will create a separate project thread dump directory under the thread dump root for each load test run. These project thread dump directories will have the following name pattern: PROJECTNAME_YYYY-MM-DD_hh-mm-ss.

• PROJECTNAME: the name of the LoadTest project.
• YYYY: current year
• MM: current month
• DD: current date
• hh: hour of the day in 24-hour format
• mm: minute
• ss: second

This format allows you to clearly see which Load Test project a thread dump is related to and when it was created. The PROJECTNAME_YYYY-MM-DD_hh-mm-ss directory will not be created if the monitor does not find threads to dump that match the threads dump configuration during a load test run.

The monitor will create thread dump files with the following name pattern inside the project thread dump directory: S..S_hh-mm-ss.txt

• S..S: Elapsed Time in seconds since the beginning of the load test run when the threads dump was taken.
• hh-mm-ss: System Time when the thread dump was taken.
  • hh: hour of the day in a 24-hour format
  • mm: minute
  • ss: second

Each thread dump file will contain one or more thread stack traces taken by the monitor. A thread stack trace will have the following format:

THREAD_STATE thread: THREAD_NAME
      thread_stack_entries

Each BLOCKED thread will be followed by a stack trace of the thread that is blocking it. A BLOCKED thread output will have the following format:

THREAD_STATE thread: THREAD_NAME
      thread_stack_entries
      ------------------
      Blocked by:      
      BLOCKING_THREAD_STATE thread: THREAD_NAME
            blocking_thread_stack_entries

Example:

BLOCKED thread: Clinet Rec: LTPub_0.4104694862292172
    com.acme.connections.ClientComms.isDisconnecting:561
    ...
    java.util.concurrent.ThreadPoolExecutor$Worker.run:617
    java.lang.Thread.run:745
    ------------------
    Blocked by:
    PARKED thread: acme_pool-2-UserCodeRunnable-135
        sun.misc.Unsafe.park:-2
        java.util.concurrent.locks.LockSupport.parkNanos:215
        java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos:2078
        java.util.concurrent.ThreadPoolExecutor.awaitTermination:1465
        ...

Matching Thread Dumps to Graph Points in the Load Test Report

To find a thread dump for a particular point in the Load Test report graph follow there steps:

1. Hover the mouse cursor over a point in the Load Test report graph
2. Use either the Elapsed Time or the System Time in the tooltip to find a dump file with the closest Elapsed or System time.

For example, the 3_10-37-37.txt file will be the thread dump file for the graph point highlighted in the screenshot.