# JMS Performance Test Framework

The distributed test (aka Perf Test) framework was written for testing the performance of a JMS provider in various common scenarios. Although it was originally written for the purpose of testing Qpid, it can be used to test the performance of any JMS provider with minimal configuration changes.

This document explains how to use the framework.

### How it works

First, you need to run a message broker. This can be Qpid, ActiveMQ etc (although see ). All messages are sent using the JMS API.

Then run a Perf Test Controller, providing the details of the test in one or more JSON or Javascript files (see Test definitions).

Now run one or more Perf Test Client processes. These will be responsible for sending/receiving the messages once the test starts. For convenience, you can instead configure the Controller to start clients in-process. The clients and the controller communicate using queues on the message broker.

The test results are written to CSV files or optionally to a database (see ).

You can use the qpid-perftests-visualisation tool () to create charts from the results.

### Test definitions

Test definition files specify details about the messages to send, how many connections and sessions to use etc. There are a lot of options available - see the .js and .json files under the `perftests/etc/testdefs/` folder for examples.

Each JSON file contains a list of tests, expressed as a JSON structure. Alternatively, JavaScript can be used to generate this data structure. If a file has a .js extension it is parsed as JavaScript and the resulting object `jsonObject` used as the test specification. JavaScript is useful for reducing duplication in test specifications (e.g. by looping or by moving repeating literals into variables).

If the ControllerRunner is pointed at a directory instead of a file, each test specification file in that directory is used.

### Example usage

The `perftests/etc/` folder contains shell scripts that can be used to run the performance tests and visualise the results. It also contains sub-folders for test config and chart definitions.

### Instructions

- Extract the perftests archive
- Start your JMS broker
- cd into the `perftests/etc/` folder

- To run the Controller and clients in a single process, run the following command:

        $ java -cp ".:../lib/*:/path/to/your-jms-client-jars/*" \
          -Dqpid.dest_syntax=BURL \ # used if the test specifications use Qpid's BURL format for queues
          org.apache.qpid.disttest.ControllerRunner \
          jndi-config=perftests-jndi.properties \
          test-config=/path/to/test-config.json \
          distributed=false

  When the test is complete, the CSV files containing the results are written to the current directory.

#### Running the clients in a separate process

When using a large number of clients, you may get more representative performance results if the clients are distributed among multiple processes, potentially on multiple machines. To do this:

- Run the Controller, providing `distributed=true`.

- Run your clients (assuming you want to launch 10 logical clients in this process):

        $ cd perftests/etc
        $ java -cp ".:../lib/*:/path/to/your-jms-client-jars/*" \
          -Dqpid.dest_syntax=BURL \
          org.apache.qpid.disttest.ClientRunner \
          jndi-config=perftests-jndi.properties \
          number-of-clients=10

### Writing results to a JDBC database

For most use cases, writing results to a CSV file is acceptable. However, there are some cases where it is desired to write to a database instead. For example, if you need to keep track of how results have varied over time, writing to a database table allows this to be easily discovered by running a SQL query and/or producing a time series chart with the visualisation tool (see ).

To write results to a database:

- Add `writeToDb=true` to the `ControllerRunner` parameters
- Add `jdbcDriverClass` and `jdbcUrl` properties to your JNDI configuration file (obviously adding the JDBC driver class to your classpath too).
- Note that the framework automatically creates the results table `RESULTS` if it does not already exist.

### Caveats for non-Qpid JMS providers

If you are not using the Qpid broker, you must create one or more queues before running the test. This is necessary because you can't use Qpid's API to create queues on the broker. The queues are:

- The controller queue. You can specify the physical name of this in `perftests/etc/perftests-jndi.properties`. This queue is used by the clients to register with the Controller and to send results to it.
- The queue(s) used by your JSON test configuration (unless you have configured a vendor-specific queue creator).

You must also override the Controller's default queue creator using the system property `qpid.disttest.queue.creator.class`. Provide the class name of an implementation of org.apache.qpid.disttest.jms.QueueCreator, or org.apache.qpid.disttest.jms.NoOpQueueCreator if you are going to create and delete the queues manually.

You can also omit the `qpid.dest_syntax` system property if your JMS provider is not Qpid.

### When message production is slower than consumption

A given test configuration may cause messages to be produced faster than they are consumed. Unless you deal with this, the broker may become overwhelmed (e.g. running out of disk space or memory). The steps you can take to mitigate this are:

- Apply a small delay between each message publication, using the Producer JSON property `_interval`.
- Use your JMS provider's built-in features for throttling incoming messages. In the case of Qpid, this is done by configuring the queue to enforce flow control, like so:

          {
            "_tests":[
            {
              "_name": "My test",
              "_queues":[
                {
                  "_name": "direct://amq.direct//myTestQueue",
                  "_attributes":
                    {
                      "x-qpid-capacity": 10000000,
                      "x-qpid-flow-resume-capacity": 8000000
                    }
                }
              ],
              ...
            }
          }

### Visualising test results

The module visualisation-jfc can be used to generate charts as .png files from the results produced by running the perf tests (or in fact from any CSV file with a header row or database table). At runtime the visualisation module accepts the following input:

- The test results (either a CSV file or a database table)
- The chart definition, which is a properties file specifying settings such as the query to run against the data, and the type of chart to generate.

  The quickest way to create a new chart definition is to base it on an existing example in `perftests/etc/chartdefs/`.

#### Chart types

Currently, the available chart types are:

- LINE
- LINE3D
- BAR
- BAR3D
- XYLINE
- TIMELINE
- STATISTICAL_BAR

#### Queries

The query in the chart definition file is an ANSI SQL query. If reading from a CSV file the SQL table name is simply the file name and the SQL column names are the CSV header row entries.

#### Running the visualisation tool

Here is an example of how to run the visualisation tool.

    $ cd perftests/etc
    $ BASE_DIR=`pwd`
    $ java -cp "${BASE_DIR}:/path/to/extracted/visualistion-module/lib/*" \
        -Djava.awt.headless=true -Dlog4j.configuration=file:log4j.properties \
        -DcsvCurrentDir=/path/to/csv-files-directory/ \            # referenced in chart definition file
        -DcsvBaselineDir=/path/to/baseline-csv-files-directory/ \  # referenced in chart definition file
        -DbaselineName="My baseline name"  \                       # referenced in chart definition file
        org.apache.qpid.disttest.charting.ChartingUtil \
        chart-defs=/path/to/chartdefs-files/ \

To fetch the results from a database table instead of a CSV file, modify the visualistion tool parameters like so:

        $ java -cp "${BASE_DIR}:/path/to/extracted/visualistion-module/lib/*" \
            ...
            org.apache.qpid.disttest.charting.ChartingUtil \
            jdbcUrl=jdbc:your-jdbc-url
            jdbcDriverClass=your.jdbc.driver.Classname