Previous Up Next

Chapter 6  Test Suites

6.1  Introduction

If you are writing software to automate the running of a test suite, then, as I discuss in this chapter, you may find some Config4* features useful.

6.2  Regression Test Suite

Over time, a software project is likely to acquire an ever-growing collection of tests to check that specific pieces of functionality work correctly. It can be useful for a project team to rerun its entire test suite each night, to check if newly added or modified code has broken existing code. In addition, when a developer modifies code in a particular subsystem of the project, it can be useful for him to be able to immediately run the subset of tests that are related to the modified subsystem, rather than wait for the nightly run of the entire test suite.

There are some programming language-specific framework libraries that simplify the task of writing and running tests, but many projects develop their own bespoke testing frameworks.

A feature common to many testing frameworks, whether bespoke or not, is that each test has a unique name, and the testing framework knows the names of all the tests. For example:

Several years ago, when I wrote a bespoke testing framework, I found it useful for the testing framework to obtain two list variables, include_tests and exclude_tests, from a configuration file. The testing framework iterated over the entire list of test names, and it executed a test only if: (1) its name matched a wildcarded pattern in the include_tests list; and (2) its name did not match any wildcarded patterns in the exclude_tests list. Pseudocode to check those conditions is provided in Figure 6.1.

Figure 6.1: Pseudocode of shouldExecuteTest()
boolean shouldExecuteTest(
    String[] includeTests,
    String[] excludeTests,
    String   testName)
{
    for (int i = 0; i < excludeTests.length; i++) {
        String pattern = excludeTests[i];
        if (Configuration.patternMatch(testName, pattern)) {
            return false;
        }
    }
    for (int i = 0; i < includeTests.length; i++) {
        String pattern = includeTests[i];
        if (Configuration.patternMatch(testName, pattern)) {
            return true;
        }
    }
    return false;
}

I found this approach to be simple and effective. For example, you can execute all tests with the following configuration: 

include_tests = ["*"];
exclude_tests = [];

Perhaps you use the name of a software component as a prefix on the names of tests related to that component. If so, then you can execute all tests except those related to the foo and bar components by using the following configuration: 

include_tests = ["*"];
exclude_tests = ["foo_*", "bar_*"];

Doing that might be useful if you know that the foo and bar components currently are unstable, but you want to test the project’s remaining components.

As a final example, you can execute the tests for just the foo and bar components by using the following configuration: 

include_tests = ["foo_*", "bar_*"];
exclude_tests = [];

6.3  Performance Test Suite

Figure 6.2 shows a (pseudocode) API of a server application that processes invoices. A client application (somehow) obtains details of invoices and then invokes submitInvoices() to send them, in batches, to the server.

Figure 6.2: API of an invoice processing server
struct InvoiceItem {
    long          productCode;
    float         quantity;
    float         price;
    String        description;
};
struct Invoice {
    String        customerName;
    String[]      billingAddress;
    String[]      shippingAddress;
    String        creditCardNumber;
    InvoiceItem[] items;
    float         totalPrice;
};
interface InvoiceProcessor {
    void submitInvoices(Invoice[] invoices);
    ... // other operations
};

Let’s assume you are part of a project team that has been asked to implement that client-server system. Before committing to the project, your manager decides to carry out performance tests of various products that will be used in the project: the database product, the middleware product, and so on.1 His goal is to determine if the performance targets for the project are feasible. In particular, he wants to determine this feasibility before he commits significant resources to implementing the project. Your manager has asked one member of his team to write a performance test for the database, and he has asked another member, you, to write a performance test for the middleware product. In particular, he wants you to find the answer to the following question: “How many invoices per second is the middleware product capable of transmitting from the client to the server?”

You will probably find it trivial to implement a server for your performance test. In particular, the implementation of the submitInvoices() operation does nothing because you are testing the performance of just the middleware product. However, writing the client for your performance test turns out to be a bit more interesting, as I now discuss.

A pseudocode outline of your client is shown in Figure 6.3. The client connects to the server. Then it initializes an array of invoices. Having done that, it starts a timer, invokes submitInvoices() one million times, stops the timer, and reports the average throughput, that is, the number of invoices sent to the server per second.

Figure 6.3: Pseudocode of a performance test
invoiceProcessorObj = ...; // connect to the server
Invoice[] invoices = ...;
numIterations = 1000 * 1000;
startTime = getCurrentTime();
for (int i = 0; i < numIterations; i++) {
    invoiceProcessorObj.submitInvoices(invoices);
}
endTime = getCurrentTime();
elapsedTime = endTime - startTime;
throughput = invoices.length * numIterations / elapsedTime;
print("Throughout is " + throughput + " invoices per second");

Common sense dictates that the throughput will depend on the size of each invoice, which, obviously, can vary. Perhaps you also suspect that the throughput will depend on the batch size, that is, the number of invoices sent in each call to submitInvoices().2 These issues mean that you will need to run the performance test multiple times, for different sizes of invoices and different batch sizes. And to be able to do that, you do not want to hard-code information about invoice size or batch size into the test client. Instead, you want that information to be configurable, which is where Config4* comes in useful.

If you are familiar with XML, then you may know that XPath is a syntax used to specify nodes in an XML document. We can borrow that concept, and apply it (albeit with a different syntax) to specify individual fields within a complex parameter that is passed to an operation. Consider the following examples: 

InvoiceProcessor.submitInvoices.invoices
InvoiceProcessor.submitInvoices.invoices[2].customerName
InvoiceProcessor.submitInvoices.invoices[2].billingAddress[0]

The first line above specifies the invoices array parameter passed to the submitInvoices() operation in the InvoiceProcessor interface. The second line specifies the customerName field of the invoices array indexed by 2. And the third line specifies the first line of that invoice’s billing address.

With that syntax in mind, now have a look at the parameter_rules configuration variable in the test_10 scope of Figure 6.4.

Figure 6.4: Parameter rules for a performance test
test_10 {
  parameter_rules = [
    # wildcarded attribute name             attribute’s value
    #-----------------------------------------------------------
      "*.invoices.length()",                "10",
      "*.invoices[*].customerName.value()", "John Smith",
      "*Address.length()",                  "5",
      "*Address[0].value()",                "29 Street name",
      "*Address[1].value()",                "Name of suburb",
      "*Address[2].value()",                "Reading",
      "*Address[3].value()",                "Berkshire RG1 2LD",
      "*Address[4].value()",                "United Kingdom",
      "*.creditCardNumber.length()",        "16",
      "*.invoices.items.length()",          "4",
      "*.description.length()",             "10",
  ];
}
test_20 {
  parameter_rules = ["*.invoices.length()", "20"]
                  + test_10.parameter_rules;
}
test_30 {
  parameter_rules = ["*.invoices.length()", "30"]
                  + test_10.parameter_rules;
}

The parameter_rules variable is arranged as a two-column table. The first column uses "*" as a wildcard character to reduce the verbosity of the syntax discussed above. The first line in the table specifies that the length of the invoices array is 10. The second line specifies that the value of all customerName fields is "John Smith". In general, the table is used to specify the length of arrays, and either the value or length of strings.

The parameter_rules table provides an intuitive and flexible way to configure the size of parameters to an operation when running a performance test. The test_20 and test_30 scopes uses the concatenation operator ("+") to reuse the value of test_10.parameter_rules but prefix it with a different length of the invoices array. If code that processes parameter_rules uses the first matching pattern found, then this prefixing provides a simple and concise way to run the performance test for different batch sizes.

The question to now ask is the following: How much effort is required to write code that can use parameter_rules to initialise the invoices parameter in the performance test?

You can find the answer to that question by looking at the pseudocode shown in Figure 6.5. For conciseness, the pseudocode assumes that the parameter_rules table has been read from the configuration file and is available in the parameterRules instance variable. The test client shown in Figure 6.3 would initialise the invoices parameter with the following statement. 

invoices = allocateInvoiceArray(
                     "InvoiceProcessor.submitInvoices.invoices");
Figure 6.5: Pseudocode to process parameter_rules
int getArrayLength(String name) {
    String nameDotLen = name + ".length()";
    for (int i = 0; i < parameterRules.length; i += 2) {
        String pattern   = parameterRules[i + 0];
        String attrValue = parameterRules[i + 1];
        if (Configuration.patternMatch(nameDotLen, pattern)) {
            return integer.parseInt(attrValue);
        }
    }
    return 0; // default value
}

String allocateString(String name) {
    String nameDotLen = name + ".length()";
    String nameDotVal = name + ".value()";
    for (int i = 0; i < parameterRules.length; i += 2) {
        String pattern   = parameterRules[i + 0];
        String attrValue = parameterRules[i + 1];
        if (Configuration.patternMatch(nameDotVal, pattern)) {
            return attrValue;
        } else if (Configuration.patternMatch(nameDotLen, pattern)) {
            int length = Integer.parseInt(attrValue);
            StringBuffer result = new StringBuffer();
            for (int j = 0; j < length; j++) {
                result.append("x");
            }
            return result.toString();
        }
    }
    return ""; // default value
}

String[] allocateStringArray(String name) {
    int length = getArrayLength(name);
    String[] result = new String[length];
    for (int i = 0; i < length; i++) {
        result[i] = allocateString(name + "[" + i + "]");
    }
    return result;
}

InvoiceItem allocateInvoiceItem(String name) {
    InvoiceItem result = new InvoiceItem();
    result.productCode = 0;
    result.quantity = 0;
    result.price = 0;
    result.description = allocateString(name + ".description");
}

InvoiceItem[] allocateInvoiceItemArray(String name) {
    int length = getArrayLength(name);
    InvoiceItem[] result = new InvoiceItem[length];
    for (int i = 0; i < length; i++) {
        result[i] = allocateInvoiceItem(name + "[" + i + "]");
    }
    return result;
}

Invoice allocateInvoice(String name) {
    Invoice result = new Invoice();
    result.customerName = allocateString(name + ".customerName");
    result.billingAddress = allocateStringArray(
                              name + ".billingAddress");
    result.shippingAddress = allocateStringArray(
                              name + ".shippingAddress");
    result.creditCardNumber = allocateString(
                              name + ".creditCardNumber");
    result.items = allocateInvoiceItemArray(name + ".items");
    result.totalPrice = 0;
}

Invoice[] allocateInvoiceArray(String name) {
    int length = getArrayLength(name);
    Invoice[] result = new Invoice[length];
    for (int i = 0; i < length; i++) {
        result[i] = allocateInvoice(name + "[" + i + "]");
    }
    return result;
}

Some readers may be discouraged by the verbose and repetitive nature of the pseudocode in Figure 6.5. Once you understand how the pseudocode works, then it becomes obvious that the verbosity will be proportional to the quantity and complexity of data-types used as parameters. However, it might be possible to use one of two techniques to eliminate the need to write such verbose, repetitive code.

First, perhaps the public API of the server is defined using a specification language (similar in spirit to CORBA IDL), and perhaps there is a code generation tool (similar in spirit to idlgen) for that specification language. If that is the case, then it should be straightforward to write a code generation tool to generate all the verbose, repetitive code.

Second, perhaps the programming language you are using to write the test client provides reflection capabilities. If so, then you could write a utility class that uses reflection to navigate over the strings, arrays, and nested structures contained within the parameter, and initialise each one.

Once you have found a viable technique for initialising parameters without having to manually write lots of repetitive code, you will discover that a simple performance test client like that shown in Figure 6.3 can be very flexible.

6.4  Summary

In this chapter, I have discussed two ways in which Config4* can be useful for implementing test suites.

First, in a regression test suite, a configuration file might contain include_tests and exclude_tests variables that specify a list of wildcarded test names. This provides a simple yet effective way to specify an arbitrary subset of tests that should be run.

Second, writing a performance test suite is conceptually simple, but often time consuming due to the need to write repetitive code to initialise parameter values. Some people hard-code parameter values into a test program, but this results in an inflexible performance test. A more flexible approach is to use a configuration file to store wildcarded metadata about the sizes and values of parameters. Unfortunately, handwritten code to retrieve such metadata and use it to initialise parameters can be verbose and error-prone. However, if you have access to a code generation tool, then you could use it to generate such code. Alternatively, if your performance test suite is written in a language that provides a reflection API, then you could use this to write a utility function that can initialise an arbitrary type of parameter from metadata in a configuration file.

1
Middleware is software that simplifies the building of client-server applications. CORBA, JMS and Web Services are examples of (competing) middleware standards.
2
CPU speed, network latency and network bandwidth are also likely to affect throughput, but I ignore those issues in the following discussion.
Previous Up Next