The Portfolio Probe software goes through a rigorous testing regime before it is released. This has three basic parts:
- the standard test suite
- the standard test suite with random parameters
- random tests
Standard Test Suite
This currently consists of about 600 individual tests. The typical test includes at least one optimization plus matching calls to random.portfolio. The calls are made with internal checks so that problems are flagged even if the final results appear okay.
The results are checked for consistency:
- the optimization output is verified
- each random portfolio is checked to make sure it really does satisfy the constraints
These might be referred to as unit tests. However, they are really testing more than one thing. But many of the tests were created to test a single bit of functionality. Other tests are created because at one time the code exhibited a bug.
The (main) code in Portfolio Probe is dependent on pseudo-randomness. The same optimization problem will take a different path when given a different starting random seed. Hence bugs can be exhibited some times on a particular problem but not in others.
The standard test suite is run numerous times (with different random seeds) on each of the platforms. It is run continuously during development.
Standard Test Suite with Random Parameters
The optimization algorithm and the algorithm for generating random portfolios each rely on a few hundred parameters. These parameters have been optimized for the speed and quality of the optimization, and the speed of the random portfolio generation.
But changing the parameters can make some bugs appear that are very rare (or possibly absent) under the standard sets of parameters.
The standard tests are run numerous times with random values (within suitable limits) for the parameters.
Random Tests
Randomness is involved in the tests described so far, but here the randomness is in a different place. Here we create random problems. Novel combinations of constraints (plus utility and costs) are created that may exhibit bugs.
The random tests very commonly cause errors to be thrown. This is code that doesn’t get well exercised in the standard tests. These tests are very good at identifying error messages that are not clearly stated, as well as errors with essentially no message at all.
The tests can find problems in addition to those with errors, and some of these are inspiration for additional standard tests.
End to End Quality
There is more to software quality than being relatively bug-free.
The real goal is for users to get the results that they want. That means that the users need to be guarded from their own mistakes. Much of the R code in Portfolio Probe is specifically for this purpose.
