HeyPuter/tools/api-tester/doc/cartesian.md

Cartesian Tests

A cartesian test is a test the tries every combination of possible inputs based on some model. It's called this because the set of possible states is mathematically the cartesian product of the list of sets of options.

Coverage Model

A coverage model is what defines all the variables and their possible value. The coverage model implies the set of all possible states (cartesian product).

The following is an example of a coverage model for testing the /write API method for Puter's filesystem:

module.exports = new CoverageModel({
    path: {
        format: ['path', 'uid'],
    },
    name: ['default', 'specified'],
    conditions: {
        destinationIsFile: []
    },
    overwrite: [],
});

The object will first be flattened. format inside path will become a single key: path.format, just as { a: { x: 1, y: 2 }, b: { z: 3 } } becomes { "a.x": 1, "a.y": 2, "b.z": 3 }

Then, each possible state will be generated to use in tests. For example, this is one arbitrary state:

{
    "path.format": "path",
    "name": "specified",
    "conditions.destinationIsFile": true,
    "overwrite": false
}

Wherever an empty list is specified for the list of possible values, it will be assumed to be [false, true].

Finding the Culprit

When a cartesian test fails, if you know the index of the test which failed you can determine what the state was just by looking at the coverage model.

For example, if tests are failing at indices 1 and 5 (starting from 0, of course) for the /write example above, the failures are likely related and occur when the default filename is used and the destination (path) parameter points to an existing file.

destination is file:  0  1  0  1  0  1  0  1
name is the default:  0  0  1  1  0  0  1  1
test results:         P  F  P  P  P  F  P  P

Interesting note about the anme

I didn't know what this type of test was called at first. I simply knew I wanted to try all the combinations of possible inputs, and I knew what the algorithm to do this looked like. I then asked Chat GPT the following question:

What do you call the act of choosing one item from each set in a list of sets?

which it answered with:

The act of choosing one item from each set in a list of sets is typically called the Cartesian Product.

Then after a bit of searching, it turns out neither Chat GPT nor I are the first to use this term to describe the same thing in automated testing for software.