Testing Patches

Writing tests for the code is a good practice in any programming language. It allows to write and debug code quickly with covering many possible cases ranging from the most common to edge cases. Moreover, code covered by tests can be modified without worrying to break something in other places where it is used.

Also, tests serve as excellent documentation of what is expected at outputs with specific parameters at inputs.

In XOD node testing implemented as tabular tests (tabtests for short). A test suite looks like a table. The first row contains pin labels of a tested node. All the next rows are test cases: a row is one test case and one transaction of the XOD program. In cells, you put input values and their expected outputs for the given pins.

Here’s how a typical tabtest looks like:

When you run tests, they either pass or fail with an error describing the difference between actual and expected outputs.

Primitive types #

Nodes with such primitive types as strings, numbers, and booleans are simplest to test. Let’s imagine that there is no to-percent node in the XOD standard library, and we implement it by ourselves.

It’s a node that converts some number into a string with the percent character concatenated. However, what’s in the output if it takes a NaN on the input, or -Inf?

First of all, place a new marker node: xod/patch-nodes/tabtest. Now double-click the tabtest node. An embedded tabtest editor opens up which looks like a table. Let’s write test cases:

All right, we just wrote the tests. It’s time to run them.

Press the “Run tests” button in the panel above and wait for a little. It can take a few seconds because these tests are transpiling to the C++ code, then compile in the cloud, and then execute. You can watch for the progress in the Deployment pane below: you’ll see a progress bar. Expand the pane to see more details.

Tests failed! Let’s inspect details in the “Tests” tab in deployment pane. Here are the lines that are important to us:

    ./to-percent.catch.inl:21: FAILED:
      REQUIRE( probe_PERC.state.lastValue == xod::XStringCString("255%") )
    with expansion:
      "2550%" == "255%"

It tells us that we get value "2550%" in the output instead of expected "255%". Oh, it looks like we make a mistake in the test case. Let’s fix it and rerun tests.

Now you see a green notification message about passed tests. Congratulations!

State keeping #

We just tested a pure node. That is a node which depends only on its input values when producing output. But how to deal with impure nodes, like count, which also depend on the internal state and past?

All right, we did not tighten the nuts so that we can test these nodes too, but we have to keep it in mind when we are writing tests for them. Each subsequence test case does not reset the state of the previous test case, so if you send a pulse to the INC pin of the count node twice, it outputs 2 in the second case. If you want to start over — add a new test case and send pulse to the RST pin.

Time flow #

Some impure nodes are dependent on the time flow. For example, fade and delay. To test them we can mock the time.

Make a column with the magic name __time(ms) and specify timestamps in milliseconds in this column. Such way you can travel through time.

However, also you have to know three things:

• A timestamp should be a non-negative integer.
• Nodes that use isTimedout function inside or contain nodes like xod/core/delay, xod/core/clock, are marked as timed out only at the next millisecond. For example, if we set a delay for 500 ms, it pulses DONE on the 501 ms.
• When you do not use this special column, the framework automatically increments the time value by one millisecond before running each test case. It is necessary to correctly support defer nodes. However, when you use the __time(ms) column you have to update it manually. So if you want to test something with at the same time point at each row, you probably do not need this column at all.

You can checkout tabtests in these nodes to understand how it looks and works: xod/core/delay, xod/core/clock, xod/core/fade, xod/core/system-time. For example, here a part of tabtest for xod/core/delta-time:

Variadics, generics, custom types #

To test nodes with variadics, generics or custom types do the following:

1. Create a special patch with a name describing the test and put your node(s) to cover with test inside
2. Place terminals with primitive types
3. Place xod/patch-nodes/tabtest
4. Write tests

For example, you can make

• test-select-4-number, which tests selecting one of the four numbers by pulses,
• test-add-vectors, which makes two custom types vector from primitive input-numbers, then add one to another and then unfold it back to primitives.

Let’s look at the first example in more detail.

And some test cases:

Now you know how to write and run tests. It will help you to write more stable programs.

Moreover, this is the time to tell you about test-driven development (known as TDD). It’s a development process when a developer writes tests before writing code. It brings productivity, quality, and confidence to the development process. Also, when you write test cases, some things can become evident for you, like “it should be two small nodes instead of the huge one” or “oh, it can return invalid values and how I should deal with it?”

Also, it is a lot faster to run tests than upload code to the board after each change and test all possible cases manually.

So we recommend using TDD when you’re making some nodes that are not about hardware. Detach all the logic from the hardware to test it quickly especially as XOD does not allow to test the nodes working with hardware. For this reason, for tabtest not provided literals for pins of the port type.

Let’s summarize:

1. Tests help to make a stable program.
2. Tests serve as proper documentation.
3. Tests help you improve the program without worrying to break things.
4. Test-driven development is possible in XOD IDE.
5. Development becomes faster and more stable with tests.