Writing supertests

A supertest states a requirement your code must satisfy. Its inputs describe the values it covers, its assertions state what must hold, and optional assumptions limit which inputs the requirement applies to. Pup uses mathematical reasoning to verify these requirements or find counterexamples that reveal bugs.

Declare a supertest#

The Python supertest library provides @supertest and assume. See Add supertest support for installation.

The Rust supertest library provides #[supertest] and assume. See Add supertest support for installation.

The C supertest library provides SUPERTEST and SUPERTEST_ASSUME. See Add supertest support for installation.

The C# supertest library provides [Supertest] and Assumptions.Assume. See Add supertest support for installation.

The JavaScript supertest library provides supertest and assume. See Add supertest support for installation.

The Java supertest library provides @Supertest and Assumptions.assume. See Add supertest support for installation.

The VHDL supertest library provides supertest_assume. See Add supertest support for installation.

We recommend keeping supertests in supertests/ at the project root, or alongside your regular tests. They can live in other directories too: Pup searches the directory you check, including its subdirectories.

We recommend keeping supertests in tests/ so Cargo compiles them as test targets. You can use a separate supertests/ directory or another location by registering each test target with [[test]] in Cargo.toml.

We recommend keeping supertests in your test project, alongside your regular tests. You can use any folder included in that project; this example uses supertests/.

We recommend keeping supertests in src/test/java/ so Maven compiles them with your test dependencies. You can use another directory if your build is configured to include it as a test source directory.

For example, consider the function from Welcome to Pup that normalizes text by replacing consecutive spaces with one space. This supertest checks that running it again leaves the result unchanged:

For example, consider the 8-bit saturating incrementer from Welcome to Pup. It should add one to its input, stopping at 255. This supertest checks that incrementing a value never makes it smaller:

from schematic import *

from text_tools.text import collapse_spaces


@supertest
def collapsing_spaces_again_changes_nothing(text: str):
    once = collapse_spaces(text)
    twice = collapse_spaces(once)

    assert twice == once

use schematic::supertest;
use text_tools::collapse_spaces;

#[supertest]
pub fn collapsing_spaces_again_changes_nothing(text: String) {
    let once = collapse_spaces(&text);
    let twice = collapse_spaces(&once);

    assert_eq!(twice, once);
}

#include <assert.h>
#include <schematic.h>
#include <stdlib.h>
#include <string.h>
#include "text.h"

SUPERTEST
void collapsing_spaces_again_changes_nothing(const char *text) {
    SUPERTEST_ASSUME(text != NULL);
    char *once = collapse_spaces(text);
    char *twice = collapse_spaces(once);
    int unchanged = strcmp(twice, once) == 0;

    free(once);
    free(twice);
    assert(unchanged);
}

using Schematic;
using Xunit;

namespace TextTools;

public static class CollapseSpaces
{
    [Supertest]
    public static void CollapsingSpacesAgainChangesNothing(string text)
    {
        string once = Text.CollapseSpaces(text);
        string twice = Text.CollapseSpaces(once);

        Assert.Equal(once, twice);
    }
}

import assert from 'node:assert/strict';
import { assume, supertest } from 'schematic-supertest';
import { collapseSpaces } from '../src/text.js';

export const collapsingSpacesAgainChangesNothing = supertest((text) => {
  assume(typeof text === 'string');
  const once = collapseSpaces(text);
  const twice = collapseSpaces(once);

  assert.equal(twice, once);
});

package texttools;

import static org.junit.Assert.assertEquals;
import static tech.schematic.Assumptions.assume;

import tech.schematic.Supertest;

public final class CollapseSpaces {
    private CollapseSpaces() {}

    @Supertest
    public static void collapsingSpacesAgainChangesNothing(String text) {
        assume(text != null);
        String once = Text.collapseSpaces(text);
        String twice = Text.collapseSpaces(once);

        assertEquals(once, twice);
    }
}

library ieee;
use ieee.numeric_std.all;
use work.incrementer.all;

entity increment_supertests is
  port (value : in natural range 0 to 255);
end entity;

architecture supertests of increment_supertests is
begin
  --% supertest
  increment_never_decreases : process
  begin
    assert to_integer(saturating_increment(to_unsigned(value, 8))) >= value
      report "Increment decreased the value" severity failure;
    wait;
  end process;
end architecture;

Writing a supertest is much like writing an ordinary test function: call your code and use regular assertions to state what should happen. Give it parameters for the inputs the requirement covers, and mark it with @supertest. Define the function at the top level of the file, outside any class or other function.

Writing a supertest is much like writing an ordinary test function: call your code and use regular assertions to state what should happen. Give it parameters for the inputs the requirement covers, and mark it with #[supertest]. Define the function at the top level of the file, outside any inline module, impl block, or other function.

Write a supertest as a void function at file scope: call your code and use assert to state what should happen. Give it parameters for the inputs the requirement covers, and place SUPERTEST before the function.

Write a supertest as a public static method: call your code and use regular assertions to state what should happen. Give it parameters for the inputs the requirement covers, and mark it with [Supertest], imported with using Schematic;.

Writing a supertest is much like writing an ordinary test function: call your code and use regular assertions to state what should happen. Give it parameters for the inputs the requirement covers. Wrap the function with supertest(...) and assign the result to a named top-level constant.

Write a supertest as a public static method: call your code and use regular assertions to state what should happen. Give it parameters for the inputs the requirement covers, and mark it with @Supertest, imported with import tech.schematic.Supertest;.

Write a supertest as a labeled process with ordinary VHDL assertions. Use input ports on the enclosing entity for its inputs, and place --% supertest immediately before the process. The process label is the supertest name.

Here, text represents any string, and the assertion requires that collapsing spaces a second time leaves the result unchanged. Pup checks that the assertions hold for every input allowed by the types and assumptions.

The function expects text to point to a NUL-terminated string. SUPERTEST_ASSUME(text != NULL) excludes null pointers from this requirement.

The assumption typeof text === 'string' limits the input to strings.

The assumption text != null excludes null references from this requirement.

Here, value covers every integer from 0 through 255, and the assertion requires that incrementing never produces a smaller value. Pup checks that the assertions hold for every input allowed by the types and assumptions.

These examples use xUnit assertions. You can use your project’s existing assertion library instead.

These examples use JUnit assertions. You can use your project’s existing assertion library instead.

Name each supertest after the behavior it checks. Keep related assertions together, and split unrelated behaviors into separate supertests.

Restrict inputs with assumptions#

Use assume from the supertest library when a requirement applies only to some inputs. Pup checks the assertions for every input that satisfies the assumption.

Use SUPERTEST_ASSUME from the supertest library when a requirement applies only to some inputs. Pup checks the assertions for every input that satisfies the assumption.

Use Assumptions.Assume from the supertest library when a requirement applies only to some inputs. Pup checks the assertions for every input that satisfies the assumption.

Use supertest_assume from the supertest library when a requirement applies only to some inputs. Pup checks the assertions for every input that satisfies the assumption.

If text already has no consecutive spaces, collapsing spaces should leave it unchanged. We can add a second supertest for this requirement, using an assumption to restrict it to already normalized text:

Below the saturation limit, the result should equal the input plus one. We can add a second supertest with an assumption that the input is less than 255:

from schematic import *

from text_tools.text import collapse_spaces


@supertest
def collapsing_spaces_preserves_normalized_text(text: str):
    assume("  " not in text)

    result = collapse_spaces(text)
    assert result == text

use schematic::{assume, supertest};
use text_tools::collapse_spaces;

#[supertest]
pub fn collapsing_spaces_preserves_normalized_text(text: String) {
    assume(!text.contains("  "));

    let result = collapse_spaces(&text);
    assert_eq!(result, text);
}

#include <assert.h>
#include <schematic.h>
#include <stdlib.h>
#include <string.h>
#include "text.h"

SUPERTEST
void collapsing_spaces_preserves_normalized_text(const char *text) {
    SUPERTEST_ASSUME(text != NULL);
    SUPERTEST_ASSUME(strstr(text, "  ") == NULL);

    char *result = collapse_spaces(text);
    int unchanged = strcmp(result, text) == 0;

    free(result);
    assert(unchanged);
}

using Schematic;
using Xunit;

namespace TextTools;

public static class NormalizedText
{
    [Supertest]
    public static void CollapsingSpacesPreservesNormalizedText(string text)
    {
        Assumptions.Assume(!text.Contains("  "));

        string result = Text.CollapseSpaces(text);
        Assert.Equal(text, result);
    }
}

import assert from 'node:assert/strict';
import { assume, supertest } from 'schematic-supertest';
import { collapseSpaces } from '../src/text.js';

export const collapsingSpacesPreservesNormalizedText = supertest((text) => {
  assume(typeof text === 'string');
  assume(!text.includes('  '));

  const result = collapseSpaces(text);
  assert.equal(result, text);
});

package texttools;

import static org.junit.Assert.assertEquals;
import static tech.schematic.Assumptions.assume;

import tech.schematic.Supertest;

public final class NormalizedText {
    private NormalizedText() {}

    @Supertest
    public static void collapsingSpacesPreservesNormalizedText(String text) {
        assume(text != null);
        assume(!text.contains("  "));

        String result = Text.collapseSpaces(text);
        assertEquals(text, result);
    }
}

library ieee;
use ieee.numeric_std.all;
use work.incrementer.all;

entity increment_below_limit_supertests is
  port (value : in natural range 0 to 255);
end entity;

use work.schematic.all;

architecture supertests of increment_below_limit_supertests is
begin
  --% supertest
  increment_below_limit_adds_one : process
  begin
    supertest_assume(value < 255);
    assert to_integer(saturating_increment(to_unsigned(value, 8))) = value + 1
      report "Increment did not add one" severity failure;
    wait;
  end process;
end architecture;

Don’t exclude a failing input just to make a supertest pass. If your code should reject that input, check the rejection with an assertion.

See pup check to check your supertests and review the results.