Generating and checking

About

In the hs-test library, tests are a set of test cases. One test case is an object of the TestCase class. The generate method should generate a list of objects of the TestCase class and return it. It's a low-level mechanism of writing tests.

In the previous sections, you saw tests with dynamically generated input, but it is possible to create tests with static predetermined input. If the program to test is rather simple you can use this kind of tests to check it.

Dynamic testing has some benefits over tests with static input:

You are able to test small parts of the output compared to checking the whole output. Parsing the whole output to determine if the program written correctly may be more error-prone rather than parsing small pieces of the output.
Some programs aren't testable with static input because the input may be dependent on the previous output (for example, Tic-Tac-Toe game).

This method of testing revealed serious weaknesses and limitations for testing complex programs. Use static input only when the program to be tested is very simple. In other cases, it's strongly recommended to use dynamic testing.

General testing process

You can see the abstract testing algorithm in the code block below.

tests = generate()
for (test : tests) {
    output = run(test)
    result = check(output, test.attach)
    if (not result.correct) {
        break
    }
}

So, you can see that there are three functions here - generate, run, check. Let's consider them below.

generate

The generate function is intended for creating test cases. This function returns the list of test cases. Each test case contains enough information for this separate test. For example, what data should be submitted to the standard input of the program or what files should be created before the start of this test, with which command-line arguments the program should be launched.

Note that the content of this function will be different for each stage of each project. That's why this is the first function you should implement while writing tests for a project.

run

This function runs user code.

First, it launches the main method of the program being tested with the command-line arguments listed in the test case.

Second, it puts the text saved in the test case in the standard input before running the main method. It looks like the user is entering something into the program from the keyboard, but in fact, it is predefined in the test case.

This method also creates the necessary files before starting the test (for example, the stage description says you should read the data from the file). After the main method is executed, these files are deleted.

Also, this function makes sure that the values of all variables in the program are the same each time. For example, in Java, there are static variables that keep their value between method executions. In Python, there are variables inside the module, which also do not change their value between main file executions, because modules are imported only once during the whole period of the program execution. So, this method also changes all the variables in the program to the original state as if before running the first test.

This method returns what the program printed to the console, i.e. the standard output.

You don't need to implement this function, it is always the same for all the stages of all the projects, so it's already written.

check

This function is intended to check the correctness of the student program. It takes two parameters - the output of the program for the current test and attach to the test (this attach should also be added to the test case in the generate function).

The function returns a special object containing 2 fields - correct and feedback. We will consider them later.

As you can see from the algorithm above, the first failed test stops testing.

Note that the content of this function will be different for each stage of each project. That's why this is the second function you should implement while writing tests for a project.

TestCase class

One test case is a single run of the user program. Before creating an object of the TestCase class, it is necessary to import it.

import org.hyperskill.hstest.testcase.TestCase;

The TestCase class contains one constructor - it is an empty one. To parameterize a test case, semi-builder pattern is used - each method returns a reference to the object itself, and method calls can be joined into a call chain.

new TestCase()
    .setInput(text)
    .setAttach(object)
    .setFile(src, content)
    .setFile(src2, content2)

Input

The first and most frequently used method is setInput(string). It is intended to simulate input using the keyboard. The method takes one parameter - what exactly should be entered from the keyboard during the user program execution.

Let's look at this example:

testCase.setInput("15\n23")

In this case, the user's program below will run normally, count two numbers, and print their sum as if the user had entered these numbers using the keyboard. If you run these program directly, you will actually have to enter these two numbers from the keyboard. And you can use the setInput method to prepare this data in advance.

...
public static void main(String[] args) {
    Scanner scanner = new Scanner(System.in);
    int firstNum = scanner.nextInt();
    int secondNum = scanner.nextInt();
    System.out.println(firstNum + secondNum);
}
...

Arguments

The test case can also be parameterized by command line arguments. These arguments are passed to the program when starting the program from the console. For example:

$ java Renamer --from old --to new --all

In this case, the Java program will get an array of five arguments: [--from, old, --to, new, --all].

An example of how to set up a command-line arguments for test cases can be seen below:

testCase.setArguments("--from", "old", "--to", "new", "--all")

An example of using command line arguments in user code can be seen below:

...
public static void main(String[] args) {
    System.out.println(args.length);
    for (String arg : args) {
        System.out.println(arg);
    }
}
...

Files

You can also create external files for testing. Before starting the test case, these files are created on the file system so that the student's program can read them. After the user program finishes, these will be deleted from the file system.

An example of how to set up external files for test cases can be seen below:

testCase
    .setFile("file1.txt", "File content")
    .setFile("file2.txt", "Another content")

Below you can see examples of using these external files in the user's code:

...
public static void main(String[] args) throws Exception {
    Scanner sc = new Scanner(new File("file1.txt"));
    System.out.println(sc.nextLine());
    sc.close();
}
...

Time limit

By default, the user program has 15 seconds to do all the necessary work. When the time runs out, the program shuts down and the user is shown a message that the time limit has been exceeded.

You can make the limit bigger, smaller, or you can turn it off. You can do this using the setTimeLimit(ms) method. Notice, that you need to specify the limit in milliseconds (so, by default it's 15000). If you want to disable time limit, make it negative or zero, for example -1.

testCase.setTimeLimit(10000)

Attach

You can also attach any object to the test case. This is an internal object, it will be available during checking inside the check method. You can put in the test input, feedback, anything that can help you to check this particular test case.

Below you can see how to attach the object to the test case. Note that it is necessary to apply generics.

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

class Attach {
    String feedback;
    int count;
    Attach(String feedback, int count) {
        this.feedback = feedback;
        this.count = count;
    }
}

public class SampleTest extends StageTest<Attach> {
    @Override
    public List<TestCase<Attach>> generate() {
        return List.of(
            new TestCase<Attach>().setAttach(
                new Attach("Sample feedback", 1)),

            new TestCase<Attach>().setAttach(
                new Attach("Another feedback", 2))
        );
    }

    @Override
    public CheckResult check(String reply, Attach attach) {
        if (!reply.contains(Integer.toString(attach.count))) {
            return CheckResult.wrong(attach.feedback);
        }
        return CheckResult.correct();
    }
}

Generate method

You can generate these test cases within the generate method. In the case of Java, this method must return List. You can see an example of using the method in the example of Attach (previous example).

Check method

Regular check method

The check method is used to check the the user's solution. This method has 2 parameters - reply and attach. The first parameter is the output of the user's program to the standard output. Below you can see examples of student programs that print data to the standard output.

...
public static void main(String[] args) {
    System.out.println("Hello World!");
    System.out.println("Second line also");
}
...

Everything that the user's program has outputted to the standard output is passed as the first argument to the check method. Therefore, in the following examples, the reply variable will be equal to "Hello world!\nSecond line also\n".

This method must return the CheckResult object. If the user's solution is incorrect, it is necessary to explain why - for example, in the example below in the check method it is checked that the user printed exactly 2 lines, and if the user not printed 2 lines, then inform the user what went wrong during checking.

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SampleTest extends StageTest {
    @Override
    public List<TestCase> generate() {
        return List.of(
            new TestCase()
        );
    }

    @Override
    public CheckResult check(String reply, Object attach) {
        if (reply.trim().split().length == 2) {
             return CheckResult.correct();
        }
        return CheckResult.wrong("You should output exactly two lines");
    }
}

Instead of returning CheckResult object you may throw WrongAnswer or TestPassed error to indicate about the result of the test. It can be useful if you are under a lot of methods and don't want to continue checking everything else.

Since throwing any exceptions in the check method is prohibited (it indicates that something went wrong in tests and tests should be corrected), a lot of tests forced to be written like this:

... deep into method calls
    if (some_parse_problem) {
        throw new Exception(feedback);
    }
...

... check method
List<Grid> grids;
try {
     grids = Grid.parse(out);
} catch (Exception ex) {
     return CheckResult.wrong(ex.getMessage());
}
...

But it's allowed to throw WrongAnswer and TestPassed errors. Now, you can write this code in a more understandable way without many unnecessary try/catch constructions.

import org.hyperskill.hstest.exception.outcomes.WrongAnswer;

... deep into method calls
    if (some_parse_problem) {
       throw new WrongAnswer(feedback);
    }
...

... check method
List<Grid> grids = Grid.parse(out);
...

Without check method

Sometimes test cases do not require a complex check method. For example, when the program output should be checked exactly. In this case, this method will always look like this:

CheckResult check(String reply, String expected) {
    boolean isCorrect = reply.trim().equals(expected.trim());
    return new CheckResult(isCorrect, feedback);
}

Therefore, there is a class SimpleTestCase. The constructor of this class accepts two arguments - input data and expected output data.

Let's look at an example of tests for a program that should sum up all the numbers that go into the standard input.

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.SimpleTestCase;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SumTest extends StageTest {
    @Override
    public List<TestCase> generate() {
        return List.of(
            new SimpleTestCase("2 3", "5"),
            new SimpleTestCase("1 2 3 4", "10"),
            new SimpleTestCase("7", "7"),
            new SimpleTestCase("0\n1 2", "3")
        );
    }
}

Here you can see that with the help of the SimpleTestCase class you do not have to write the check method yourself. One of the user's approaches to this task may look like this:

...
public static void main(String[] args) {
    Scanner scanner = new Scanner(System.in);
    String[] integers = scanner.nextLine().split(" ");
    int sum = 0;
    for (String num : integers) {
        sum += Integer.parseInt(num);
    }
    System.out.println(sum);
}
...

And after the checking the student gets: Wrong answer in test #4 and doesn't understand what is wrong with the solution. In fact, the solution expects all numbers to be in the first line, although the task says about all the numbers to be input, including the other lines. To do this, you can set a special feedback, which will appear next to the Wrong answer in test #4, so that the student can understand where the program error is. An updated, more correct example of the tests can be seen below:

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.SimpleTestCase;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SumTest extends StageTest {
    @Override
    public List<TestCase> generate() {
        return List.of(
            new SimpleTestCase("2 3", "5")
                .setFeedback("The numbers were: 2 3"),

            new SimpleTestCase("1 2 3 4", "10")
                .setFeedback("There are 4 numbers in the input"),

            new SimpleTestCase("7", "7")
                .setFeedback("Input may contain a single number, isn\'t it?"),

            new SimpleTestCase("0\n1 2", "3")
                .setFeedback("Numbers in this test appear in multiple lines")
        );
    }
}

You can mix tests that do not require a check method and regular tests. For normal tests, you also need to override the check method. In the example below, the check method is called only in the fifth test, the first four tests are tested as in the example above. Suppose the task says that if there are no numbers in the standard input, you should output an error message containing the word error.

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.SimpleTestCase;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SumTest extends StageTest {
    @Override
    public List<TestCase> generate() {
        return List.of(
            new SimpleTestCase("2 3\n", "5")
                .setFeedback("The numbers was: 2 3"),

            new SimpleTestCase("1 2 3 4", "10")
                .setFeedback("There are 5 numbers in the output"),

            new SimpleTestCase("7", "7")
                .setFeedback("There may be a single number, isn't it?"),

            new SimpleTestCase("0\n1 2", "3")
                .setFeedback("Numbers in this test appear in multiple lines"),

            new SimpleTestCase("1 -1", "0")
                .setFeedback("Sum of numbers in this test equals to 0"),

            new TestCase()
        );
    }

    @Override
    public CheckResult check(String reply, Object clue) {
        if (reply.toLowerCase().contains("error")) {
            return CheckResult.correct();
        }
        return CheckResult.wrong(
            "If there are no numbers in the standard input " +
                "you should print an error message");
    }
}

The correct solution for these tests is presented below:

...
public static void main(String[] args) {
    Scanner scanner = new Scanner(System.in);

    int sum = 0;
    boolean hasNumbers = false;
    while (scanner.hasNextInt()) {
        sum += scanner.nextInt();
        hasNumbers = true;
    }

    if (!hasNumbers) {
        System.out.println("error");
    } else {
        System.out.println(sum);
    }
}
...

Different check methods

Testing by check method is launched by default for test cases. But you can redefine this behavior by providing the test case with your own verification method. Thus, some test cases may check the student's solution using one method, while others may check the student's solution using another test method.

To change the standard check method, you need to call the setCheckFunc method and pass another method. This can be seen in the example below. Let's imagine that you need to write tests for a program that adds two numbers if the numbers are input, otherwise it will concatenate two lines. In the example below, the first two test cases are tested using the checkIntegerSum method; the next three test cases are tested using the checkStringConcat method.

import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;

public class SumTest extends StageTest<String> {
    @Override
    public List<TestCase<String>> generate() {
        List<TestCase<String>> tests = List.of(
            new TestCase<String>()
                .setCheckFunc(this::checkIntegerSum)
                .setAttach("12\n34"),

            new TestCase<String>()
                .setCheckFunc(this::checkIntegerSum)
                .setAttach("43\n23"),

            new TestCase<String>()
                .setCheckFunc(this::checkStringConcat)
                .setAttach("qw\ner"),

            new TestCase<String>()
                .setCheckFunc(this::checkStringConcat)
                .setAttach("12\new"),

            new TestCase<String>()
                .setCheckFunc(this::checkStringConcat)
                .setAttach("qw\n12")
        );

        for (TestCase<String> testCase : tests) {
            testCase.setInput(testCase.getAttach());
        }

        return tests;
    }

    private CheckResult checkIntegerSum(String reply, String attach) {

        String[] ints = attach.split("\n");

        int first = Integer.parseInt(ints[0]);
        int second = Integer.parseInt(ints[1]);

        int replied;

        try {
             replied = Integer.parseInt(reply.trim());
        } catch (NumberFormatException ex) {
            return CheckResult.wrong(
                "Your program didn't output a number!"
            );
        }

        if (first + second != replied) {
            return CheckResult.wrong(
                "You should output a sum of numbers " +
                    "when both numbers is integers"
            );
        }

        return CheckResult.correct();
    }

    private CheckResult checkStringConcat(String reply, String attach) {

        String[] words = attach.split("\n");

        if (!reply.strip().equals(words[0] + words[1])) {
            return CheckResult.wrong(
                "You should output a concatenation of words " +
                    "when at least one of the words in not integer"
            );
        }

        return CheckResult.correct();
    }
}