Application Testing

The levels in Unit Testing all reach the code by importing it. That leaves a layer untested, and it is the layer a user actually meets: the console script, the argument parser, what is written to STDOUT versus STDERR, and the exit code. None of it runs when a test does from myPackage import Thing.

Application testing starts the program the way a user does and looks at what came back.

See also

Unit Testing

→ The four levels below this one.

Application Testing

→ The reference for ApplicationTestcase.

What only a subprocess can tell you

Calling a program’s main() from a test is not the same as running it. Four things differ, and each has been a real bug:

  • The console script. pip install generates it from an entry point. A typo there ships a package whose command does not exist, and every import-based test still passes.

  • The argument parser. It is built when the program starts, not when a module is imported.

  • The exit code. main() returning 2 and the process exiting with 2 are different claims.

  • The streams. Which messages go to STDOUT and which to STDERR is only observable from outside.

from pyTooling.Testing import ApplicationTestcase


class Version(ApplicationTestcase):
  _consoleScript  = "myprogram"
  _runnableModule = "myPackage"

  def test_Version(self) -> None:
    result = self.RunEntrypoint("--version")

    self.assertExitCode(result)
    self.assertIn("myprogram", result.stdout)

Both class variables are mandatory. A class naming neither can run nothing, so setUpClass() says so once instead of letting every testcase in the class fail with a less obvious error.

Two ways to start the program

RunEntrypoint() runs the installed console script, resolved on PATH. RunModule() runs python -m <module>, bypassing it.

Having both is a diagnosis, not a convenience: if RunModule passes while RunEntrypoint fails, the packaging is at fault and the code is fine. That distinction is invisible from a single test.

def test_TheEntryPointIsWiredUp(self) -> None:
  viaScript = self.RunEntrypoint("--version")
  viaModule = self.RunModule("--version")

  self.assertExitCode(viaScript)
  self.assertEqual(viaModule.stdout, viaScript.stdout)

Reading the result

assertExitCode() compares the exit code and, when it differs, reports the command line together with everything the program printed. That output is what explains the failure, and it is gone once the test has finished - so it belongs in the assertion message, not in the console.

A program writing to a terminal colors its output; the same program in a pipe usually does not, but that depends on the program. stripANSIColorCodes() removes the escape sequences, which is more robust than encoding a rule about when they appear:

self.assertIn("ERROR", stripANSIColorCodes(result.stderr))

What to test at this level

Application testing is expensive - every testcase is a process - so it tests the wiring, not the logic. The logic belongs to the levels below, where a failure is cheaper to read.

  • The program starts at all, and --version and --help work.

  • Every sub-command is reachable and its arguments parse.

  • Each documented exit code is actually produced.

  • An error message reaches STDERR rather than STDOUT.

  • A file the program writes exists afterwards and has the expected content.