report:unittest-summary

The report:unittest-summary directive renders a unit test report as a table: a row per testsuite and testcase, with the counts of testcases, skipped, errored, failed and passed ones, the assertions and the runtime. The hierarchy of testsuites is shown by indentation, a status by a symbol and a CSS class. A legend below the table explains the symbols.

The report is read by pyEDAA.Reports in the Any JUnit XML format, so the reports of pytest, OSVVM and other JUnit dialects are accepted, or in pyTooling’s own XML format. Report files are declared in conf.py, each with an identifier the directive names in its report:unittest-summary:reportid option - so a documentation shows as many reports as it declares.

A report can get a page per testsuite and testcase, which the table links to and the roles tc and ts refer to.

The directive is part of pyTooling.Sphinx; it needs the extra reports.

.. report:unittest-summary::
   :reportid: example

This is how the example renders, from a report of five testcases. The report is declared with pages, so every testsuite and testcase links to its page:

Testsuite / Testcase	Testcases	Skipped	Errored	Failed	Passed	Assertions	Runtime (HH:MM:SS.sss)
❗pytest tests	5	1	1	1	2		00:00:00.290
❗pytest	5	1	1	1	2		00:00:00.290
❗tests	5	1	1	1	2		00:00:00.220
❗unit	5	1	1	1	2		00:00:00.220
❗Arithmetic	5	1	1	1	2		00:00:00.220
✅Addition	2	0	0	0	2		00:00:00.110
✅test_Negative						0	00:00:00.060
✅test_Positive						0	00:00:00.050
❗Division	3	1	1	1	0		00:00:00.110
❌test_ByZero						0	00:00:00.070
⚠️test_Fraction						0	00:00:00.000
❗test_Rounding						0	00:00:00.040
❗ ERRORED	5	1	1	1	2		00:00:00.290

Legend: ✅ passed, ❌ failed, ⚠️ skipped, ➖ excluded, ❗ errored, ⛔ setup error, 🚫 aborted, ❓ unknown

The documentation’s own unit test report is rendered by this directive too, with a page per testsuite and testcase.

Installing the extra `reports`

The directives of domain report are registered by the extension pyTooling.Sphinx, but read the reports - and analyze the documentation coverage - with pyEDAA.Reports, an optional dependency. A project using a report installs it with the extra reports:

pip install pyTooling.Sphinx[reports]

Without it, the extension and every other directive work; a report directive reports on the page and in the build’s log that it needs it:

Caught ReportsPackageMissingError when reading '../report/unit/unittest.xml'.
  ReportsPackageMissingError: Reading a unit test report needs 'pyEDAA.Reports', which isn't installed.
    Install it with: pip install pyTooling.Sphinx[reports]

Configuration

The report files are declared in conf.py as a dictionary, keyed by the identifier a directive names in report:unittest-summary:reportid. A path is relative to the documentation’s source directory.

pyTooling_Unittest_Testsuites

→ The unit test reports, by identifier. An entry has these fields:

xml_report: → The report file, in Any JUnit XML format or pyTooling’s XML format. Required.
pages: → The document name the pages per testsuite and testcase are generated below, e.g. unittests/src. Optional; without it, the report gets no pages.

# doc/conf.py
pyTooling_Unittest_Testsuites = {
  "src": {
    "xml_report": "../report/unit/unittest.xml",
    "pages":      "unittests/src",
  }
}

The first entry is the default report of the roles tc and ts.

The configuration is checked when the build starts. A report file that doesn’t exist is logged as an error, and the entries after it aren’t read: an entry whose file is written later - e.g. by a test run in CI - goes last.

Report formats

The format of a report file is recognized by its root element:

<testsuites> or <testsuite>: → A JUnit XML file, read by pyEDAA.Reports in the Any JUnit dialect - e.g. pytest’s --junit-xml. A testcase’s failure, error or skip message, its traceback and its captured output are shown on its page.
<TestReport>: → A test report in pyTooling’s XML format, written by pyTooling’s pytest plugin with --pytooling-xml. Its testsuites nest, and a testsuite or testcase carries a title, a summary and a description - e.g. from the markers @testsuite and @testcase and the doc-strings. A page shows them below its name, a link shows the title on hover.

Testsuite / Testcase	Testcases	Skipped	Errored	Failed	Passed	Runtime (HH:MM:SS.sss)
❌TestReport	6	2	0	2	2	00:00:00.006
❌tests	6	2	0	2	2	00:00:00.006
❌unit	6	2	0	2	2	00:00:00.006
❌Arithmetic	6	2	0	2	2	00:00:00.006
❌Addition	3	0	0	1	2	00:00:00.005
✅Negative						00:00:00.000
✅OnePlusOne						00:00:00.004
❌OnePlusTwo						00:00:00.000
❌Division	3	2	0	1	0	00:00:00.001
⚠️ByFraction						00:00:00.000
❌ByZero						00:00:00.000
⚠️RoundsUp						00:00:00.000
❌ FAILED	6	2	0	2	2	00:00:00.006

Legend: ✅ passed, ❌ failed, ⚠️ skipped, ➖ excluded, ❗ errored, ⛔ setup error, 🚫 aborted, ❓ unknown

Pages per testsuite and testcase

A report declared with the key pages gets a page per testsuite and per testcase. They are generated when the documentation is built - no source file is written - below the document name the key states:

a testsuite’s page: <pages>/<testsuite>/<testsuite>/..., the names of the testsuites from the report’s top down, e.g. unittests/example/pytest/tests/unit/Arithmetic;
a testcase’s page: <pages>/<testsuite>/.../<testcase>, e.g. unittests/example/pytest/tests/unit/Arithmetic/Division/test_ByZero.

Characters of a name other than letters, digits, _, - and . - e.g. the brackets of a parametrized pytest testcase - are replaced by _. A name colliding with a sibling’s, also when differing in case only, gets the suffix -2, -3, and so on.

A testcase’s page shows its status, duration, number of assertions (if the report has it) and its testsuite, linked; then the message and details of a failure, error or skip, and the captured standard output and standard error.

A testsuite’s page shows its status, the number of testcases per status, the duration, the number of assertions (if the report has it) and its parent testsuite; then a table of its testsuites and a table of its testcases, each linked to its page.

The title, summary and description of a pyTooling report are shown below the name, as plain text: they are doc-strings, which aren’t necessarily ReST of this documentation.

The report:unittest-summary table links every testsuite and testcase to its page; that is the entry point. The directive lists the report’s top-level testsuites in a hidden table of contents, and a testsuite’s page lists its testsuites and testcases, so the pages are below the summary’s document in the navigation - in the section holding the directive. A page’s own sections - Summary, Testcases, … - aren’t listed there. A report is best shown on a page of its own: a title, an introduction and the directive.

Hint

A testcase is as many levels below the summary’s document as the report nests testsuites - e.g. seven for pytest/tests/unit/Arithmetic/Division/test_ByZero. A theme limits how deep its navigation goes, e.g. the option navigation_depth of the Read the Docs theme (default 4; -1 = unlimited). pyedaa-reports removes levels like pytest/tests/unit from a JUnit report with --pytest=reduce-depth:pytest.tests.unit.

The pages are orphans, so a report with pages, but without a summary directive, causes no warnings; its pages are reached by the roles only.

Sphinx regards the report file as the source of its pages, so a changed report regenerates its pages. As the pages are in the table of contents, LaTeX writes them too - below the summary’s section, the deepest levels as paragraphs.

Hint

A report can have thousands of testcases - pages are only generated for a report declaring pages.

Roles

The roles tc and ts refer to a testcase and a testsuite of a report with pages. They are registered in domain report - as :report:tc: and :report:ts: - and without the domain’s prefix.

:tc:

Refers to a testcase’s page, showing the testcase’s name, or the role’s title: :tc:`<target>` or :tc:`title <<target>>`.

The target is [<report ID>:]<qualified name>. The qualified name is the path from the report’s top down, the names joined by .: e.g. pytest.tests.unit.Arithmetic.Division.test_ByZero. A suffix of it starting at a name - Division.test_ByZero or test_ByZero - is accepted, if it’s unique. Without a report ID, the first report of pyTooling_Unittest_Testsuites is meant.

A target matching nothing or several testcases is warned about.

:ts:: Refers to a testsuite’s page, as tc refers to a testcase’s.

* :tc:`test_ByZero`
* :tc:`example:Division.test_Fraction`
* :ts:`Arithmetic`
* :ts:`pytooling:Addition`
* :tc:`the skipped one <test_Fraction>`

The testsuites and testcases are objects of domain report, named <report ID>:<qualified name> in the documentation’s inventory (objects.inv), so another documentation refers to them with intersphinx.

Legend

A status is shown by a symbol, the same for a testsuite and a testcase. The legend below the table names each symbol with its status; option report:unittest-summary:legend draws it above the table instead, or leaves it out, e.g. when the page explains the symbols once for several tables.

Symbol	Status
✅	passed; a testcase failing as expected
❌	failed; a testcase passing unexpectedly, and a status without a symbol of its own
⚠️	skipped
➖	excluded
❗	errored
⛔	setup error
🚫	aborted
❓	unknown

Options

.. report:unittest-summary::

Renders a unit test report as a table, on landscape pages in LaTeX.

:reportid: <identifier>: The report, as declared in pyTooling_Unittest_Testsuites. Required.

:testsuite-summary-name: <name>: The name of the row summarizing all testsuites, instead of the report’s own.

:show-testcases: all | not-passed: Which testcases are listed per testsuite: all of them, or only those that didn’t pass. Default: all.

:no-assertions:: Leaves out the column of assertions.

:hide-testsuite-summary:: Leaves out the row summarizing all testsuites above the testsuites.

:legend: top | bottom | none: Where the legend of the status symbols is drawn: above the table, below it, or not at all. Default: bottom, also for the option without a value.

:class: <CSS classes>: Additional CSS classes on the table.

Styling

The table carries the classes report-unittest-table and report-unittest-<identifier>. A row carries its kind - report-testsuitesummary, report-testsuite, report-testcase or report-summary - and its status, e.g. testcase-failed. The legend is a paragraph with the classes report-unittest-legend and report-unittest-legend-<position>, each of its items an inline with the class report-unittest-legend-item. The stylesheet of pyTooling.Sphinx colours the rows by status; a project overrides these rules in a stylesheet of its own, listed in html_css_files:

table.report-unittest-table > tbody > tr.testcase-skipped {
  background: #eee;
}

A name is indented by its depth, two steps per level, as in the code coverage tables.

Deriving a table

A project reading a test report of its own - e.g. a simulation framework’s build summary with alert counts and functional coverage - derives its directive from UnittestSummaryBase, the base-class of report:unittest-summary. The base-class renders the rows - a testsuite summary, its testsuites and its listed testcases, indented by depth, the first column linked to the pages -, the derived class names the other columns:

from pyTooling.Sphinx.Unittest import Column, ColumnGroup, UnittestSummary

class AssertionSummary(UnittestSummary):
  directiveName = "assertion-summary"

  def _Columns(self):
    return [
      Column("Testcases", 1, testsuite=lambda testsuite: f"{testsuite.TestcaseCount}"),
      ColumnGroup("Assertions", [
        Column("Passed", 1, testcase=lambda testcase: f"{testcase.PassedAssertionCount}"),
        Column("Total",  1, testcase=lambda testcase: f"{testcase.AssertionCount}")
      ])
    ]

A Column names a function per kind of row - testsuite for a testsuite’s and the testsuite summary’s rows, testcase for a testcase’s row -, which returns the cell’s content: a text, a node, or the cell itself, e.g. to give it a CSS class. A ColumnGroup puts columns under a common title, in a second header row.

A directive reading another kind of report derives from UnittestSummaryBase itself: it reads its options in _CheckOptions, its reports into its own class variables _reports and _readErrors, and returns its pages by _Pages. Its pages derive from UnittestReportPages, which adds fields to a testcase’s summary by _TestcaseFields, and columns to the testcase table of a testsuite’s page by _TestcaseColumns and _TestcaseCells. A domain of its own, with roles of its own, derives from ReportDomain and names its roles in _ROLES.

A derived table gets the legend below it. It accepts option :legend: by adding "legend": LegendPosition.Parse (LegendPosition) to its option_spec; a table showing other symbols overrides the class variable _LEGEND, a tuple of (symbol, text) pairs.

Errors

A mistake in the options, or a report that can’t be read, is reported on the page where the table would be, and in the build’s log:

Caught ReportExtensionError when checking options for directive 'unittest-summary'.
  ReportExtensionError: No unit testing configuration item for 'src'.