UnitTesting

This template runs multiple jobs from a matrix as a cross of Python versions and systems. The summary report in junit XML format is optionally uploaded as an artifact.

Configuration options to pytest should be given via section [tool.pytest.ini_options] in a pyproject.toml file.

Instantiation

Simple Example

The following instantiation example creates a UnitTesting job derived from job template UnitTesting version @r5. For providing the job matrix as a JSON string, the Parameters job template is used. Additionally, the job needs configuration settings, which are stored in pyproject.toml. Instead of duplicating these settings, the ExtractConfiguration job template is used to extract these settings.

name: Pipeline

on:
  push:
  workflow_dispatch:

jobs:
  ConfigParams:
    uses: pyTooling/Actions/.github/workflows/ExtractConfiguration.yml@r5
    with:
      package_name: myPackage

  UnitTestingParams:
    uses: pyTooling/Actions/.github/workflows/Parameters.yml@r5
    with:
      package_name: myPackage

  UnitTesting:
    uses: pyTooling/Actions/.github/workflows/UnitTesting.yml@r5
    needs:
      - ConfigParams
      - UnitTestingParams
    with:
      jobs: ${{ needs.UnitTestingParams.outputs.python_jobs }}
      requirements: "-r tests/unit/requirements.txt"
      unittest_report_xml_directory:  ${{ needs.ConfigParams.outputs.unittest_report_xml_directory }}
      unittest_report_xml_filename:   ${{ needs.ConfigParams.outputs.unittest_report_xml_filename }}
      unittest_xml_artifact:          ${{ fromJson(needs.UnitTestingParams.outputs.artifact_names).unittesting_xml }}
      coverage_sqlite_artifact:       ${{ fromJson(needs.UnitTestingParams.outputs.artifact_names).codecoverage_sqlite }}

jobs:
  Params:
    # ...

  UnitTesting:
    uses: pyTooling/Actions/.github/workflows/UnitTesting.yml@r5
    needs:
      - Params
    with:
      jobs: ${{ needs.Params.outputs.python_jobs }}
      artifact: ${{ fromJson(needs.Params.outputs.artifact_names).unittesting }}

See also

Parameters

Parameters is usually used to pre-compute the job matrix as a JSON string with all system × environment × Python version combinations.

PublishTestResults

PublishTestResults can be used to merge all JUnit test reports into one file.

PublishCoverageResults

PublishCoverageResults can be used to merge all code coverage reports into one file.

Parameter Summary

Goto input parameters

Parameter Name

Required

Type

Default

jobs

yes

string

— — — —

apt

no

string

''

brew

no

string

''

pacboy

no

string

''

requirements

no

string

'-r tests/requirements.txt'

mingw_requirements

no

string

''

macos_before_script

no

string

''

macos_arm_before_script

no

string

''

ubuntu_before_script

no

string

''

mingw64_before_script

no

string

''

ucrt64_before_script

no

string

''

root_directory

no

string

''

tests_directory

no

string

'tests'

unittest_directory

no

string

'unit'

unittest_report_xml_directory

no

string

'report/unit'

unittest_report_xml_filename

no

string

'TestReportSummary.xml'

coverage_config

no

string

'pyproject.toml'

coverage_report_xml_directory

no

string

'report/coverage'

coverage_report_xml_filename

no

string

'coverage.xml'

coverage_report_json_directory

no

string

'report/coverage'

coverage_report_json_filename

no

string

'coverage.json'

coverage_report_html_directory

no

string

'report/coverage/html'

unittest_xml_artifact

no

string

''

unittest_html_artifact

no

string

''

coverage_sqlite_artifact

no

string

''

coverage_xml_artifact

no

string

''

coverage_json_artifact

no

string

''

coverage_html_artifact

no

string

''

Goto secrets

This job template needs no secrets.

Goto output parameters

This job template has no output parameters.

Input Parameters

jobs

Type:

string

Required:

yes

Default Value:

— — — —

Possible Values:

A JSON string with an array of dictionaries with the following key-value pairs:

  • sysicon - icon to display

  • system - name of the system

  • runs-on - virtual machine image and base operating system

  • runtime - name of the runtime environment if not running natively on the VM image

  • shell - name of the shell

  • pyicon - icon for CPython or pypy

  • python - Python version

  • envname - full name of the selected environment

Description:

A JSON encoded job matrix to run multiple Python job variations.

apt

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for apt install.
Packages are specified as a space separated list like 'graphviz curl gzip'.

Description:

Additional Ubuntu system dependencies to be installed through apt.

Example:
UnitTests:
  ...
  with:
    apt: >-
      graphviz
      curl
      gzip

brew

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for brew install.
Packages are specified as a space separated list.

Description:

Additional macOS system dependencies to be installed through brew.

pacboy

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for pacboy.
Packages are specified as a space separated list like 'python-lxml:p python-numpy:p'.

Description:

Additional MSYS2 system dependencies to be installed through pacboy (pacman).
Usually, Python packages start with python-. The suffix :p ensures pacboy figures out the correct package repository prefix for MinGW64, UCRT64, …

Note

Internally, a dedicated workflow step reads the requirements file for Python and compares requested packages with a list of packages that should be installed through pacman/pacboy compared to installation via pip. These are mainly core packages or packages with embedded C code.
The list of identified packages is handed over to pacboy for preinstallation. Otherwise pip will later raise an error.
The packages listed by this parameter will be installed in addition to the identified packages.

Attention

Ensure your Python requirements match the available version from MSYS2 packages list, otherwise if your requirements.txt requests a newer version then provided by MSYS2, such a dependency will fail.

Example:
UnitTests:
  ...
  with:
    pacboy: >-
      python-lxml:p
Packages:

The following list of Python packages is identified to be installed via pacboy:

requirements

Type:

string

Required:

no

Default Value:

'-r tests/requirements.txt'

Possible Values:

Any valid list of parameters for pip install.
Either a requirements file can be referenced using '-r path/to/requirements.txt', or a list of packages can be specified using a space separated list like 'coverage pytest'.

Description:

Python dependencies to be installed through pip.

mingw_requirements

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for pip install.
Either a requirements file can be referenced using '-r path/to/requirements.txt', or a list of packages can be specified using a space separated list like 'coverage pytest'.

Description:

Override Python dependencies to be installed through pip in MSYS2 (MinGW64/UCRT64) only.

macos_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash instructions as single-line or multi-line string suitable for macOS (Intel platform).

Description:

These optional Bash instructions for macOS are executed after setting up the environment and installing the platform specific dependencies and before running the unit test.

macos_arm_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash instructions as single-line or multi-line string suitable for macOS (ARM platform).

Description:

These optional Bash instructions for macOS are executed after setting up the environment and installing the platform specific dependencies and before running the unit test.

ubuntu_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash instructions as single-line or multi-line string suitable for Ubuntu.

Description:

These optional Bash instructions for Ubuntu are executed after setting up the environment and installing the platform specific dependencies and before running the unit test.

mingw64_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash instructions as single-line or multi-line string suitable for MinGW64 on Windows.

Description:

These optional Bash instructions for MinGW64 on Windows are executed after setting up the environment and installing the platform specific dependencies and before running the unit test.

ucrt64_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash instructions as single-line or multi-line string suitable for UCRT64 on Windows.

Description:

These optional Bash instructions for UCRT64 on Windows are executed after setting up the environment and installing the platform specific dependencies and before running the unit test.

Hint

The next parameters allow running different test kinds (unit tests, performance tests, platform tests, …) with the same job template, but isolated in sub-directories, thus pytest only discovers a subset of tests. The following code blocks showcase how the job template uses these parameters and how it relates to a proposed directory structure.

Relation between root_directory, tests_directory and unittest_directory
cd <RepositoryRoot>
cd ${root_directory}

python -m \
  pytest -raP \
    --color=yes ..... \
    "${tests_directory}/${unittest_directory}"
Directory Structure
<RepositoryRoot>/
  doc/
  myPackage/
    __init__.py
  tests/
    unit/
      myTests.py
Example for Default Values
cd <RepositoryRoot>
cd .

python -m \
  pytest -raP \
    --color=yes ..... \
    "tests/unit"

root_directory

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid directory or sub-directory.

Description:

Working directory for running tests.
Usually, this is the repository’s root directory. Tests are called relatively from here. See tests_directory and unittest_directory.

tests_directory

Type:

string

Required:

no

Default Value:

'tests'

Possible Values:

Any valid directory or sub-directory.

Description:

Path to the directory containing tests (relative from root_directory).

unittest_directory

Type:

string

Required:

no

Default Value:

'unit'

Possible Values:

Any valid directory or sub-directory.

Description:

Path to the directory containing unit tests (relative from tests_directory).

unittest_report_xml_directory

Type:

string

Required:

no

Default Value:

'report/unit'

Possible Values:

Any valid directory or sub-directory.

Description:

Directory or sub-directory where the unittest summary report in XML format will be saved.
This path is configured in pyproject.toml and can be extracted by ExtractConfiguration.

unittest_report_xml_filename

Type:

string

Required:

no

Default Value:

'TestReportSummary.xml'

Possible Values:

Any valid filename accepted by pytest ... --junitxml=${unittest_report_xml_filename}.

Description:

Filename of the generated JUnit XML report.
This filename is configured in pyproject.toml and can be extracted by ExtractConfiguration.

coverage_config

Type:

string

Required:

no

Default Value:

'pyproject.toml'

Possible Values:

TBD

coverage_report_xml_directory

Type:

string

Required:

no

Default Value:

'report/coverage'

Possible Values:

Any valid directory or sub-directory.

Description:

Directory or sub-directory where the code covergae report in XML format will be saved.
This path is configured in pyproject.toml and can be extracted by ExtractConfiguration.

coverage_report_xml_filename

Type:

string

Required:

no

Default Value:

'coverage.xml'

Possible Values:

Any valid XML filename.

Description:

Filename of the generated code coverage report in Cobertura format.
This filename is configured in pyproject.toml and can be extracted by ExtractConfiguration.

coverage_report_json_directory

Type:

string

Required:

no

Default Value:

'report/coverage'

Possible Values:

Any valid directory or sub-directory.

Description:

Directory or sub-directory where the code covergae report in JSON format will be saved.
This path is configured in pyproject.toml and can be extracted by ExtractConfiguration.

coverage_report_json_filename

Type:

string

Required:

no

Default Value:

'coverage.json'

Possible Values:

Any valid JSON filename.

Description:

Filename of the generated code coverage report in Coverage.py JSON format.
This filename is configured in pyproject.toml and can be extracted by ExtractConfiguration.

coverage_report_html_directory

Type:

string

Required:

no

Default Value:

'report/coverage/html'

Possible Values:

Any valid directory or sub-directory.

Description:

Directory or sub-directory where the code covergae report in HTML format will be saved.
This path is configured in pyproject.toml and can be extracted by ExtractConfiguration.

unittest_xml_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the unittest report summary in XML format.

unittest_html_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the unittest report in HTML format.

coverage_sqlite_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the code coverage report as SQLite database.

coverage_xml_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the code coverage report in XML format.

coverage_json_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the code coverage report in JSON format.

coverage_html_artifact

Type:

string

Required:

no

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the code coverage report in HTML format.

Secrets

This job template needs no secrets.

Outputs

This job template has no output parameters.

Optimizations

The following optimizations can be used to reduce the template’s runtime.

Disable unit test XML generation

If parameter unittest_xml_artifact is empty, no unit test summary report will be generated and no JUnit XML artifact will be uploaded.

Disabled code coverage collection

If parameter coverage_config is empty, no code coverage will be collected.

Disable code coverage SQLite database artifact upload

If parameter coverage_sqlite_artifact is empty, the collected code coverage database (SQLlite format) wont be uploaded as an artifact.

Disable code coverage report conversion to the Cobertura XML format.

If parameter coverage_xml_artifact is empty, no Cobertura XML file will be generated from code coverage report. As no Cobertura XML file exists, no code coverage XML artifact will be uploaded.

Disable code coverage report conversion to the Coverage.py JSON format.

If parameter coverage_json_artifact is empty, no Coverage.py JSON file will be generated from code coverage report. As no JSON file exists, no code coverage JSON artifact will be uploaded.

Disable code coverage report conversion to an HTML website.

If parameter coverage_html_artifact is empty, no coverage report HTML report will be generated from code coverage report. As no HTML report exists, no code coverage HTML artifact will be uploaded.