ApplicationTesting

The ApplicationTesting job template runs tests against the packaged and installed Python package, on a matrix of Python versions and systems. It is the counterpart of UnitTesting, which runs tests against the sources in the repository.

The distinction matters, because the two find different defects. Unit testing imports the package from the checked out working directory, so it passes even when a module is missing from the wheel, an entry point is misspelled or a py.typed marker was never packaged. Application testing downloads the wheel artifact produced by Package, installs it with pip and runs the tests from tests/app against that installation.

Configuration options for pytest should be given via section [tool.pytest.ini_options] in a pyproject.toml file.

Instantiation

The following instantiation example creates an AppTesting job derived from job template ApplicationTesting version @r8. The job matrix comes from Parameters and the wheel from Package, so both jobs must be listed as dependencies.

jobs:
  Params:
    uses: pyTooling/Actions/.github/workflows/Parameters.yml@r8
    with:
      package_name: myPackage

  Package:
    uses: pyTooling/Actions/.github/workflows/Package.yml@r8
    needs:
      - Params
    with:
      artifact: ${{ fromJson(needs.Params.outputs.artifact_names).package_all }}

  AppTesting:
    uses: pyTooling/Actions/.github/workflows/ApplicationTesting.yml@r8
    needs:
      - Params
      - Package
    with:
      jobs:                 ${{ needs.Params.outputs.python_jobs }}
      wheel:                ${{ fromJson(needs.Params.outputs.artifact_names).package_all }}
      apptest_xml_artifact: ${{ fromJson(needs.Params.outputs.artifact_names).apptesting_xml }}

See also

UnitTesting

Runs the same kind of tests against the sources instead of against the installed package.

Package

Produces the wheel artifact this template installs.

PublishTestResults

Merges the produced JUnit XML reports and publishes them.

Parameter Summary

Goto input parameters

Parameter Name

Required

Type

Default

jobs

yes

string

— — — —

wheel

no

string

''

apt

no

string

''

brew

no

string

''

pacboy

no

string

''

requirements

no

string

'-r ./requirements.txt'

mingw_requirements

no

string

''

macos_before_script

no

string

''

macos_arm_before_script

no

string

''

ubuntu_before_script

no

string

''

windows_before_script

no

string

''

windows_arm_before_script

no

string

''

ucrt64_before_script

no

string

''

root_directory

no

string

''

tests_directory

no

string

'tests'

apptest_directory

no

string

'app'

parallel

no

string

'auto'

apptest_report_xml

no

string (JSON)

{"directory": "report/app", "filename": "TestReportSummary.xml", "fullpath": "report/app/TestReportSummary.xml"}

apptest_xml_artifact

no

string

''

unittest_html_artifact

no

string

''

Goto secrets

This job template needs no secrets.

Goto output parameters

This job template has no output parameters.

Input Parameters

jobs

Type:

string

Required:

yes

Default Value:

— — — —

Possible Values:

A JSON string with an array of dictionaries with the following key-value pairs:

sysicon:

icon to display

system:

name of the system

runs-on:

virtual machine image and base operating system

runtime:

name of the runtime environment if not running natively on the VM image

shell:

name of the shell

pyicon:

icon for CPython or pypy

python:

Python version

envname:

full name of the selected environment

Description:

A JSON encoded job matrix to run multiple Python job variations.
Usually taken from python_jobs.

wheel

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid artifact name.

Description:

Name of the artifact containing the wheel package to install and test.
Produced by Package. If empty, no package is downloaded and the tests run against whatever is installed in the environment.

apt

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for apt install.
Packages are specified as a space separated list like 'graphviz curl gzip'.

Description:

Additional Ubuntu system dependencies to be installed through apt.

brew

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for brew install.
Packages are specified as a space separated list.

Description:

Additional macOS system dependencies to be installed through homebrew.

pacboy

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for pacboy sync.
Packages are specified as a space separated list like 'python-pip:p graphviz:p'.

Description:

Additional MSYS2 dependencies to be installed through pacboy (pacman).

requirements

Type:

string

Required:

no

Default Value:

'-r ./requirements.txt'

Possible Values:

Any valid list of parameters for pip install.
Either a requirements file can be referenced using '-r path/to/requirements.txt', or a list of packages can be specified using a space separated list.
A path starting with ./ is resolved relative to the application test directory, which is the concatenation of root_directory, tests_directory and apptest_directory; with the defaults '-r ./requirements.txt' refers to tests/app/requirements.txt. Any other path is used as given, thus relative to the repository root.
A missing file aborts the job with a FileNotFoundError annotation naming the resolved path. mingw_requirements is resolved and checked the same way, so ./ addresses the same directory in both parameters.

Description:

Python dependencies needed to run the application tests, installed through pip.
The package under test is not installed from here - it comes from wheel.

mingw_requirements

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid list of parameters for pip install.

Description:

Overrides requirements on MSYS2 only.
MSYS2 provides some Python packages through pacboy, so the pip requirements often differ there.
The value is resolved and its existence checked exactly like requirements.
If left empty, requirements is installed on MSYS2 as well.

macos_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash script.

Description:

Scripts to execute on macOS (Intel) before pytest is started.

macos_arm_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash script.

Description:

Scripts to execute on macOS (Apple silicon) before pytest is started.

ubuntu_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash script.

Description:

Scripts to execute on Ubuntu (x86-64 and aarch64) before pytest is started.

windows_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid PowerShell script.

Description:

Scripts to execute on Windows (x86-64) before pytest is started.

windows_arm_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid PowerShell script.

Description:

Scripts to execute on Windows (aarch64) before pytest is started.

ucrt64_before_script

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid Bash script.

Description:

Scripts to execute on Windows within MSYS2 UCRT64 before pytest is started.

root_directory

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any path relative to the repository root. An empty string means the repository root.

Description:

Working directory from which pytest is started.

tests_directory

Type:

string

Required:

no

Default Value:

'tests'

Possible Values:

Any path relative to root_directory.

Description:

Directory containing all tests.

apptest_directory

Type:

string

Required:

no

Default Value:

'app'

Possible Values:

Any path relative to tests_directory.

Description:

Directory containing the application tests.
With the defaults, the tests are collected from tests/app.

parallel

Type:

string

Required:

no

Default Value:

'auto'

Possible Values:

'auto', a number of workers, or 'false'.

Description:

Run the tests in parallel with pytest-xdist, passed as -n <value>. With 'auto', pytest-xdist starts one worker per CPU of the runner. With 'false', the tests run serially.

Attention

pytest-xdist has to be listed in the requirements (requirements). If it’s missing, the tests run serially and the job reports a warning.

Note

The tests have to be independent of each other and of the order they run in, as each worker runs a share of them in a process of its own.

apptest_report_xml

Type:

string (JSON)

Required:

no

Default Value:

{"directory": "report/app", "filename": "TestReportSummary.xml", "fullpath": "report/app/TestReportSummary.xml"}

Possible Values:

Any valid JSON string containing a JSON object with fields:

directory:

Directory or sub-directory where the report will be saved.

filename:

File name of the report.

fullpath:

Directory and file name of the report.

Description:

Path of the application test summary report in JUnit XML format, as a JSON object.
This path is configured in pyproject.toml and can be extracted by ExtractConfiguration.

apptest_xml_artifact

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid artifact name. An empty string disables the upload.

Description:

Name of the artifact receiving the application test report in JUnit XML format.
If empty, pytest is run without --junitxml and no report is uploaded.

unittest_html_artifact

Type:

string

Required:

no

Default Value:

''

Possible Values:

Any valid artifact name. An empty string disables the upload.

Description:

Name of the artifact receiving the application test report in HTML format.

Note

The parameter is named unittest_html_artifact although it carries the application test report. The name is kept for backwards compatibility with existing pipeline instantiations.

Secrets

This job template needs no secrets.

Outputs

This job template has no output parameters.