Overview

The module pyTooling.Dependency models the dependencies between packages: which package depends on which version of which other package, where those packages are published, and what the resulting graph looks like.

Data Model

The generic data model is storage agnostic - it describes packages and versions without assuming where they come from:

Package is a package by name, with the versions it has.
PackageVersion is one version of a package, when it was released, and which versions of which other packages it depends on.
PackageStorage is a place packages are published - an index, a registry, a repository.
PackageDependencyGraph collects the packages known from one or more storages.

Conversion to a Graph

ToGraph() converts a dependency graph into a pyTooling.Graph.Graph. The algorithms of pyTooling.Graph then apply - IterateTopologically() yields the versions dependencies first, HasCycle() finds a circular dependency, and pyTooling.Graph.GraphViz or pyTooling.Graph.GraphML write it for a viewer.

Every package version of every storage becomes a Vertex:

Vertex member	Content
`ID`	the `PackageVersion`, so `Graph.GetVertexByID` finds a version’s vertex. Written by a graph writer as `<package> - <version>`.
`Value`	empty (`None`) - the ID already references the version.
key `license`	the license as `LicenseExpression`, e.g. `BSD-3-Clause`; an `UnknownLicense` (`NOASSERTION`) if it’s unknown. Always set.
key `releasedAt`	the release time as `datetime`.
keys `licenseURL`, `repositoryURL`, `documentationURL`, `issueTrackerURL`, `projectURL` and `changelogURL`	the URL of the license text, the source repository, the documentation, the issue tracker, the project’s homepage and the changelog, each a `URL`.

The key-value pairs keep the version’s typed objects; a graph writer converts them to text. All but license are set only if the version knows the value - for a Python package, after its details were loaded; converting a graph doesn’t load them.

Every dependency becomes an Edge from the version needing to the version it needs, so an edge reads needs. An edge has no ID, value or key-value pairs.

from pyTooling.Graph.GraphViz import Graph as DotGraph

graph = dependencyGraph.ToGraph()
for vertex in graph.IterateTopologically():
  print(vertex.ID, vertex["license"])

dot = DotGraph("Example")
dot.FromGraph(graph)
print(dot)

For a graph in which myApp 1.0.0 depends on colorama 0.4.6, Graphviz’ DOT language reads:

digraph "Example" {
  "myApp - 1.0.0";
  "colorama - 0.4.6";
  "myApp - 1.0.0" -> "colorama - 0.4.6";
}

pyTooling.Graph.GraphML writes each key-value pair as text in a <data> element of the node, declared by a key named node<key>, e.g. <data key="nodelicense">BSD-3-Clause</data>.

Python Packages

pyTooling.Dependency.Python implements that model for Python packages published on PyPI:

Project and Release are the Python flavours of a package and a package version.
Distribution describes a single distribution file of a release - a wheel or a source distribution.
PythonPackageIndex queries PyPI, and PythonPackageDependencyGraph is the graph built from it.

Details are fetched on demand rather than up front: a project knows its releases before it knows anything about them, and LazyLoadableMixin loads the rest when it is first asked for. A dependency graph is otherwise thousands of HTTP requests wide.

Attention

Querying PyPI needs aiohttp, requests and packaging, which are optional dependencies. Install them with the pypi extra:

pip install pyTooling[pypi]

Without it, importing pyTooling.Dependency.Python raises an exception naming the extra. The generic data model in pyTooling.Dependency has no such requirement.

Resolving Requirements

Resolver turns requirements into releases. For each requirement it selects the newest release matching the specifier, downloads that release’s details and follows its requirements. A requirement naming extras (pyTooling[yaml]) brings in those extras’ requirements as well, and their requirements in turn - an extra can pull in a whole subtree, with licenses of its own. It takes its requirements from

a list of Requirement objects (ResolveRequirements()),
a requirements file read by RequirementsFile.Parse (ResolveRequirementsFile()), or
the newest releases of packages the pre-release policy admits, optionally one of their extras (ResolvePackages()).

The result is a Resolution, a tree of ResolvedRequirement objects. Each one carries the selected release, or the reason there is none as its ResolutionStatus: the index doesn’t know the project, no release matches, the index can’t describe the selected release, the index failed to answer, the environment marker doesn’t apply, the project closes a cycle, or the requirement is below the limit. The selected releases are connected in the index’s dependency graph, too, so ToGraph() shows them.

The resolver is configured by four parameters:

strategy: Lazy (the default) resolves a release’s requirements when its Dependencies are first asked for; Eager resolves everything up front, one level at a time. Either way, the downloads of one level run in parallel.
limit: The number of levels resolved below the given requirements; 0 resolves only the given requirements, None (the default) everything.
preReleases: Exclude (the default) selects a pre-release only if the requirement names one (pyTooling >= 10.0.0rc3) or if no final release matches - what an installer would pick (PEP 440); Include selects pre-releases like final releases.
environment: The environment markers are evaluated against. None (the default) evaluates no marker, so every requirement is resolved. CurrentEnvironmentResolver uses the running interpreter’s environment.

A project or release the index fails to answer for - a server error (5xx) or a failed connection - gets the status IndexFailure and is reported as a critical PackageIndexFailureWarning: the resolution continues, but a WarningCollector has to collect the warning, otherwise it raises.

from packaging.requirements import Requirement

from pyTooling.Dependency.Python   import PythonPackageDependencyGraph, PythonPackageIndex
from pyTooling.Dependency.Resolver import CurrentEnvironmentResolver, ResolutionStatus
from pyTooling.Warning             import WarningCollector

graph = PythonPackageDependencyGraph("documentation")
index = PythonPackageIndex("PyPI", "https://pypi.org/", "https://pypi.org/pypi/", graph)

with WarningCollector() as warnings:
  resolution = CurrentEnvironmentResolver(index, limit=2).ResolveRequirements([Requirement("pyTooling[yaml]")])

for requirement in resolution.Requirements:
  if requirement.Status is ResolutionStatus.Resolved:
    print(f"{requirement.Requirement} -> {requirement.Release.Version}")

Attention

It is no installer’s solver: every requirement is resolved on its own, and two requirements of one project with conflicting specifiers select two releases. Nothing is backtracked.

A resolution can be limited in time with a Timeout around it (see Timeout). Each level checks the timeout before it starts, and the downloads of a level get the time left as their timeout. Requirements not resolved when it expires get the status TimeoutExpired and are reported, once per level, as a critical ResolutionTimeoutWarning. The resolution returned is partial, not lost: what was resolved before stays resolved. Checks needing no download - marker, cycle, limit - still apply. With the strategy Lazy, a level resolved later uses the timeout current then.

from pyTooling.Dependency.Resolver import ResolutionStrategy
from pyTooling.Timeout             import Timeout

with WarningCollector() as warnings, Timeout(20.0):
  resolver =   CurrentEnvironmentResolver(index, strategy=ResolutionStrategy.Eager)
  resolution = resolver.ResolveRequirements([Requirement("pyTooling[yaml]")])

Release.PostProcess (behind DependsOn) stays what it was: it attaches every release matching a requirement, not the one a resolver selects.

Exceptions and Warnings

DependencyError is the base of the module’s exceptions: NoSessionAvailableError when a query is attempted without an open session, ProjectNotFoundError and ReleaseNotFoundError when the index does not know what was asked for, and PackageIndexError when the index responds with another HTTP error status. The latter is caused by the HTTPError. ExtraNotFoundError is raised when the requirements of an extra are asked for, which a release doesn’t declare.

A malformed requirement or unreadable release metadata does not abort the traversal - it is reported as a BrokenRequirementWarning or ReleaseDetailsWarning, because one bad package should not hide the rest of the graph. A license that can’t be resolved is reported as an UnknownLicenseWarning. A requirement a Resolver can’t resolve, because the index doesn’t know its project or can’t describe the selected release, is reported as an UnresolvedRequirementWarning; it stays in the resolution, with the reason as its status.

These warnings are non-critical: they are reported to a WarningCollector, and without a collector up the call-hierarchy, they are dropped silently. A caller collects them to learn what was left out, see Warnings.

A release whose version number the parser doesn’t support, like pytz’ 2004d from before PEP 440, is skipped and reported as an UnsupportedVersionWarning. It is a critical warning: without a collector up the call-hierarchy, UnhandledCriticalWarningError is raised.

A resolver whose Timeout expired reports the requirements it left unresolved as a critical ResolutionTimeoutWarning, with the configured seconds and the projects’ names.

Competing Solutions

pyTooling.Dependency is a library: a data model of packages, their versions and dependencies, read from PyPI’s JSON API without installing anything, with the license of every release and hand-written LicenseOverrides for packages whose metadata states none. The tools below are command-line tools first, or parse a requirements file only.

pipdeptree

Source: pipdeptree, on PyPI as pipdeptree.

Disadvantages

A command-line tool: a program receives its tree as text, JSON, Mermaid or Graphviz, not as objects.

Advantages

Shows the installed environment, and reports conflicting requirements and cycles.
from-index resolves a package or a requirements file against an index without installing it, and from-lock reads a resolved PEP 751 lock file.
--summary reports package counts, depth, conflicts, cycles, licenses and size.

pipgrip

Source: pipgrip, on PyPI as pipgrip.

Disadvantages

A command-line tool: its result is a list of pins or a tree, as text or JSON.

Standoff

Both select the latest versions satisfying every constraint: pipgrip with the PubGrub algorithm, which poetry uses too, PackageVersion.SolveLatest by backtracking over the versions in the graph.

Advantages

Installs the resolved tree, and combines the trees of several packages into one set of pins.

johnnydep

Source: johnnydep, on PyPI as johnnydep.

Disadvantages

A command-line tool printing a tree of names and summaries.

Standoff

Both read a package’s dependencies from the index, not from the installed environment.

Advantages

Resolves a package’s tree into pinned versions (--output-format pinned).

requirements-parser

Source: requirements-parser, on PyPI as requirements-parser.

Disadvantages

Options traversing the local file system are not handled, so a -r reference to another file isn’t followed. RequirementsFile.Parse reads the referenced files as a tree, keeps which file a requirement was stated in, and raises a CircularRequirementsFileError for a cycle. A file two files reference is read once; the second -r line is a RequirementsFileReference to it.

Advantages

Parses editables, version control URIs, hashes and URLs, which RequirementsFile.Parse skips as instructions to the installer.