GitHub Actions Workflow Files

pyTooling.CI.GitHub.WorkflowFile models a GitHub Actions workflow file - the YAML file below .github/workflows, not a run of it (that’s GitHub Actions):

from pathlib import Path
from pyTooling.CI.GitHub.WorkflowFile import Workflow

workflow = Workflow.FromFile(Path(".github/workflows/CompletePipeline.yml"))

print(f"{workflow.Name}: {len(workflow.Inputs)} inputs, {workflow.JobCount} jobs")
for job in workflow.IterateJobs():
  print(f"  {job.Name:<24} {job.Uses or job.RunsOn}  needs {', '.join(job.NeedNames)}")

The file is read with ruamel.yaml, so the module needs the github extra - see Dependencies.

The Tree

Workflow                 a workflow file
+-- Input                an input of 'on.workflow_call'
+-- Output               an output of 'on.workflow_call'
+-- Secret               a secret of 'on.workflow_call'
+-- Permission           a permission the workflow declares
+-- Job                  a job, in file order
    +-- UsesReference    the reusable workflow the job calls
    +-- Permission       a permission the job declares
    +-- Matrix           the job's 'strategy.matrix'
    +-- Step             a step of the job
        +-- UsesReference    the action the step runs

Action                   an action's file, 'action.yml'
+-- Step                 a step of a composite action
    +-- UsesReference    the action the step runs

  • A workflow is named by its file’s stem - CompletePipeline - because a caller names it that way in uses. The name key is DisplayName.

  • An input keeps the type its default is written with - '3.14' is a string, false a boolean - and a multi-line default keeps its line breaks.

  • A job either runs steps on RunsOn, or calls the reusable workflow in Uses. UsesReference takes a reference apart into Repository, Path and Reference:

    pyTooling/Actions/.github/workflows/Package.yml@r8   repository, path and ref
./.github/workflows/Package.yml                       a file of the same repository and commit

  • Matrix.IsDynamic says whether a matrix’ instances are known at run time only, as for include: ${{ fromJson(inputs.jobs) }}.

  • A job’s Container and Services are the images of the containers it runs in.

  • An action is named by its directory - ComputeRequirements for .github/actions/ComputeRequirements/action.yml. Of a composite action, the steps are read; of a Docker action, the Image.

  • Expressions - if, runs-on: ${{ matrix.runs-on }}, an output’s value - are kept as written and are not evaluated.

Source Lines

Every element knows the file it was read from - a workflow’s or an action’s - and the line it starts at, so a message can say where a finding comes from:

job = workflow.Jobs["PublishOnPyPI"]
print(f"{job.Location}: job '{job.Name}' has a condition")   # CompletePipeline.yml:532: ...

A file that is not a well-formed workflow - a job needing a job the workflow doesn’t have, jobs needing each other in a cycle, an input without type - raises WorkflowError. It carries the file and the line in Path and Line, and names both in a note.

Dependencies Between Jobs

Job.Needs resolves the names of needs to the jobs. The pipeline they form is built by ToPipeline() (see Building a Pipeline) and converted into a Graph by ToGraph(), whose edges read needs, and which by default drops a dependency a longer path already implies:

Package  needs Prepare               Local --> Package --> Prepare
Local    needs Prepare, Package
                                     (Local --> Prepare is implied)

Called Workflows

A job calling a reusable workflow names it by repository, path and ref. WorkflowResolver reads the called file from a local directory, mapped per repository, whatever the ref:

from pyTooling.CI.GitHub.WorkflowFile import WorkflowResolver

resolver = WorkflowResolver({"pyTooling/Actions": Path(".github/workflows")})
pipeline = resolver.Load(Path(".github/workflows/CompletePipeline.yml"))

for job in pipeline.IterateJobs():
  if job.Uses is not None and (called := resolver.Resolve(job.Uses)) is not None:
    print(f"{job.Name} calls {called.Name} with {called.JobCount} jobs")

  • A local reference - ./.github/workflows/Package.yml - is read from the directory of the calling workflow.

  • A repository without a directory answers None: its files are not fetched.

  • Every file is read once; resolving it again returns the same Workflow.

ResolveAction() reads the action a step runs the same way, from its action.yml, so the actions a composite action runs in turn are known:

  • An action of a mapped repository - pyTooling/Actions/.github/actions/ComputeRequirements@r8 - is read from the repository’s root, the directory holding the .github directory the mapped directory is in.

  • A local action - ./.github/actions/ComputeRequirements - is read from the root of the repository of the calling workflow or action.

Permissions

A called workflow can keep or reduce the permissions of the GITHUB_TOKEN, never raise them, so what its jobs declare is what a caller has to grant. CollectPermissions() collects the permissions a workflow and its jobs declare, and - given a resolver - those of the workflows they call:

for scope, permission in pipeline.CollectPermissions(resolver).items():
  print(f"{permission}   asked for at {permission.Location}")

# contents: write   asked for at CompletePipeline.yml:500
# pages: write      asked for at PublishToGitHubPages.yml:67

When several elements declare a scope, the permission granting the most access is returned, so its location names where that access is asked for.

Building a Pipeline

ToPipeline() builds the pipeline a workflow defines as a pyTooling.CI model (Pipeline), expanding the workflows its jobs call as far as a resolver reads them and depth allows:

pipeline = workflow.ToPipeline(resolver, depth=1)
graph =    pipeline.ToGraph()                       # transitively reduced

for vertex in graph.IterateTopologically():         # in an order the jobs can run in
  element = vertex.Value
  print(f"{element.QualifiedName}  {element.Definition.Location}")

Job in the workflow file

Element of the pipeline

with steps

DefinedJob, its steps as DefinedStep

with uses

DefinedWorkflow, the uses text as Reference, holding the elements of the called workflow - or none, if the call isn’t expanded

with strategy.matrix

DefinedMatrix, holding a DefinedMatrixJob - or a DefinedMatrixWorkflow for a job with uses - per combination

needs

Needs between the elements

if

Condition

  • An element is named by its job’s key, as the file names it in needs. A step is named as GitHub displays it: by its name, or Run followed by its action or the first line of its script.

  • Every element links to what it was built from: Definition is the Job - with its line, its uses reference and its permissions - or the Step, and for the pipeline the Workflow. A called workflow’s CalledWorkflow is the file it was expanded from.

  • A matrix yields its combinations as GitHub computes them from its dimensions, exclude and include - Matrix.Combinations -, and an instance is named by its values: Test (ubuntu, 3.14). Its Dimensions are the combination, the values formatted as GitHub prints them: {"os": "ubuntu", "python": "3.14"}. A dynamic matrix - include: ${{ fromJson(...) }} - is a DefinedMatrix without instances, since its combinations are known at run time only.

  • A workflow calling itself, directly or through others, raises WorkflowError.

Linking a Run

A run read from the GitHub REST API (GitHub Actions) has no needs: the API doesn’t report them. ApplyNeeds() gives a run the dependencies its workflow file - named by Pipeline.Path - declares:

run =      Pipeline.FromJSON(runJSON, jobsJSON)
workflow = resolver.Load(Path(run.Path))

for name in workflow.ApplyNeeds(run, resolver):
  print(f"Job '{name}' isn't in the run.")

graph = run.ToGraph()

  • A job is looked up in the run by its display name - its name -, or else by its key: group.GetElement(name). A matrix is found as the Matrix its instances were grouped into.

  • A job calling a reusable workflow is followed into the run’s called workflow, and a matrix of calls into each instance, as far as the resolver reads the called file.

  • A run names an instance’s dimensions by position - {"0": "ubuntu", "1": "3.14"} -, since a job’s name carries the values only. For a static matrix, an instance whose values are those of a combination gets the combination’s names: {"os": "ubuntu", "python": "3.14"}. The instances of a dynamic matrix, and one matching no combination, keep the positions.

  • A job named by an expression - ${{ matrix.os }} Tests - can’t be looked up and is skipped. A job with a condition may have been skipped in the run, so it isn’t reported when it’s missing. Every other job missing in the run is returned by its qualified name, as the run would name it: Local / Static for the job Static of the workflow the job Local calls.

Competing Solutions

No package on PyPI reads a workflow file into a Python object model. The packages below check a workflow file, or write one.

Outside Python, zizmor has typed models of workflows, actions and Dependabot files - the Rust crate github-actions-models - and checks a workflow for security problems, as a template injection or an unpinned action. actionlint, written in Go, checks a file’s syntax and expressions, the needs of its jobs and the inputs of the reusable workflows it calls. Both are installed from PyPI as command line tools, so a Python program gets their findings, not a model.

JSON Schema

Source: the schemas of SchemaStore, checked by check-jsonschema.

Disadvantages

  • A file is validated against the schema. A program reading it gets nested dictionaries and lists.

  • Classes generated from the schema - e.g. by datamodel-code-generator - are typed, but know neither the line of an element nor its parent, and don’t check the needs of a job.

Advantages

  • The schema is used by editors too, so a file is checked the same way while it is written.

Workflow Generators

Source: github-actions-cdk, pygha.

Disadvantages

  • They write a workflow file from Python code, but don’t read one.