Pipeline Run

pyTooling.GitLab models a GitLab CI/CD pipeline, as GitLab’s REST API reports it:

from pyTooling.GitLab import Pipeline

pipeline = Pipeline.Parse(pipeline, jobs, bridges)  # the REST API's payloads

print(f"{pipeline.Name}: {pipeline.Status} in {pipeline.Duration} s")
for job in pipeline.IterateJobs():
  print(f"  {job.QualifiedName:<30} {job.Stage:<10} queued {job.QueuedDuration} s, ran {job.Duration} s")

The Tree

  • PipelineGroupevery pipeline started for one commit
    • Pipelinea pipeline
      • Matrixparallel or parallel:matrix of a job
        • MatrixJobone instance it produced
        • MatrixDownstreamPipelineone instance of a trigger job
      • Joba job that ran on a runner
      • DownstreamPipelinea trigger job and the pipeline it started
        • Matrixthe same elements a pipeline contains
        • Job
        • DownstreamPipeline

The tree is pyTooling.CI’s (see pyTooling’s pipeline model, which maps GitLab CI onto it): a matrix is that model’s Matrix, which this module imports, and the pipeline group, the pipeline, a job, a matrix instance and a downstream pipeline derive from that model’s classes. StatusMixin adds what GitLab reports about a pipeline, a downstream pipeline and a job - the Status, the failure reason and the URL - and the status is reported as the model’s Outcome too (Status.ToOutcome). JobMixin adds what GitLab reports about a job and a trigger job: its identifier, its stage, whether it may fail, and how long it was queued.

Every element knows its Parent, and holds a reference to the pipeline it belongs to in Pipeline - a job of a child pipeline the pipeline at the top, which started the child.

IterateElements() yields what a group contains one level down: a pipeline its jobs, its matrices and its downstream pipelines, a matrix its instances. To reach every job below a pipeline at once - those of its matrices and of its child pipelines included - use IterateJobs():

for element in pipeline.IterateElements():  # one level: jobs, matrices, downstream pipelines
  print(f"{type(element).__name__}: {element}")

for job in pipeline.IterateJobs():          # every job below the pipeline, at any depth
  print(job.QualifiedName)

An element is placed in its group under its own name - a downstream pipeline and a matrix as that key of Workflows respectively Matrices, a job by its name - so an element is asked for by that name:

pipeline.ContainsElement("build")              # a job, a matrix or a downstream pipeline of the pipeline
matrix.ContainsElement("rspec: [3.4]")         # an instance carries the values telling it from its siblings
pipeline.Workflows["components"].Jobs          # the jobs of a child pipeline

  • A matrix is not an element GitLab reports. It names an instance after the job and the instance’s values - rspec: [3.4] for parallel:matrix, integration test 2/4 for parallel: 4 - and Parse() reads the name back into the tree: a MatrixJob below a Matrix named rspec.

  • A matrix instance’s dimensions are named by position. A job’s name carries the values, not the variables’ names - only the pipeline file names them. So Dimensions of test: [3.14, linux] is {"0": "3.14", "1": "linux"}. An instance of parallel: N carries the variables GitLab sets for it: {"CI_NODE_INDEX": "2", "CI_NODE_TOTAL": "4"}.

  • The suffix is how GitLab names a job rather than a field, so a job genuinely named deploy 1/2 by its pipeline file becomes a matrix of one instance.

  • A group iterates what it holds in the order it was created. GitLab lists the newest job first; Parse() adds the jobs by their identifier, which grows with a job’s creation.

  • A pipeline without a name - GitLab reports one only for a pipeline file setting workflow:name - is named by its number within the project, #47177, as GitLab’s interface shows it.

  • A job has no steps. GitLab reports a job as a whole; its script’s sections exist in the job’s log only.

  • QueuedDuration is the time GitLab reports a job was pending for a runner. The time a job waited for the jobs it depends on lies between CreatedAt and StartedAt as well, and isn’t counted.

Downstream Pipelines

A trigger job - GitLab calls it a bridge and lists it apart from the jobs - starts a downstream pipeline: a child pipeline of the same project (trigger:include) or a pipeline of another project (trigger:project). It becomes a DownstreamPipeline, a Workflow named after the trigger job:

pipeline = Pipeline.Parse(pipeline, jobs, bridges, downstream)

child = pipeline.Workflows["components"]
print(child.DownstreamID, child.DownstreamStatus, child.IsMultiProject)
for job in child.Jobs:
  print(job.QualifiedName)                     # components / component:redis:test

  • It reports the downstream pipeline’s own Status, URL and times, and identifies it: DownstreamID, DownstreamProjectID. What GitLab reports about the trigger job is kept beside: ID, Stage, TriggerStatus and TriggerURL.

  • The downstream pipeline’s contents are its jobs, if its payloads are given in parameter downstream: a mapping of the downstream pipeline’s identifier to a JSON object with pipeline, jobs, bridges and its own downstream - the shape of Parse()’s parameters, so a recorded pipeline is read with Pipeline.Parse(**recording). Its status, URL and times are those of its pipeline payload.

  • A downstream pipeline whose payloads weren’t read reports what the trigger job’s downstream_pipeline summarizes: its status, URL and creation, and - once it ended - its last update as its completion. GitLab reports no start there.

  • A multi-project pipeline names its project as Reference, e.g. gitlab-org/ci-cd/tests/runner-incept; its contents aren’t read, as they belong to another project. A child pipeline has no reference: GitLab doesn’t report the file the trigger job included.

  • A trigger job’s status and its downstream pipeline’s status differ for a trigger job without strategy: depend: it succeeds as soon as it started the downstream pipeline, whatever the downstream pipeline does later. A trigger job waiting to be started by hand hasn’t started a pipeline: the downstream pipeline has no status and no times yet.

  • A trigger job with parallel:matrix becomes a MatrixDownstreamPipeline below a Matrix, named like a matrix job: deploy: [staging].

Stages and Needs

The REST API reports no dependencies. A job reports its Stage, but GitLab reports neither the order of the stages nor a job’s needs: - both are in the pipeline file. ApplyNeeds() adds the dependencies from them, as pyTooling’s mapping of GitLab CI describes it:

pipeline.ApplyNeeds(
  ["build", "test", "deploy"],                  # the pipeline file's 'stages:'
  {"unit": ["compile"], "release": []}          # the 'needs:' of the jobs having them
)
pipeline.Validate()

  • An element named in the second parameter needs the siblings named there - a matrix or a downstream pipeline by its name -, and [] needs nothing.

  • Every other element needs every element of the nearest earlier stage holding elements, so a stage without jobs is skipped.

  • A downstream pipeline’s contents have stages of their own: apply them to the downstream pipeline.

Strings Become Enumerations

status and source arrive as text. Status and Source turn them into members, so a value GitLab doesn’t document raises GitLabError instead of quietly matching no comparison:

if pipeline.Source is Source.MergeRequestEvent:   # not: pipeline["source"] == "merge_request_event"
  ...

A status ending a pipeline or a job is an Outcome too (STATUS_TO_OUTCOME), and a job failed for a timeout - failure reason job_execution_timeout or stuck_or_timeout_failure - has the outcome Timeout. A job waiting to be started by hand (Manual) hasn’t ended, so it has no outcome, and neither has a matrix of such jobs.

Timestamps are parsed once, and a timestamp without a time zone is read as UTC, so every time of a pipeline can be compared with every other.

Several Pipelines per Commit

A commit can have several pipelines: one for the push of a branch, one for a merge request proposing it, one at a tag of the same commit, a scheduled one. PipelineGroup holds them. GitLab lists a project’s pipelines without its child pipelines, which belong to the pipeline that triggered them.

GitReference is a branch’s or a tag’s name, or a merge request’s reference, and IsTag tells a tag from a branch. ByGitReference() separates the pipelines by it:

group = PipelineGroup.Parse(pipelines)        # GET /projects/{id}/pipelines?sha=...

for reference, pipelines in group.ByGitReference().items():
  print(f"{reference}: {len(pipelines)} pipeline(s)")

# main: 1 pipeline(s)
# refs/merge-requests/6293/merge: 1 pipeline(s)