Pipeline File

pyTooling.GitLab.PipelineFile models a GitLab CI/CD configuration file - the project’s .gitlab-ci.yml, a file it includes, or the file of a child pipeline -, not a run of it (that’s Pipeline Run):

from pathlib import Path
from pyTooling.GitLab.PipelineFile import PipelineFile

pipelineFile = PipelineFile.FromFile(Path(".gitlab-ci.yml"))

print(f"{pipelineFile}: {pipelineFile.JobCount} jobs in stages {', '.join(pipelineFile.Stages)}")
for job in pipelineFile.IterateJobs():
  print(f"  {job.Name:<24} {job.Stage:<10} {job.Image or ''}")

The file is read with ruamel.yaml, a requirement of this package.

The Tree

PipelineFilea configuration file
- Inputan input of the header’s spec:inputs
- Workflowthe workflow key
  - Rulea rule deciding whether a pipeline is created
- Includean entry of include
  - Rulea rule deciding whether the file is included
- Defaultthe default key
- Variablea default variable, declared in the top-level variables
- Joba job or a hidden job, in file order
  - Needan entry of the job’s needs
    
    Parallelthe matrix instances needed
  - Rulea rule deciding whether the job is added to a pipeline
  - Variablea variable of the job
  - Inputan input of the job
  - Parallelthe job’s parallel: how often it runs, or its matrix
  - Triggerthe child pipeline or other project’s pipeline it starts
    
    Includea file of the child pipeline

Every element knows its Parent, and holds a reference to the configuration file it belongs to in PipelineFile.

A file may begin with a header document, separated from the configuration by ---, which holds the spec key - see Inputs.

Stages

DeclaredStages holds the stages key as written, or None. Stages holds the stages the pipeline runs, in their order: the declared ones, or build, test and deploy (DEFAULT_STAGES) for a file without stages, always between .pre and .post:

stages:              # Stages: .pre, lint, build, .post
  - lint
  - build

Variables

A variable of the top-level variables is a Variable. Its Value is the text GitLab reads - a number or a boolean written in the file is its text, as 3.14 or true. A variable declared as mapping may also carry:

Description - GitLab prefills the variable in the form running a pipeline by hand, with the description beside it;
Options - the values that form offers to select from;
Expand - False, if variables in the value aren’t expanded.

variables:
  DEPLOY_ENVIRONMENT:
    value: "staging"
    description: "The deployment target."
    options: ["production", "staging"]

Jobs

A top-level key is a Job, unless it is a global keyword (RESERVED_KEYS). A job whose name starts with a dot is a hidden job: GitLab doesn’t run it, other jobs extend it. The file keeps both apart, in file order:

pipelineFile.Jobs                       # the jobs, by name
pipelineFile.HiddenJobs                 # the hidden jobs, by name
pipelineFile.ContainsJob(".template")   # False: a hidden job isn't a job

A job runs in its Stage - test, if it names none -, which FromFile() checks against the pipeline’s stages. The keys a job’s documentation needs are read too:

Script, BeforeScript and AfterScript - the lines, a single line as a list of one, nested lists flattened;
When - a RuleOutcome;
AllowFailure and AllowedExitCodes - allow_failure as a boolean, or as a mapping of exit_codes;
Image, Services and Environment - the names, written as a string or as the name of a mapping; Tags;
Variables - the job’s variables.

Dependencies Between Jobs

A job runs after the jobs of the earlier stages, unless it has a needs key: then it runs as soon as the jobs it needs have run, and needs: [] lets it start with the pipeline. Needs tells the three apart - None, the entries, or an empty tuple. Each entry is a Need, whose Kind says what it refers to:

`needs` entry	`NeedKind`
`build` or `job: build`	`Job` - a job of the same pipeline
`pipeline: $PARENT_PIPELINE_ID` and `job`	`PipelineJob` - a job of another pipeline of the parent-child hierarchy
`project`, `job` and `ref`	`ProjectJob` - a job of another project’s pipeline, whose artifacts are downloaded
`pipeline: other/project` alone	`PipelineStatus` - another project’s pipeline, whose status the job mirrors

Artifacts and IsOptional hold artifacts and optional. NeededJobs resolves the needs of the same pipeline to the jobs of the file:

for job in pipelineFile.IterateJobs():
  if job.Needs is not None:
    print(f"{job.Name} needs {', '.join(needed.Name for needed in job.NeededJobs)}")

FromFile() checks that a job needs only jobs of the file - an optional need may name a missing job - and that the jobs don’t need each other in a cycle.

Rules and Workflow

A job’s rules decide whether it is added to a pipeline, the rules of the workflow key whether a pipeline is created. GitLab evaluates them in order, and the first matching one decides; the model keeps them as written, without evaluating them. A Rule holds:

Condition - the expression of if;
Changes and CompareTo - the paths of changes, written as a list or as changes.paths, and changes.compare_to;
Exists - the paths of exists, as a list or as exists.paths;
When, AllowFailure and Variables - what applies, if the rule matches.

PipelineFile.Workflow holds the Workflow - the pipeline’s Name and its Rules -, or None for a file without workflow:

workflow:
  name: "Pipeline for $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_COMMIT_TAG
      when: never
    - when: always

Includes and Defaults

Each entry of include is an Include, whose Kind says where the files come from and whose Location names them, as written:

`include` entry	`IncludeKind`, `Location`
`/ci/build.yml` or `local: /ci/build.yml`	`Local`, the path
`https://example.com/ci.yml` or `remote: ...`	`Remote`, the URL
`project: group/shared`, `file`, `ref`	`Project`, the project’s path; `Files` and `Ref`
`template: Auto-DevOps.gitlab-ci.yml`	`Template`, the template’s name
`component: $CI_SERVER_FQDN/group/components/lint@1.2`	`Component`, the component’s reference

An include’s Inputs keep the types they are written with, and its Rules decide whether the files are included. The included files are read only to resolve extends and !reference tags, with an IncludeResolver - see Extends, References and Defaults.

PipelineFile.Default holds the default key as a Default - image, services, tags and the scripts around a job’s script. The global keywords image, services, before_script and after_script, deprecated forms of default, are read into it too, unless default sets them.

Parallel Jobs and Triggers

A job’s parallel is a Parallel: a Count, or a Matrix whose entries map variables to values. Combinations holds one combination per instance, each entry contributing the product of its lists, and InstanceNames the names GitLab gives the instances:

deploy:                      # deploy: [aws, monitoring], deploy: [aws, app1], deploy: [gcp, data]
  parallel:
    matrix:
      - PROVIDER: aws
        STACK: [monitoring, app1]
      - PROVIDER: gcp
        STACK: data
test:                        # test 1/3, test 2/3, test 3/3
  parallel: 3

A need’s Parallel names the matrix instances it needs, written as needs:parallel:matrix.

A job’s trigger is a Trigger. A trigger including files starts a child pipeline - Includes holds the files as Includes, also the artifact a job created for a dynamic child pipeline. A trigger naming a Project - a string, or project of a mapping, with an optional Branch - starts that project’s pipeline. Strategy says whether the job waits for the pipeline.

Extends, References and Defaults

A job’s keys are merged as GitLab merges them, before the job is built:

!reference tags are replaced by what they refer to - in a list, a referred list is spliced in, as in script: [!reference [.setup, script], make]. References nest up to 10 levels (MAX_REFERENCE_DEPTH).
extends merges the jobs a job extends, in order, and then its own keys: mappings merge deeply, other values - lists included - replace, and null removes a key. Jobs extend each other up to 11 levels (MAX_EXTENDS_DEPTH). Extends keeps the names.
A job gets every key of default, and of the global keywords standing for it, which it doesn’t set - unless inherit.default is false, or lists the keys it inherits. Hidden jobs get none.

Every key keeps the line it is written at, also when another job contributes it.

A file may extend, or refer to, jobs of the files it includes. Read with an IncludeResolver, the local files it includes - and those they include - are read too, and their jobs and keys resolve; a key an included file contributes gets the line of the extends key or the reference bringing it in. The included jobs aren’t jobs of the file:

pipelineFile = PipelineFile.FromFile(Path(".gitlab-ci.yml"), IncludeResolver(Path(".")))

A missing job or key is an error, unless the file includes files that weren’t read - another project’s files, a URL, a template, a component, or a local file without a resolver. Then the job is built from what was read, and Unresolved names what it lacks, as extends .template or !reference [.setup, script].

Inputs

A CI/CD component’s template, or a file included with inputs, declares its parameters in the header document’s spec:inputs. PipelineFile.Inputs holds them as Inputs, and PipelineFile.Description holds spec:description:

spec:
  description: "Lints a Python package."
  inputs:
    stage:
      default: test
      description: "Stage of the job."
    python:
      options: ["3.13", "3.14"]
      default: "3.14"
    version:
      regex: ^v\d+$
---
lint:
  stage: $[[ inputs.stage ]]

An input has a Type - string, if it doesn’t say -, a Default with the type it is written with, a Description, Options, a Regex and conditional Rules. It is Required without a default key. FromFile() checks that a default matches the type and the options.

A job’s inputs - set when the job is run or retried by hand - are inputs too, in Job.Inputs; each needs a default.

A value interpolating an input, as $[[ inputs.stage ]], is kept as written: it is known when the file is included only, so a stage or a needed job written that way isn’t checked.

Building a Pipeline

ToPipeline() builds the pipeline a file defines as a pyTooling.CI model, as pyTooling maps GitLab CI onto it. Each element links back to the job it was built from by Definition:

Job	Element
a job running a script	`DefinedJob`
a job with `parallel`	`DefinedMatrix`, a `DefinedMatrixJob` per instance, whose dimensions are the matrix’ variables, or `CI_NODE_INDEX` for `parallel: N`, named as GitLab names it: `test: [3.14, linux]`, `test 1/3`
a trigger job	`DefinedWorkflow`, whose reference is the other project or the child pipeline’s files; with `parallel`, a `DefinedMatrixWorkflow` per instance
a hidden job	none

The pipeline is named by workflow:name, or else by the file’s name. A job without needs needs every element of the nearest earlier stage holding elements; needs replaces that, and needs: [] needs nothing. A need of some instances of a matrix needs the whole matrix.

Each rule of a job becomes a Rule of the element, in the order GitLab evaluates them: its condition is the rule’s if, its When the rule’s when, else the job’s, else on_success. changes, compare_to, exists, allow_failure and variables are the rule’s key-value-pairs. A job with when, but without rules, gets one rule without condition; workflow:rules are the pipeline’s rules, deciding always unless they say never. The result converts into a graph:

from pathlib import Path
from pyTooling.GitLab.PipelineFile import IncludeResolver, PipelineFile

pipelineFile = PipelineFile.FromFile(Path(".gitlab-ci.yml"))
pipeline =     pipelineFile.ToPipeline(IncludeResolver(Path(".")))
graph =        pipeline.ToGraph()

An IncludeResolver reads a trigger job’s child pipeline from the repository’s directory, if its files are local; the workflow then holds the child pipeline’s elements, and ChildFiles the files. depth limits how many levels of child pipelines are expanded.

Source Lines

Every element keeps the Line it is written at, starting at 1, so a consumer can point at the place a finding comes from. A malformed file raises PipelineFileError, which carries the file in Path and the line in Line, and names both in a note:

pyTooling.GitLab.PipelineFile.PipelineFileError: Variable 'A' has unknown key 'secret'.
  In '.gitlab-ci.yml:4'.