Overview

pyTooling.Tracing records a software execution trace: a tree of timespans, each with its own duration, attributes and events, built with with-statements as a program runs.

from pyTooling.Tracing import Trace, Span, Event

with Trace("build") as trace:
  trace["version"] = "10.0.0"

  with Span("compile") as compile:
    compile["files"] = 12
    Event("cache miss", parent=compile)

  with Span("link"):
    ...

print("\n".join(trace.Format()))

A Trace is the root; every Span inside it attaches to whichever span is active on the current thread, so the tree follows the program’s structure without being wired up by hand. An Event is a point in time rather than a span, and names its span explicitly.

Format() renders the tree as indented lines for a terminal. For anything else, the trace is exported.

Recorded Timespans

A timespan that was measured elsewhere - by a CI service, or read from a log file - already has its times. It is constructed with them, and attached to its parent by the parent parameter instead of a with-statement:

from datetime import datetime, timezone
from pyTooling.Tracing import Trace, Span

trace = Trace("Pipeline", beginTime=datetime(2026, 9, 15, 6, 35, 21, tzinfo=timezone.utc),
                          endTime=datetime(2026, 9, 15, 6, 44, 30, tzinfo=timezone.utc))
job =   Span("UnitTesting", parent=trace, beginTime=datetime(2026, 9, 15, 6, 35, 32, tzinfo=timezone.utc),
                                          endTime=datetime(2026, 9, 15, 6, 37, 10, tzinfo=timezone.utc))
job["runner"] = "ubuntu-26.04"

print(job.Duration)   # 98.0

endTime requires beginTime, can’t precede it, and both are either time zone aware or naive.
A source reporting a length instead of an end gives duration in place of endTime, as a timedelta or as a number of seconds - int for whole, float for fractional seconds, the unit Duration reports. It is converted to endTime, so every form is stored alike. Giving both endTime and duration raises an exception.
A sub-timespan attached with parent has to lie within its parent’s range. Only the direct parent is checked, because containment is transitive.
A timespan with a beginTime but no endTime is still running: its Duration is the time since its recorded begin. Assigning StopTime reports an end that is already known, and Stop() ends a timespan that is still running now. Either accepts the end exactly once.
State tells which times are filled in - Empty, Running or Complete - regardless of whether they were measured or recorded. Only an empty timespan can be entered, so a timespan can’t be timed twice.
Without beginTime, a timespan is timed by its with-statement. A timespan constructed with recorded times can’t be entered - that raises a TracingError.

OTLP/JSON Export

A Trace converts itself to OTLP/JSON, the OpenTelemetry Protocol’s JSON encoding. One format reaches both usual destinations: an OpenTelemetry collector accepts OTLP natively, and Jaeger has accepted it since v1.35 - so no translation step stands between a trace and a viewer.

from pathlib import Path

trace.WriteJSONFile(Path("trace.json"), serviceName="myProgram")

curl -X POST -H "Content-Type: application/json" -d @trace.json http://localhost:4318/v1/traces

Three methods, for the three things a caller does with the document:

Method	Returns
`ToJSON()`	the document as an `OTLPDocument`, for a caller posting it directly
`ToJSONString()`	the document encoded as a `str`
`WriteJSONFile()`	nothing - it writes the document to the given `Path`

All three take scopeName and scopeVersion, which name the instrumentation scope - the library the spans are reported as coming from. They default to OTLP_SCOPE_NAME and pyTooling’s version, so a program that wraps this tracing in its own API reports itself by passing them rather than by patching the module.

A Span and an Event convert themselves too, but not publicly: a lone span is no OTLP document, because it has no service to be reported under. Each level returns its own part - Span._ToOTLPJSON() returns itself and everything below it, flattened - and the trace wraps the result in the document envelope.

The document is not an untyped mapping: every level of it is a TypedDict named after the OTLP message it encodes, from OTLPDocument down to OTLPAnyValue. A caller can annotate what it received, and a typo in a key is a typing error rather than a document a collector silently rejects.

How a trace is mapped

pyTooling	OTLP
the trace	one `resourceSpans` entry, whose `service.name` attribute is `serviceName` - or the trace’s name
the tree of spans	a flat list, whose `parentSpanId` references carry the hierarchy
`TraceID`, drawn when the trace is constructed	`traceId` on every span of the trace
`SpanID`, drawn when the timespan is constructed	`spanId`, and the `parentSpanId` of everything below it
`StartTime` and `Duration`	`startTimeUnixNano` and `endTimeUnixNano`
a span’s attributes	`attributes`, each value wrapped by its type
a span’s events	`events`

Three details of the encoding are easy to get wrong, and each has a testcase:

Identifiers are hex, not base64. OTLP/JSON deviates from proto3’s JSON mapping for traceId (16 bytes) and spanId (8 bytes), and writes them as lower-case hex.
64-bit integers are strings. A JSON number cannot carry 64 bits exactly, so timestamps and intValue attributes are strings - that part is proto3’s mapping.
A duration is nanoseconds. Duration is in seconds, and the end timestamp is computed from it rather than from StopTime, because the duration comes from a nanosecond performance counter while the wall clock has microsecond resolution.

What an attribute may hold

A timespan and an event carry their attributes like a dictionary: span["key"], span["key"] = 1, "key" in span, del span["key"], len(span), and iteration yielding (key, value) pairs. A key that may not be there is read by get(), which returns a default value instead of raising a KeyError.

Both are TraceElements: a name, the timespan enclosing them, and those attributes. A Span adds the times and what it contains, an Event the moment it happened.

An attribute’s value is one of AttributeValue: bool, int, float, str, bytes, or a list, tuple or dict of those, nested as deeply as needed. Each maps to the matching field of OTLP’s AnyValue, with bytes encoded as base64 and a dict becoming a kvlistValue.

A value of any other type raises a TracingError when the trace is exported. Rendering it with str() instead would put a Python repr into a document that a backend then indexes and offers as a searchable field, which is worse than a failed export.

Note

Both identifiers are drawn when the object is constructed, so exporting one trace twice reports the same traceId and the same spanId values, and TraceID can be handed to another process. That is the identifier a distributed trace is grouped by, as the Trace documentation describes; propagating it between processes is the remaining step.

Attention

An Event always carries a timestamp: the constructor stamps the current system time when none is given. OTLP has no way to say unknown - a missing timeUnixNano reads as the Unix epoch - so an event without a time would be exported as having happened in 1970.

CI Pipelines

pyTooling.Tracing.CI reads the timing of a CI pipeline into a trace, built from recorded timespans. A trace read this way renders and exports like any other, so the time a pipeline spends waiting for runners and running jobs and steps can be inspected in the same viewers.

GitHub Actions

WorkflowRunReader reads a workflow run through the GitHub REST API, using the standard library only:

from os import getenv
from pathlib import Path
from pyTooling.Tracing.CI.GitHub import WorkflowRunReader

reader = WorkflowRunReader("pyTooling/Actions", token=getenv("GITHUB_TOKEN"))
trace = reader.ReadRun(34937615362)      # optionally: attempt=2
trace.WriteJSONFile(Path("report/Pipeline.otlp.json"))

Inside a workflow, GITHUB_TOKEN with the actions: read permission suffices. A job can’t see itself: it is still running when it reads the run, so a timing job depends on every other job and runs last.

A request failing transiently - HTTP 429, 500, 502, 503 or 504, a timeout, or an unreachable API - is tried again, retries times (default: 3), after a pause of retryDelay seconds (default: 2), which doubles with every attempt or lasts as long as a Retry-After header demands, up to a minute. HTTP 401, 403 and 404 fail at once.

WorkflowRunTrace.FromJSON does the conversion alone, for a run and jobs that were fetched another way. It is a class method, so converting needs no reader and therefore no token. It reads both payloads into a Pipeline - see GitHub Actions - and hands that to FromPipeline(), which is the entry point when the model was built elsewhere. Reading the payloads is therefore the model’s job, and a field GitHub doesn’t document raises GitHubError - as does an answer the reader itself can’t read, so everything GitHub says that can’t be made sense of is one exception type. A request that fails is a RESTError, because nothing about GitHub’s answer was wrong - there wasn’t one.

The run becomes the trace, and every timespan below it is marked by Kind with a member of SpanKind. Each kind is a class of its own - a JobSpan sets ci.span.kind to job because that is what it is, and takes the attributes of a job as parameters - so a reader states values and never a key, and a reader of another service builds the same classes:

Kind	Timespan
`pipeline`	The workflow run, from its start to its last update once it completed.
`workflow`	A called workflow: the jobs named `Caller / Job` are grouped below a timespan `Caller`.
`matrix`	A matrix: the jobs named `Job (ubuntu-26.04, 3.14)` are grouped below a timespan `Job`.
`queued`	`<job> (queued)`, the time a job waited for a runner, in front of the job.
`job`	A job, from its start to its completion.
`step`	A step that started, below its job.

Every timespan also carries the attributes of OpenTelemetry’s semantic conventions for CI/CD, which OTLP names as a namespace nested the way the keys are - so OTLP.CICD.Pipeline.Task.Run.ID spells cicd.pipeline.task.run.id and the path can be read to check the key. The values a result may take are Result. What only GitHub reports is named the same way by GitHub, e.g. github.conclusion beside the result it was mapped to. A job’s timespan names its runner and the labels it was requested by, so a renderer can group waiting times per operating system, and a matrix instance additionally lists the values it was produced for in github.matrix.dimensions.

A task is named the way GitHub reports it - Caller / Build (ubuntu-26.04) - while the timespan itself is named by the part the model holds, so a timespan reads in the context its parents already give.

Each flavour builds itself from the model: JobSpan.FromJob takes a Job and produces the job’s timespan, the waiting timespan in front of it, and a timespan per step. So reading a service means mapping its model onto these classes, and everything else - the kinds, the attribute keys, and skipping what the service doesn’t report - is pyTooling.Tracing.CI’s.

What the payloads say is the model’s, including the two facts a timeline depends on: a group’s elements come in the order they were queued, and a job’s times contain its steps, because GitHub reports both in whole seconds and a step is sometimes reported as running outside the job holding it - see GitHub Actions.

GitHub reports timestamps in whole seconds. A step shorter than a second lasts zero seconds, and an end reported a second before its begin is moved to the begin.

Rendering

A trace renders as a Gantt chart: one row per timespan, in the tree’s order and indented by depth.

from pathlib import Path
from pyTooling.Tracing.Render import GanttLayout, StepExclusion, ciSpanFilter
from pyTooling.Tracing.Render.Matplotlib import MatplotlibRenderer

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
MatplotlibRenderer(layout).Write(Path("report/Pipeline.svg"))

Laying out and drawing are two objects. GanttLayout arranges the timespans, and a Renderer draws what it arranged - so a second backend draws the same chart without repeating the arrangement. The layout:

A bar keeps the times it was built from: BeginTime and EndTime are what the trace recorded, while Begin and End are the same times as seconds after the trace began, which is the scale a chart is drawn on. A running timespan ends at the layout’s current time.
The pipeline and a called workflow are a line from their begin to their end, a job is a bar. A job’s waiting timespan (queued) is a light gray bar in front of the job’s bar. A job that didn’t start yet has only its waiting bar, on a row of its own.
A spanFilter hides timespans, and a hidden timespan hides its sub-spans. A filter created by ciSpanFilter() hides the steps of CI jobs - all of them, the skipped ones, or none, as StepExclusion selects - and the jobs that were skipped. Steps outnumber jobs by far: a pipeline of 74 jobs has 1653 steps, 635 of them skipped.
Bars are colored by category. runnerCategory() names the runner image a job ran on, and the MSYS2 environment of a job using one, e.g. windows-2025 + UCRT64, because such a job takes significantly longer than a native job on the same runner. The environment is taken from a successful step matching MSYS2_SETUP_STEP, like Setup MSYS2 for UCRT64.

The legend summarizes the pipeline: when it started and finished, with the time zone; its wall time; its runner time - the time all jobs ran, added up, which runners were occupied for; and per category the number of jobs and their minimum, average and maximum waiting and running times. These statistics count every job that wasn’t skipped, independently of the filter.

The renderer draws it. Write() writes the chart to a file, in the format the suffix names, and Render() returns the backend’s own object for further changes - MatplotlibRenderer a Figure, which isn’t registered with pyplot, so no display is needed. It writes SVG, PNG and PDF, and matplotlib is an optional dependency, installed by the extra pyTooling[diagram].

What no drawing library decides is decided once, on the base-class: the color of every category (Color()), the chart’s title, and the texts of the legend (LegendTitle() and LegendLabel()). A renderer for another backend derives from Renderer, names the file formats it writes in FORMATS, and implements two methods: Render() and _Write.

In an SVG file, every bar or line is a group with the identifier span-<SpanID>, a waiting bar span-<SpanID>-queued and the end marks of a line span-<SpanID>-ends. Together with ParentSpanID, a script can find all elements below a called workflow.