Overview

pyTooling.Tracing records a software execution trace: a tree of timespans, each with its own duration, attributes and events, built with with-statements as a program runs.

from pyTooling.Tracing import Trace, Span, Event

with Trace("build") as trace:
  trace["version"] = "10.0.0"

  with Span("compile") as compile:
    compile["files"] = 12
    Event("cache miss", parent=compile)

  with Span("link"):
    ...

print("\n".join(trace.Format()))

A Trace is the root; every Span inside it attaches to whichever span is active on the current thread, so the tree follows the program’s structure without being wired up by hand. An Event is a point in time rather than a span, and names its span explicitly.

Format() renders the tree as indented lines for a terminal. For anything else, the trace is exported.

Data Model

  • TraceElementa name, a parent and attributes
    • Eventa point in time, attached to a span
    • Spana timespan, holding sub-spans and events
      • Tracethe root timespan, holding the trace’s identifier

  • Every element has a Parent - the enclosing timespan - and attributes, read and written with dictionary syntax: span["files"] = 12. get() reads one that may not be there.

  • A Trace is a Span without a parent. Its TraceID is shared by every timespan below it, and each timespan has its own SpanID. Both are drawn when the element is constructed, so exporting a trace twice reports the same identifiers.

  • An Event names its span with parent; it never attaches to the active span.

  • A timespan left by an exception records it in Exception; the exception isn’t caught and propagates as before. A timespan with an exception has failed.

The active span of a thread

A with-statement attaches a timespan to the active span of the current thread:

  1. Entering a Trace makes it the thread’s active trace and its active span.

  2. Entering a Span appends it to the active span’s sub-spans and makes it the active span; leaving it makes its parent the active span again.

  3. Leaving the trace ends the tracing on that thread.

Span.CurrentSpan() and Trace.CurrentTrace() return them, or None outside a trace.

Each thread has an active span of its own, so a thread started inside a trace has none: entering a span there raises TracingError. A worker thread that wants its timespans in the trace records them with parent and its own times - see Recorded Timespans.

Recorded Timespans

A timespan that was measured elsewhere - by a CI service, or read from a log file - already has its times. It is constructed with them, and attached to its parent by the parent parameter instead of a with-statement:

from datetime import datetime, timezone
from pyTooling.Tracing import Trace, Span

trace = Trace("Pipeline", beginTime=datetime(2026, 9, 15, 6, 35, 21, tzinfo=timezone.utc),
                          endTime=datetime(2026, 9, 15, 6, 44, 30, tzinfo=timezone.utc))
job =   Span("UnitTesting", parent=trace, beginTime=datetime(2026, 9, 15, 6, 35, 32, tzinfo=timezone.utc),
                                          endTime=datetime(2026, 9, 15, 6, 37, 10, tzinfo=timezone.utc))
job["runner"] = "ubuntu-26.04"

print(job.Duration)   # 98.0

  • endTime requires beginTime, can’t precede it, and both are either time zone aware or naive.

  • A source reporting a length instead of an end gives duration in place of endTime, as a timedelta or as a number of seconds - int for whole, float for fractional seconds, the unit Duration reports. It is converted to endTime, so every form is stored alike. Giving both endTime and duration raises an exception.

  • A sub-timespan attached with parent has to lie within its parent’s range. Only the direct parent is checked, because containment is transitive.

  • A timespan with a beginTime but no endTime is still running: its Duration is the time since its recorded begin. Assigning StopTime reports an end that is already known, and Stop() ends a timespan that is still running now. Either accepts the end exactly once.

  • State tells which times are filled in - Empty, Running or Complete - regardless of whether they were measured or recorded. Only an empty timespan can be entered, so a timespan can’t be timed twice.

  • Without beginTime, a timespan is timed by its with-statement. A timespan constructed with recorded times can’t be entered - that raises a TracingError.

OTLP/JSON Export

A Trace converts itself to OTLP/JSON, the OpenTelemetry Protocol’s JSON encoding. One format reaches both usual destinations: an OpenTelemetry collector accepts OTLP natively, and Jaeger has accepted it since v1.35 - so no translation step stands between a trace and a viewer.

from pathlib import Path

trace.WriteJSONFile(Path("trace.json"), serviceName="myProgram")

curl -X POST -H "Content-Type: application/json" -d @trace.json http://localhost:4318/v1/traces

Three methods, for the three things a caller does with the document:

Method

Returns

ToJSON()

the document as an OTLPDocument, for a caller posting it directly

ToJSONString()

the document encoded as a str

WriteJSONFile()

nothing - it writes the document to the given Path

All three take scopeName and scopeVersion, which name the instrumentation scope - the library the spans are reported as coming from. They default to OTLP_SCOPE_NAME and pyTooling’s version, so a program that wraps this tracing in its own API reports itself by passing them rather than by patching the module.

A Span and an Event convert themselves too, but not publicly: a lone span is no OTLP document, because it has no service to be reported under. Each level returns its own part - Span._ToOTLPJSON() returns itself and everything below it, flattened - and the trace wraps the result in the document envelope.

The document is not an untyped mapping: every level of it is a TypedDict named after the OTLP message it encodes, from OTLPDocument down to OTLPAnyValue. A caller can annotate what it received, and a typo in a key is a typing error rather than a document a collector silently rejects.

How a trace is mapped

pyTooling

OTLP

the trace

one resourceSpans entry, whose service.name attribute is serviceName - or the trace’s name

the tree of spans

a flat list, whose parentSpanId references carry the hierarchy

TraceID, drawn when the trace is constructed

traceId on every span of the trace

SpanID, drawn when the timespan is constructed

spanId, and the parentSpanId of everything below it

StartTime and Duration

startTimeUnixNano and endTimeUnixNano

a span’s attributes

attributes, each value wrapped by its type

a span’s events

events

Exception of a failed span

status with code 2 (STATUS_CODE_ERROR) and the exception’s type and message; a successful span has no status

Four details of the encoding are easy to get wrong, and each has a testcase:

  • Identifiers are hex, not base64. OTLP/JSON deviates from proto3’s JSON mapping for traceId (16 bytes) and spanId (8 bytes), and writes them as lower-case hex.

  • 64-bit integers are strings. A JSON number cannot carry 64 bits exactly, so timestamps and intValue attributes are strings - that part is proto3’s mapping.

  • A duration is nanoseconds. Duration is in seconds, and the end timestamp is computed from it rather than from StopTime, because the duration comes from a nanosecond performance counter while the wall clock has microsecond resolution.

  • A timestamp is converted with integers. timestamp() * 1e9 goes through a float, whose 16 significant digits leave the nanoseconds of a current timestamp off by up to a few hundred. Seconds and microseconds are converted separately, so a timestamp is exact to the microsecond.

What an attribute may hold

A timespan and an event carry their attributes like a dictionary: span["key"], span["key"] = 1, "key" in span, del span["key"], len(span), and iteration yielding (key, value) pairs. A key that may not be there is read by get(), which returns a default value instead of raising a KeyError.

Both are TraceElements: a name, the timespan enclosing them, and those attributes. A Span adds the times and what it contains, an Event the moment it happened.

An attribute’s value is one of AttributeValue: bool, int, float, str, bytes, or a list, tuple or dict of those, nested as deeply as needed. Each maps to the matching field of OTLP’s AnyValue, with bytes encoded as base64 and a dict becoming a kvlistValue.

A value of any other type raises a TracingError when the trace is exported. Rendering it with str() instead would put a Python repr into a document that a backend then indexes and offers as a searchable field, which is worse than a failed export.

Note

Both identifiers are drawn when the object is constructed, so exporting one trace twice reports the same traceId and the same spanId values, and TraceID can be handed to another process. That is the identifier a distributed trace is grouped by, as the Trace documentation describes; propagating it between processes is the remaining step.

Attention

An Event always carries a timestamp: the constructor stamps the current system time when none is given. OTLP has no way to say unknown - a missing timeUnixNano reads as the Unix epoch - so an event without a time would be exported as having happened in 1970.

CI Pipelines

pyTooling.Tracing.CI reads the timing of a CI pipeline into a trace, built from recorded timespans. A trace read this way renders and exports like any other, so the time a pipeline spends waiting for runners and running jobs and steps can be inspected in the same viewers.

A service’s reader builds such a trace: pyTooling.GitHub reads a GitHub Actions workflow run through the GitHub REST API.

Rendering

A trace renders as a Gantt chart: one row per timespan, in the tree’s order and indented by depth.

from pathlib import Path
from pyTooling.Tracing.Render import GanttLayout, StepExclusion, ciSpanFilter
from pyTooling.Tracing.Render.Matplotlib import MatplotlibRenderer

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
MatplotlibRenderer(layout).Write(Path("report/Pipeline.svg"))

Laying out and drawing are two objects. GanttLayout arranges the timespans, and a Renderer draws what it arranged - so a second backend draws the same chart without repeating the arrangement. The layout:

  • A bar keeps the times it was built from: BeginTime and EndTime are what the trace recorded, while Begin and End are the same times as seconds after the trace began, which is the scale a chart is drawn on. A running timespan ends at the layout’s current time.

  • The pipeline and a called workflow are a line from their begin to their end, a job is a bar. A job’s waiting timespan (queued) is a light gray bar in front of the job’s bar. A job that didn’t start yet has only its waiting bar, on a row of its own.

  • A spanFilter hides timespans, and a hidden timespan hides its sub-spans. A filter created by ciSpanFilter() hides the steps of CI jobs - all of them, the skipped ones, or none, as StepExclusion selects - and the jobs that were skipped. Steps outnumber jobs by far: a pipeline of 74 jobs has 1653 steps, 635 of them skipped.

  • Bars are colored by category. runnerCategory() names the runner image a job ran on, and the MSYS2 environment of a job using one, e.g. windows-2025 + UCRT64, because such a job takes significantly longer than a native job on the same runner. The environment is taken from a successful step matching MSYS2_SETUP_STEP, like Setup MSYS2 for UCRT64.

The legend summarizes the pipeline: when it started and finished, with the time zone; its wall time; its runner time - the time all jobs ran, added up, which runners were occupied for; and per category the number of jobs and their minimum, average and maximum waiting and running times. These statistics count every job that wasn’t skipped, independently of the filter.

The renderer draws it. Write() writes the chart to a file, in the format the suffix names, and Render() returns the backend’s own object for further changes - MatplotlibRenderer a Figure, which isn’t registered with pyplot, so no display is needed. It writes SVG, PNG and PDF, and matplotlib is an optional dependency, installed by the extra pyTooling[diagram].

What no drawing library decides is decided once, on the base-class: the color of every category (Color()), the chart’s title, and the texts of the legend (LegendTitle() and LegendLabel()). A renderer for another backend derives from Renderer, names the file formats it writes in FORMATS, and implements two methods: Render() and _Write.

In an SVG file, every bar or line is a group with the identifier span-<SpanID>, a waiting bar span-<SpanID>-queued, the end marks of a line span-<SpanID>-ends and a row’s label label-<SpanID>. Together with ParentSpanID, a script can find all elements below a called workflow.

Collapsible SVG

With collapsible=True, MatplotlibRenderer adds a script to an SVG file: clicking the label, bar or line of a row with sub-rows hides the rows below it - a called workflow’s jobs, or a job’s steps - and moves the following rows up. A second click shows them again, and a marker in front of the label shows the state.

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
MatplotlibRenderer(layout, collapsible=True).Write(Path("report/Pipeline.svg"))

The rows of the kinds in collapsedKinds start collapsed - jobs by default, so a chart with steps opens as compact as one without them. The script runs where an SVG file is a document: opened in a browser, or embedded by <object> or inline. An SVG file shown as an image - by <img>, in Markdown or in a pipeline’s job summary - is static and shows every row expanded.

The script and its stylesheet are resource files, CollapsibleGantt.js and CollapsibleGantt.css in pyTooling.Resources.Tracing, whose content is embedded, so the SVG file stays self-contained. getResourceFile() returns their paths, e.g. to use them in a page of your own.

Competing Solutions

pyTooling.Tracing is no replacement for OpenTelemetry’s SDK: it records the timespans a program states itself and writes them as an OTLP document, which OpenTelemetry’s backends read. The packages below are compared for the three things this package does - recording a trace, reading a CI pipeline into one, and drawing it as a Gantt chart.

Outside Python, a CI pipeline becomes a trace by the GitHub receiver of the OpenTelemetry Collector - fed by GitHub’s webhooks, it traces runs as they happen - or by the GitHub Action otel-export-trace-action, which exports a run to an OTLP endpoint. A trace exported as OTLP is shown by trace viewers like Jaeger and Perfetto.

OpenTelemetry SDK

Source: opentelemetry-python, on PyPI as opentelemetry-sdk and the OTLP exporters.

Disadvantages

  • The SDK and its OTLP/HTTP exporter install about a dozen packages, among them protobuf.

  • OTLP/JSON is written by a separate exporter, which is still a beta release (opentelemetry-exporter-otlp-json-http).

  • There is no rendering of a trace; a backend - Jaeger, Grafana Tempo - shows it.

Standoff

  • Both write OTLP, so a trace of this package and a trace of the SDK reach the same collector and viewers.

  • Both accept times measured elsewhere: the SDK takes a start_time and an end_time for a span.

Advantages

  • The standard of the CNCF, with instrumentation for many libraries and frameworks.

  • Context propagation between processes, sampling, batching, and exporters for gRPC and many backends.

otlp-json

Source: otlp-json.

Disadvantages

  • It encodes the spans of OpenTelemetry’s SDK, so the SDK records them.

Advantages

  • Writes OTLP/JSON without any dependency of its own.

Profilers

Source: VizTracer, pyinstrument.

Disadvantages

  • They record every function call - or samples of them - not the timespans a program names, and they don’t write OTLP.

Advantages

  • Nothing in the code has to be changed, and each comes with a viewer for its own format.

Gantt Charts

Source: plotly (plotly.express.timeline), matplotlib (Axes.broken_barh).

Disadvantages

  • A timeline of plotly or matplotlib draws the bars it is given: the rows, their order and indentation, the waiting bars, the colors per category and the statistics of the legend are left to the caller.

Advantages

  • plotly’s chart is interactive in a browser.