Overview

pyTooling.Tracing records a software execution trace: a tree of timespans, each with its own duration, attributes and events, built with with-statements as a program runs.

from pyTooling.Tracing import Trace, Span, Event

with Trace("build") as trace:
  trace["version"] = "10.0.0"

  with Span("compile") as compile:
    compile["files"] = 12
    Event("cache miss", parent=compile)

  with Span("link"):
    ...

print("\n".join(trace.Format()))

A Trace is the root; every Span inside it attaches to whichever span is active on the current thread, so the tree follows the program’s structure without being wired up by hand. An Event is a point in time rather than a span, and names its span explicitly.

Format() renders the tree as indented lines for a terminal. For anything else, the trace is exported.

Data Model

• ▾▸TraceElementa name, a parent and attributes

  • Eventa point in time, attached to a span
  • ▾▸Spana timespan, holding sub-spans and events

    • Tracethe root timespan, holding the trace’s identifier

• Every element has a Parent - the enclosing timespan - and attributes, read and written with dictionary syntax: span["files"] = 12. get() reads one that may not be there.

• A Trace is a Span without a parent. Its TraceID is shared by every timespan below it, and each timespan has its own SpanID. Both are drawn when the element is constructed, so exporting a trace twice reports the same identifiers.

• An Event names its span with parent; it never attaches to the active span.

• A timespan left by an exception records it in Exception; the exception isn’t caught and propagates as before. A timespan with an exception has failed.

The active span of a thread

A with-statement attaches a timespan to the active span of the current thread:

1. Entering a Trace makes it the thread’s active trace and its active span.

2. Entering a Span appends it to the active span’s sub-spans and makes it the active span; leaving it makes its parent the active span again.

3. Leaving the trace ends the tracing on that thread.

Span.CurrentSpan() and Trace.CurrentTrace() return them, or None outside a trace.

Each thread has an active span of its own, so a thread started inside a trace has none: entering a span there raises TracingError. A worker thread that wants its timespans in the trace records them with parent and its own times - see Recorded Timespans.

Recorded Timespans

A timespan that was measured elsewhere - by a CI service, or read from a log file - already has its times. It is constructed with them, and attached to its parent by the parent parameter instead of a with-statement:

from datetime import datetime, timezone
from pyTooling.Tracing import Trace, Span

trace = Trace("Pipeline", beginTime=datetime(2026, 9, 15, 6, 35, 21, tzinfo=timezone.utc),
                          endTime=datetime(2026, 9, 15, 6, 44, 30, tzinfo=timezone.utc))
job =   Span("UnitTesting", parent=trace, beginTime=datetime(2026, 9, 15, 6, 35, 32, tzinfo=timezone.utc),
                                          endTime=datetime(2026, 9, 15, 6, 37, 10, tzinfo=timezone.utc))
job["runner"] = "ubuntu-26.04"

print(job.Duration)            # 0:01:38
print(job.DurationInSeconds)   # 98.0

• endTime requires beginTime, can’t precede it, and both are either time zone aware or naive.

• A source reporting a length instead of an end gives duration in place of endTime, as a timedelta - what Duration reports - or as a number of seconds - int for whole, float for fractional seconds, as DurationInSeconds reports them. It is converted to endTime, so every form is stored alike. Giving both endTime and duration raises an exception.

• A sub-timespan attached with parent has to lie within its parent’s range. Only the direct parent is checked, because containment is transitive.

• A timespan with a beginTime but no endTime is still running: its Duration is the time since its recorded begin. Assigning StopTime reports an end that is already known, and Stop() ends a timespan that is still running now. Either accepts the end exactly once.

• State tells which times are filled in - Empty, Running or Complete - regardless of whether they were measured or recorded. Only an empty timespan can be entered, so a timespan can’t be timed twice.

• Without beginTime, a timespan is timed by its with-statement. A timespan constructed with recorded times can’t be entered - that raises a TracingError.

OTLP/JSON Export

A Trace converts itself to OTLP/JSON, the OpenTelemetry Protocol’s JSON encoding. One format reaches both usual destinations: an OpenTelemetry collector accepts OTLP natively, and Jaeger has accepted it since v1.35 - so no translation step stands between a trace and a viewer.

from pathlib import Path

trace.WriteOTLPJSONFile(Path("trace.json"), serviceName="myProgram")

curl -X POST -H "Content-Type: application/json" -d @trace.json http://localhost:4318/v1/traces

Three methods, for the three things a caller does with the document:

Method	Returns
ToOTLPJSON()	the document as an OTLPDocument, for a caller posting it directly
ToOTLPJSONString()	the document encoded as a str
WriteOTLPJSONFile()	nothing - it writes the document to the given Path

A missing directory of that path is created. Before, a DirectoryCreatedWarning naming it is raised to the current WarningCollector, so a program can report it:

from pyTooling.Warning import WarningCollector

with WarningCollector(handler=lambda warning: print(warning)):
  trace.WriteOTLPJSONFile(Path("report/trace.json"))

All three take scopeName and scopeVersion, which name the instrumentation scope - the library the spans are reported as coming from. They default to OTLP_SCOPE_NAME and pyTooling’s version, so a program that wraps this tracing in its own API reports itself by passing them rather than by patching the module.

A Span and an Event convert themselves too, but not publicly: a lone span is no OTLP document, because it has no service to be reported under. Each level returns its own part - Span._ToOTLPJSON() returns itself and everything below it, flattened - and the trace wraps the result in the document envelope.

The document is not an untyped mapping: every level of it is a TypedDict named after the OTLP message it encodes, from OTLPDocument down to OTLPAnyValue. A caller can annotate what it received, and a typo in a key is a typing error rather than a document a collector silently rejects.

How a trace is mapped

pyTooling	OTLP
the trace	one resourceSpans entry, whose service.name attribute is serviceName - or the trace’s name
the tree of spans	a flat list, whose parentSpanId references carry the hierarchy
TraceID, drawn when the trace is constructed	traceId on every span of the trace
SpanID, drawn when the timespan is constructed	spanId, and the parentSpanId of everything below it
StartTime and DurationInSeconds	startTimeUnixNano and endTimeUnixNano
a span’s attributes	attributes, each value wrapped by its type
a span’s events	events
Exception of a failed span	status with code 2 (STATUS_CODE_ERROR) and the exception’s type and message; a successful span has no status

Four details of the encoding are easy to get wrong, and each has a testcase:

• Identifiers are hex, not base64. OTLP/JSON deviates from proto3’s JSON mapping for traceId (16 bytes) and spanId (8 bytes), and writes them as lower-case hex.

• 64-bit integers are strings. A JSON number cannot carry 64 bits exactly, so timestamps and intValue attributes are strings - that part is proto3’s mapping.

• A duration is nanoseconds. DurationInSeconds is in seconds, and the end timestamp is computed from it rather than from StopTime, because the duration comes from a nanosecond performance counter while the wall clock has microsecond resolution.

• A timestamp is converted with integers. timestamp() * 1e9 goes through a float, whose 16 significant digits leave the nanoseconds of a current timestamp off by up to a few hundred. Seconds and microseconds are converted separately, so a timestamp is exact to the microsecond.

What an attribute may hold

A timespan and an event carry their attributes like a dictionary (KeyValuePairsMixin): span["key"], span["key"] = 1, "key" in span, del span["key"], len(span), and iteration yielding the keys. IterateAttributes() yields (key, value) pairs. A key that may not be there is read by get(), which returns a default value instead of raising a KeyError.

Both are TraceElements: a name, the timespan enclosing them, and those attributes. A Span adds the times and what it contains, an Event the moment it happened.

An attribute’s value is one of AttributeValue: bool, int, float, str, bytes, or a list, tuple or dict of those, nested as deeply as needed. Each maps to the matching field of OTLP’s AnyValue, with bytes encoded as base64 and a dict becoming a kvlistValue.

A value of any other type raises a TracingError when the trace is exported. Rendering it with str() instead would put a Python repr into a document that a backend then indexes and offers as a searchable field, which is worse than a failed export.

Note

Both identifiers are drawn when the object is constructed, so exporting one trace twice reports the same traceId and the same spanId values, and TraceID can be handed to another process. That is the identifier a distributed trace is grouped by, as the Trace documentation describes; propagating it between processes is the remaining step.

Attention

An Event always carries a timestamp: the constructor stamps the current system time when none is given. OTLP has no way to say unknown - a missing timeUnixNano reads as the Unix epoch - so an event without a time would be exported as having happened in 1970.

OTLP/JSON Import

A Trace reads itself back from an OTLP/JSON document, so a trace written by one process - a build step, a worker, an earlier run - can be inspected, formatted or merged by another.

This is the path for a complete trace in a document of its own. A document that carries several traces, or only part of one, is read by a TraceCollection - see Collecting traces.

Reading needs the package jsonschema: install pyTooling[tracing]. Writing needs nothing beyond pyTooling.

from pathlib import Path
from pyTooling.Tracing import Trace

trace = Trace.ReadOTLPJSONFile(Path("trace.json"))
print("\n".join(trace.Format()))

Three class-methods mirror the three export methods:

Class-method	Reads
FromOTLPJSON()	an OTLPDocument, as json.load() returns it
FromOTLPJSONString()	the document encoded as a str
ReadOTLPJSONFile()	the document from the given Path

Reassembling the tree

OTLP has no nesting: a trace is a flat list of spans, and the hierarchy lives in the parentSpanId references. Reading is therefore not the mirror image of writing - the references have to be resolved:

• The document is validated against the schema OTLP-Traces v1.0 first - see Validation.

• The spans of every resourceSpans and scopeSpans entry are collected, because nothing requires a producer to put one trace into one entry, and they are grouped by their traceId.

• The span without a parentSpanId becomes the Trace itself; every other span is created as a sub-span of the one it names, keeping the order the document has it in.

• A document holding more than one trace needs the traceID parameter to say which one to read. Without it, a document of several traces is an error rather than a guess.

FromOTLPJSON() reads the document into a TraceCollection and returns the requested trace - a Trace of Span objects. A Span and an Event read themselves too, but not publicly: Span._ParseOTLPJSON() and Event._ParseOTLPJSON() construct one level and attach it to its parent, which is what the collection walks the resolved references with. A lone span cannot be read publicly for the same reason it cannot be written publicly: it has no trace to belong to.

What a round-trip carries

Exporting a trace, reading it back and exporting it again produces the same document, and there is a testcase saying so. Identifiers, names, the tree, attributes, events and durations all survive.

Five things do not come back. Four of them are a property of OTLP rather than of this reader, and the fifth is a Python object no document carries:

Not read back	Why
service.name	a parameter of ToOTLPJSON(), not a field of the data model
the instrumentation scope	the same - the scope names the library that produced the spans
a span’s kind	every timespan of this data model is SPAN_KIND_INTERNAL
a tuple attribute	it returns as a list, because OTLP has a single arrayValue
a failed span’s status	it is written from Exception, and an exception can’t be rebuilt from its type name and message - the read timespan has no exception, so exporting it again writes no status

Two details are worth knowing about the timestamps. A datetime holds microseconds while the document holds nanoseconds, so a timestamp is rounded to the nearest microsecond rather than truncated - the export scales a timestamp() float by 1e9, whose precision at today’s epoch is about 256 ns, and rounding lands back on the microsecond it came from. The duration, in turn, is the difference of the two timestamps and stays exact, because the performance counter that measured it ran in another process.

Validation

A document is validated against the JSON schema OTLP-Traces v1.0 before anything is read. Every violation is reported - as one note of a TracingError, naming its position in the document:

TracingError: OTLP/JSON document isn't valid according to JSON schema 'OTLP-Traces-v1.0.schema.json'.
  document/resourceSpans/0/scopeSpans/0/spans/0/spanId: 'abc' does not match '^(?!0{16}$)[0-9a-fA-F]{16}$'
  document/resourceSpans/0/scopeSpans/0/spans/0/kind: 'internal' is not of type 'integer'

The schema states what OTLP’s JSON encoding requires, and allows fields of unknown names, which a receiver ignores. What a schema can’t state is checked while the document is read, and raises a TracingError naming the field:

• two spans of a trace with the same spanId (DuplicateSpanError), or two attributes with the same key,

• more than one span of a trace without a parentSpanId (DuplicateTraceError), or spans referencing each other in a cycle (SpanCycleError),

• a span with one timestamp instead of both or neither, or ending before it starts,

• a timestamp a datetime can’t hold,

• a span or an event without a name, an event without timeUnixNano, an attribute without a value - this data model has no unnamed timespans, no undated events and no empty attributes.

What is accepted although the export never writes it: an upper-case identifier - normalized to lower case, so its references still resolve - an intValue or timeUnixNano written as a JSON number instead of a string, a doubleValue of "NaN" or "Infinity", an integer where a double is expected, an empty parentSpanId instead of an absent one, a missing list instead of an empty one, base64 in the URL-safe alphabet or without its padding, and fields of unknown names, which are ignored. Each of those is proto3’s JSON mapping being read as it is written, which is what a document from another producer looks like.

Collecting traces

A document is not a trace. It carries whatever spans a producer had to hand, and two things follow from that:

• it may hold several traces at once, and

• it may hold only part of a trace. A distributed execution is exported by each process separately, so the timespan enclosing a span is often in a different document than the span itself.

A TraceCollection holds both cases. It keeps the traces whose root span has arrived, and the fragments - timespans whose parentSpanId names a span nothing has delivered yet. A fragment is a real timespan with its own sub-spans; only its place in the tree is unknown. Its root is a Fragment, which keeps the identifiers of its trace and of the parent it is waiting for (TraceID, ParentSpanID), and the timespans below it name the fragment as their Trace until the parent arrives.

from pathlib import Path
from pyTooling.Tracing import TraceCollection

collection = TraceCollection.ReadOTLPJSONFile(Path("frontend.json"))
collection.AddOTLPJSONFile(Path("worker.json"))
collection.AddOTLPJSONFile(Path("database.json"))

for trace in collection:
  print("\n".join(trace.Format()))

if collection.HasFragments:
  print(f"{collection.FragmentCount} timespans are still waiting for their parent.")

The three documents may be added in any order. Each one is linked in both directions: a fragment it brings finds a parent that arrived earlier, and a span it brings collects the fragments that were waiting for it. When a fragment is attached, its parent’s sub-spans are re-sorted by start time, so the reassembled trace reads like a local one instead of like the order the files happened to be read in.

Looking things up

Both indexes are keyed by identifier, which is what makes the linking possible - and what makes the collection useful on its own:

Expression	Yields
collection[traceID]	the Trace, for an identifier of 32 hex digits
collection[spanID]	the Span, for an identifier of 16 hex digits - fragments included
Traces / Fragments	the complete traces, and the root timespan of each fragment
TraceIDOfSpan()	which trace a timespan belongs to - for a timespan of a fragment too, whose trace’s root span hasn’t arrived
for trace in collection / len(collection)	the complete traces; a fragment is not a trace and is neither iterated nor counted as one

A trace identifier is 32 hex digits and a span identifier is 16, so which of the two indexes [...] searches follows from the length. The two can’t be confused. An identifier nothing has delivered raises a TraceNotFoundError or a SpanNotFoundError, both a LookupError; an identifier of another length raises a ValueError.

Writing a collection back

ToOTLPJSON() writes every trace as its own resourceSpans entry, and a fragment into the entry of the trace it belongs to - carrying the parentSpanId it is waiting for, so reading the document back produces the same fragment rather than a second root span. A trace whose root span never arrived has no name to report, so its service.name is its trace identifier.

Which entry point?

Use Trace.FromOTLPJSON when the document holds one complete trace and a Trace is what the caller wants - it rejects anything else rather than returning half a tree. Use TraceCollection.FromOTLPJSON when the document is whatever a collector handed over.

CI Pipelines

pyTooling.Tracing.CI reads the timing of a CI pipeline into a trace, built from recorded timespans. A trace read this way renders and exports like any other, so the time a pipeline spends waiting for runners and running jobs and steps can be inspected in the same viewers.

A service’s reader builds such a trace: pyTooling.GitHub reads a GitHub Actions workflow run through the GitHub REST API.

Rendering

A trace renders as a Gantt chart: one row per timespan, in the tree’s order and indented by depth.

from pathlib import Path
from pyTooling.Tracing.Render import GanttLayout, StepExclusion, ciSpanFilter
from pyTooling.Tracing.Render.Matplotlib import MatplotlibRenderer

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
MatplotlibRenderer(layout).Write(Path("report/Pipeline.svg"))

Laying out and drawing are two objects. GanttLayout arranges the timespans, and a Renderer draws what it arranged - so a second backend draws the same chart without repeating the arrangement. The layout:

• A bar keeps the times it was built from: BeginTime and EndTime are what the trace recorded, while Begin and End are the same times as seconds after the trace began, which is the scale a chart is drawn on. A running timespan ends at the layout’s current time.

• The pipeline and a called workflow are a line from their begin to their end, a job is a bar. A job’s waiting timespan (queued) is a light gray bar in front of the job’s bar. A job that didn’t start yet has only its waiting bar, on a row of its own.

• A spanFilter hides timespans, and a hidden timespan hides its sub-spans. A filter created by ciSpanFilter() hides the steps of CI jobs - all of them, the skipped ones, or none, as StepExclusion selects - and the jobs that were skipped. Steps outnumber jobs by far: a pipeline of 74 jobs has 1653 steps, 635 of them skipped.

• Bars are colored by category. runnerCategory() names the runner image a job ran on, and the MSYS2 environment of a job using one, e.g. windows-2025 + UCRT64, because such a job takes significantly longer than a native job on the same runner. The environment is taken from a successful step matching MSYS2_SETUP_STEP, like Setup MSYS2 for UCRT64.

The legend summarizes the pipeline: when it started and finished, with the time zone; its wall time; its runner time - the time all jobs ran, added up, which runners were occupied for; and per category the number of jobs and their minimum, average and maximum waiting and running times. These statistics count every job that wasn’t skipped, independently of the filter.

The renderer draws it. Write() writes the chart to a file, in the format the suffix names, and Render() returns the backend’s own object for further changes - MatplotlibRenderer a Figure, which isn’t registered with pyplot, so no display is needed. It writes SVG, PNG and PDF, and matplotlib is an optional dependency, installed by the extra pyTooling[diagram] together with plotly, which draws the same chart as an interactive HTML page - see Interactive HTML.

What no drawing library decides is decided once, on the base-class: the color of every category (Color()), the chart’s title, and the texts of the legend (LegendTitle() and LegendLabel()). A renderer for another backend derives from Renderer, names the file formats it writes in FORMATS, and implements two methods: Render() and _Write.

In an SVG file, every bar or line is a group with the identifier span-<SpanID>, a waiting bar span-<SpanID>-queued, the end marks of a line span-<SpanID>-ends and a row’s label label-<SpanID>. Together with ParentSpanID, a script can find all elements below a called workflow.

Collapsible SVG

With collapsible=True, MatplotlibRenderer adds a script to an SVG file: clicking the label, bar or line of a row with sub-rows hides the rows below it - a called workflow’s jobs, or a job’s steps - and moves the following rows up. A second click shows them again, and a marker in front of the label shows the state.

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
MatplotlibRenderer(layout, collapsible=True).Write(Path("report/Pipeline.svg"))

The rows of the kinds in collapsedKinds start collapsed - jobs by default, so a chart with steps opens as compact as one without them. The script runs where an SVG file is a document: opened in a browser, or embedded by <object> or inline. An SVG file shown as an image - by <img>, in Markdown or in a pipeline’s job summary - is static and shows every row expanded.

The script and its stylesheet are resource files, CollapsibleGantt.js and CollapsibleGantt.css in pyTooling.Resources.Tracing, whose content is embedded, so the SVG file stays self-contained. getResourceFile() returns their paths, e.g. to use them in a page of your own.

Interactive HTML

PlotlyRenderer draws the chart with plotly and writes it as an HTML page, or as the plotly figure’s JSON, chosen by the file’s suffix. Render() returns a plotly figure for further changes. It draws the same layout as the matplotlib renderer, and the colors and legend come from the same base-class, so both charts look alike.

from pyTooling.Tracing.Render.Plotly import PlotlyRenderer

layout = GanttLayout(trace, spanFilter=ciSpanFilter(excludeSteps=StepExclusion.Skipped))
PlotlyRenderer(layout).Write(Path("report/Pipeline.html"))

The page can be zoomed and panned. Hovering a bar or line shows the timespan’s name, its absolute begin and end, and its duration, and a click on a legend entry hides or shows a category’s bars. The page embeds plotly’s JavaScript library - about 4 MiB - so it works offline, e.g. downloaded from a pipeline’s artifacts; includePlotlyJS=PlotlyJavaScript.CDN (PlotlyJavaScript) loads the library from plotly’s CDN instead.

The time axis is a date axis showing the time since the trace began as hh:mm:ss, so its ticks adapt when zooming. For a trace longer than a day, the ticks start again at 00:00:00.

Competing Solutions

pyTooling.Tracing is no replacement for OpenTelemetry’s SDK: it records the timespans a program states itself and writes them as an OTLP document, which OpenTelemetry’s backends read. The packages below are compared for the three things this package does - recording a trace, reading a CI pipeline into one, and drawing it as a Gantt chart.

Outside Python, a CI pipeline becomes a trace by the GitHub receiver of the OpenTelemetry Collector - fed by GitHub’s webhooks, it traces runs as they happen - or by the GitHub Action otel-export-trace-action, which exports a run to an OTLP endpoint. A trace exported as OTLP is shown by trace viewers like Jaeger and Perfetto.

OpenTelemetry SDK

Source: opentelemetry-python, on PyPI as opentelemetry-sdk and the OTLP exporters.

Disadvantages

• The SDK and its OTLP/HTTP exporter install about a dozen packages, among them protobuf.

• OTLP/JSON is written by a separate exporter, which is still a beta release (opentelemetry-exporter-otlp-json-http).

• There is no rendering of a trace; a backend - Jaeger, Grafana Tempo - shows it.

Standoff

• Both write OTLP, so a trace of this package and a trace of the SDK reach the same collector and viewers.

• Both accept times measured elsewhere: the SDK takes a start_time and an end_time for a span.

Advantages

• The standard of the CNCF, with instrumentation for many libraries and frameworks.

• Context propagation between processes, sampling, batching, and exporters for gRPC and many backends.

otlp-json

Source: otlp-json.

Disadvantages

• It encodes the spans of OpenTelemetry’s SDK, so the SDK records them.

Advantages

• Writes OTLP/JSON without any dependency of its own.

Profilers

Source: VizTracer, pyinstrument.

Disadvantages

• They record every function call - or samples of them - not the timespans a program names, and they don’t write OTLP.

Advantages

• Nothing in the code has to be changed, and each comes with a viewer for its own format.

Gantt Charts

Source: plotly (plotly.express.timeline), matplotlib (Axes.broken_barh).

Disadvantages

• A timeline of plotly or matplotlib draws the bars it is given: the rows, their order and indentation, the waiting bars, the colors per category and the statistics of the legend are left to the caller.

Advantages

• plotly’s chart is interactive in a browser.