Attributes

A framework usually needs to know things about a user’s code that the code itself doesn’t say: which classes are plugins, which function or methods handle a callback. An attribute is a decorator that carries that data and remembers where it was applied - the same idea as .NET attributes, which is where pyTooling.Attributes takes it from.

This tutorial builds one from scratch: a plug-in system where a plug-in is a class, its metadata travels with it, and the framework finds every plug-in without the plug-in ever calling the framework.

@Plugin(name="MyPlugin", version="1.0.0")
class MyPlugin:
  ...

for plugin in Plugin.GetClasses():
  ...   # every class annotated with @Plugin, without a registry anywhere

Three ways to know what a program declares

Attributes are the third of them, and the differences are worth seeing side by side before the mechanics.

Approach	How the framework learns	Costs
imperative	the plug-in module calls `register(MarkdownReader, "markdown")` itself	the call is separate from the thing it registers, so the two drift; and every plug-in has to remember to make it
meta-class / registry	a base-class’ `__init_subclass__()` - or a meta-class’ `__new__()` - adds each subclass to a class variable as it is created	nothing to remember, but it only reaches classes, it forces the plug-in to derive from your base-class, and the metadata has nowhere to live except more class variables
attribute	the decorator records itself on the entity, and the attribute class keeps a registry of where it was applied	reaches classes, methods and functions, carries typed data of its own, and leaves the plug-in’s inheritance alone

The registry approach is not a poor relation - Attribute uses __init_subclass__() internally for exactly that reason, to give every derived attribute class its own registry. It is the right tool when being a subclass is already the thing you want to enumerate. It stops being the right tool the moment the annotation carries data, or has to sit on a function.

When an attribute is the right tool

Reach for one when the data belongs to the entity, and the framework has to find the entity by the data. Both halves matter:

Use instead	When
a plain decorator	the decorator changes what the function does. An attribute annotates, it doesn’t wrap.
a class field or a constant	nothing ever searches for it. A dictionary is simpler than an attribute class.
a naming convention	the framework can find entities by name and the name is not also the description - `test_*` is the classic case, and its limits are exactly why markers exist.
an attribute	the annotation is data - a name, a version, a priority - and something has to enumerate every entity carrying it.

A plug-in system needs both halves at once, which is why it is the example this tutorial uses: the name and version belong to the plug-in class, and the host application has to enumerate every plug-in it can find.

Example 1: a class-based plug-in mechanism

A plug-in is a class, its metadata travels with it, and the host finds every plug-in without the plug-in ever calling the host. Four steps: define the attribute, apply it, load the modules, find what was annotated.

Step 1: define an attribute class

Derive from Attribute and give the initializer the parameters the annotation should carry. Expose them as read-only properties, because whoever finds the annotation later has to read them.

The parameter is accepted as a string and stored as a SemanticVersion, so two plug-in versions can be compared instead of only printed.

That is the whole definition. Two things come from the base-class: __call__, which is what lets the class be used as a decorator, and the registry that the Get*** methods read.

Hint

For a quick and simple annotation that carries positional or keyword data and nothing else, SimpleAttribute skips the class definition entirely:

from pyTooling.Attributes import SimpleAttribute

@SimpleAttribute(kind="reader", order=3)
class MarkdownReader:
  ...

attribute = SimpleAttribute.GetAttributes(MarkdownReader)[0]
attribute.Args     # -> ()
attribute.KwArgs   # -> {'kind': 'reader', 'order': 3}

from pyTooling.Attributes import Attribute
from pyTooling.Decorators import export, readonly
from pyTooling.Versioning import SemanticVersion

@export
class Plugin(Attribute):
  """Marks a class as a plug-in of this application."""

  _name:    str              #: Name the plug-in is selected by.
  _version: SemanticVersion  #: Version of the plug-in, as its author declares it.

  def __init__(self, name: str, version: str = "0.0.0") -> None:
    self._name =    name
    self._version = SemanticVersion.Parse(version)

  @readonly
  def Name(self) -> str:
    return self._name

  @readonly
  def Version(self) -> SemanticVersion:
    return self._version

Step 2: apply it

An attribute goes on a class, a method or a function. Several attributes stack, and an entity may carry the same attribute class more than once.

The plug-in declares what it is through the attribute, and what it does through an ordinary base-class the application defines. The two are independent, and that separation is the point: Reader is the interface, and @Plugin is the metadata.

An attribute carrying no data at all is written without parentheses - @Plugin rather than @Plugin() - but a plug-in almost always has a name, so the parenthesised form is the usual one here.

Important

The interface is built with ExtendedType, and that is not decoration. It makes abstractmethod() reject a plug-in that forgot to implement Read, and - as the second example shows - it is what makes annotated methods findable at all. A plug-in class alone does not need it; a plug-in class whose methods carry attributes does.

from pathlib import Path

from pyTooling.MetaClasses import ExtendedType, abstractmethod

# the application's own interface
class Reader(metaclass=ExtendedType):
  @abstractmethod
  def Read(self, path: Path) -> str:
    """Read the document at 'path' and return its text."""

# plugins/markdown.py
@Plugin(name="markdown", version="1.2.0")
class MarkdownReader(Reader):
  def Read(self, path: Path) -> str:
    return path.read_text(encoding="utf-8")

Step 3: load the plug-in modules

Attention

An entity is registered when its module is imported, because that is when the decorator runs. A plug-in in a module nothing imports is invisible to GetClasses().

This is the single thing that catches people out, and for a plug-in system it is not an edge case - it is the mechanism. Discovery is two steps that are easy to mistake for one: import the modules, then ask the attribute. A framework that only does the second finds nothing and looks broken.

Before that sweep Plugin.GetClasses() yields nothing; afterwards it yields one class per imported plug-in. Nothing else changed - the registry was simply empty because no decorator had run yet.

Tip

Scanning a directory is the simplest loader and the right one for plug-ins shipped inside the application. For plug-ins installed as separate distributions, use entry points: the installer records them, importlib.metadata.entry_points() lists them without importing anything, and calling load() on one imports that module - at which point the attribute registry fills exactly as above. The two compose; the attribute doesn’t care which one imported the module.

from importlib import import_module
from pathlib   import Path

def LoadPlugins(directory: Path) -> None:
  """Import every plug-in module, so its decorators run and register what they annotate."""
  for module in sorted(directory.glob("*.py")):
    if not module.stem.startswith("_"):
      import_module(f"{directory.name}.{module.stem}")

Step 4: find what was annotated

Three class-methods answer the three kinds of entity, and each yields what was annotated - not the attribute instances:

Method	Yields
`Plugin.GetClasses()`	every class annotated with `@Plugin`; `subclassOf=` narrows further
`Plugin.GetFunctions()`	every function annotated with it
`Plugin.GetMethods()`	every method annotated with it
`Plugin.GetAttributes(entity)`	the attribute instances on one entity - this is where the data is read

So finding the plug-ins and reading their metadata are two steps:

subclassOf= is what makes this usable in an application with more than one kind of plug-in. It filters the annotated classes by their own base-class, so one attribute can mark every plug-in while each extension point asks only for the ones it can use:

scope= restricts the result to entities declared in one module or nested in one class - useful when two plug-ins declare a class of the same name.

Attention

The registry records one entry per application, so an entity carrying the attribute twice is yielded twice by GetClasses(). Wrap the result in a set (or dict.fromkeys(), to keep the order) when each entity should be processed once.

for pluginClass in Plugin.GetClasses():
  for attribute in Plugin.GetAttributes(pluginClass):
    print(f"{attribute.Name:<10} {attribute.Version}  -> {pluginClass.__name__}")

markdown   1.2.0  -> MarkdownReader
asciidoc   0.1.0  -> AsciiDocReader

Plugin.GetClasses(subclassOf=Reader)   # -> MarkdownReader, AsciiDocReader
Plugin.GetClasses(subclassOf=Writer)   # -> HtmlWriter

Example 2: hooks on a plug-in’s methods

The first example annotated classes. The same mechanism works one level down, on the methods inside a plug-in: a plug-in that reacts to events declares which method handles which event, instead of the host calling methods by a magic name.

This example builds on the first - Plugin still marks the class; Hook marks the methods within it.

Step 1: define a second attribute

An attribute class per kind of annotation. Hook carries the event name:

class Hook(Attribute):
  """Marks a method as the handler of one application event."""

  _event: str  #: Name of the event this method handles.

  def __init__(self, event: str) -> None:
    self._event = event

  @readonly
  def Event(self) -> str:
    return self._event

Step 2: ask one object what it offers

GetMethods() returns annotated methods of every class, which is rarely what a host wants - it holds one plug-in instance and asks what does this one offer? A class built with ExtendedType answers that itself:

Attention

GetMethodsWithAttributes() returns a dictionary of method to attributes, so iterating it yields methods. .items() is what gives the pairs; iterating the result directly and unpacking raises TypeError: cannot unpack non-iterable function object.

OnError is a perfectly ordinary public method; it is absent from the result because it carries no @Hook, not because of how it is named. That is the whole difference from a naming convention.

The same answer can be had from the attribute instead of from the object, which is the right way round when the host wants every plug-in’s handlers rather than one plug-in’s:

Important

GetMethods() only ever finds methods of classes built with ExtendedType. When the decorator runs, a method in a class body is still a plain function - it becomes a method only once the class object exists - so the annotation is filed under functions. ExtendedType re-files it while creating the class. Without the meta-class, the same method turns up in GetFunctions() and GetMethods() stays empty.

predicate= accepts an attribute class or an iterable of them, and it matches sub-classes too - which is the point of the next step.

from pyTooling.MetaClasses import ExtendedType

@Plugin(name="markdown", version="1.2.0")
class MarkdownReader(Reader, metaclass=ExtendedType):
  @Hook("open")
  def OnOpen(self, path):
    ...

  @Hook("close")
  def OnClose(self, path):
    ...

  def OnError(self, error):   # not annotated, so not found
    ...

plugin = MarkdownReader()
for method, attributes in plugin.GetMethodsWithAttributes(predicate=Hook).items():
  for attribute in attributes:
    print(f"{attribute.Event:<6} -> {method.__name__}")

open   -> OnOpen
close  -> OnClose

for method in Hook.GetMethods():
  for attribute in Hook.GetAttributes(method):
    print(f"{attribute.Event:<6} -> {method.__qualname__}")

open   -> MarkdownReader.OnOpen
close  -> MarkdownReader.OnClose

Step 3: build a hierarchy and filter by it

Attribute classes inherit, and each derived class gets its own registry - so a specialised attribute is found by its own name, and only by its own name:

Important

Base and derived registries are separate, not nested. Plugin.GetClasses() does not return the class annotated with @ExperimentalPlugin. Each derived class receives fresh registries in Attribute.__init_subclass__, which is what stops a derived attribute from reporting entities it was never attached to - and the cost is that the base doesn’t collect its children’s.

So an application wanting every plug-in has to ask each attribute class it defines. Where sub-class matching is wanted for free, it comes from GetMethodsWithAttributes() with predicate=Plugin, and from GetAttributes() with includeSubClasses=True - both of which test isinstance rather than reading one registry.

class ExperimentalPlugin(Plugin):
  """A plug-in that may be withdrawn without a deprecation period."""

@Plugin(name="markdown", version="1.2.0")
class MarkdownReader(Reader): ...

@ExperimentalPlugin(name="asciidoc", version="0.1.0")
class AsciiDocReader(Reader): ...

ExperimentalPlugin.GetClasses()   # -> AsciiDocReader
Plugin.GetClasses()               # -> MarkdownReader  (only!)

A word on `AttributeScope`

An attribute class may declare where it is meant to be used:

from pyTooling.Attributes import Attribute, AttributeScope

class Hook(Attribute):
  _scope = AttributeScope.Method

Caution

_scope currently documents intent, and nothing enforces it - see #384. Applying a Class-scoped attribute to a function is accepted silently, and the function is then registered in a list that attribute’s own scope says it can never hold.

Two cases have to be told apart, because only one of them is fixable where the attribute is applied:

class versus function/method is decidable at that moment, so this mismatch should raise and today does not;
method versus function is not decidable there - as the previous example explains, a method is still a plain function while the decorator runs.

So until #384 is resolved: check the entity kind yourself if a misapplication has to fail.

Note also that _scope reads back from an attribute instance - Hook("open").Scope - and not from the class.

Attribute use cases

Plug-ins are one application of the pattern. It recurs wherever a framework has to find code by what it means rather than by what it is called - and three of those are inside pyTooling itself.

Finding testcases and testsuites

A test runner finds tests by a magic name - test_* - which forces the identifier to be both the selector and the description, and leaves nowhere to put a title with spaces in it. An attribute separates the two:

The method keeps a name a developer can type on a command line, and the annotation carries the sentence a report should print. It is the same trade the plug-in example makes: MarkdownReader keeps a class name Python can import, and @Plugin(name="markdown") carries the name a user types.

pyTooling ships this as finding testcases and testsuites.

class testcase(Attribute):
  """Marks a method as a testcase and gives it a human-readable title."""

  def __init__(self, title: str) -> None:
    self._title = title

  @readonly
  def Title(self) -> str:
    return self._title

class ArithmeticTests(metaclass=ExtendedType):
  @testcase("adds two positive numbers")
  def CheckAddition(self) -> None:
    ...

Declarative argparse

A command line parser is usually built imperatively - one add_argument() call per flag, far away from the function that handles it. ArgParse inverts that: the commands, their flags and their handlers are attributes on the handler methods, and the ArgumentParser is assembled from them.

The flag and the code that receives it cannot drift apart, because they are the same declaration.

@CommandHandler("build", help="Build the project.")
@LongValuedFlag("--target", dest="target", help="Build target.")
def HandleBuild(self, args) -> None:
  ...

Collecting a program’s CLI arguments

CLIArgument marks the nested argument classes of a Program, so the outer class collects them when it is created - the same find-by-annotation step as Plugin.GetClasses(), scoped to one class instead of a whole module.

Attributes

Three ways to know what a program declares

When an attribute is the right tool

Example 1: a class-based plug-in mechanism

Step 1: define an attribute class

Step 2: apply it

Step 3: load the plug-in modules

Step 4: find what was annotated

Example 2: hooks on a plug-in’s methods

Step 1: define a second attribute

Step 2: ask one object what it offers

Step 3: build a hierarchy and filter by it

A word on AttributeScope

Attribute use cases

Finding testcases and testsuites

Declarative argparse

Collecting a program’s CLI arguments

A word on `AttributeScope`