Overview

The pyTooling.Attributes package offers the base implementation of .NET-like attributes realized with Python decorators. The annotated and declarative data is stored as instances of Attribute classes in an additional __pyattr__ field per class, method or function. The annotation syntax allows users to attach any structured data to classes, methods or functions. In many cases, a user will derive a custom attribute from Attribute and override the __init__ method, so user-defined parameters can be accepted when the attribute is constructed.

Later, classes, methods or functions can be searched for by querying the attribute class for attribute instance usage locations (see ‘Function Attributes’ example). Another option for class and method attributes is defining a new classes using pyTooling’s ExtendedType meta-class. Here the class itself offers helper methods for discovering annotated methods (see ‘Method Attributes’ example). While all user-defined (and pre-defined) attributes offer a powerful API derived from Attribute class, the full potential can only be experienced when using class declarations constructed by the pyTooling.MetaClasses.ExtendedType meta-class.

Attributes can create a complex class hierarchy. This helps in finding and filtering for annotated data.

Design Goals

The main design goals are:

Allow meta-data annotation to Python language entities (class, method, function) as declarative syntax.
Find applied attributes based on attribute type (methods on the attribute).
Find applied attributes in a scope (find on class and on class’ methods).
Allow building a hierarchy of attribute classes.
Filter attributes based on their class hierarchy.
Reduce overhead to class creation time (do not impact object creation time).

Example

Function Attributes

from pyTooling.Attributes import Attribute

class Command(Attribute):
  def __init__(self, cmd: str, help: str = ""):
    pass

class Flag(Attribute):
  def __init__(self, param: str, short: str = None, long: str = None, help: str = ""):
    pass

@Command(cmd="version", help="Print version information.")
@Flag(param="verbose", short="-v", long="--verbose", help="Default handler.")
def Handler(self, args):
  pass

for function in Command.GetFunctions():
  pass

Method Attributes

from pyTooling.Attributes import Attribute
from pyTooling.MetaClasses import ExtendedType

class TestCase(Attribute):
  def __init__(self, name: str):
    pass

class Program(metaclass=ExtendedType):
   @TestCase(name="Handler routine")
   def Handler(self, args):
     pass



prog = Program()
for method, attributes in prog.GetMethodsWithAttributes(predicate=TestCase).items():
  pass

Use Cases

In general all classes, methods and functions can be annotated with additional meta-data. It depends on the application, framework or library to decide if annotations should be applied imperatively as regular code or declaratively as attributes via Python decorators.

With this in mind, the following use-cases and ideas can be derived:

Derived Use Cases:

Describe a command line argument parser (like ArgParse) in a declarative form.
See pyTooling.Attributes.ArgParse Package and Examples
Mark nested classes, so later when the outer class gets instantiated, these nested classes are indexed or automatically registered.
See CLIAbstraction → CLIArgument
Mark methods in a class as test cases and classes as test suites, so test cases and suites are not identified based on a magic method name.
Investigation ongoing / planned feature.
Mark class members as public or private and control visibility in auto-generated documentation.
See SphinxExtensions → DocumentMemberAttribute

Predefined Attributes

pyTooling’s attributes offers the Attribute base-class to derive further attribute classes. A derive SimpleAttribute is also offered to accept any *args, **kwargs parameters for annotation of semi-structured meta-data.

It’s recommended to derive an own hierarchy of attribute classes with well-defined parameter lists for the __init__ method. Meta-data stored in attribute should be made accessible via (readonly) properties.

In addition, an pyTooling.Attributes.ArgParse subpackage is provided, which allows users to describe complex argparse command line argument parser structures in a declarative way.

Partial inheritance diagram:

Attribute

The Attribute class implements half of the attribute’s feature set. It implements the instantiation and storage of attribute internal values as well as the search and lookup methods to find attributes. The second half is implemented in the ExtendedType meta-class. It adds attribute specific methods to each class created by that meta-class.

Any attribute is applied on a class, method or function using Python’s decorator syntax, because every attribute is actually a decorator. In addition, such a decorator accepts parameters, which are used to instantiate an attribute class and handover the parameters to that attribute instance.

Every instance of an attribute is registered at its class in a class variable. Further more, these instances are distinguished if they are applied to a class, method or function.

GetClasses() returns a generator to iterate all classes, this attribute was applied to.
GetMethods() returns a generator to iterate all methods, this attribute was applied to.
GetFunctions() returns a generator to iterate all functions, this attribute was applied to.
GetAttributes() returns a tuple of applied attributes to the given method.

Apply a class attribute

from pyTooling.Attributes import Attribute


@Attribute()
class MyClass:
  pass

Apply a method attribute

from pyTooling.Attributes import Attribute

class MyClass:
  @Attribute()
  def MyMethod(self):
    pass

Apply a function attribute

from pyTooling.Attributes import Attribute


@Attribute()
def MyFunction(param):
  pass

Find attribute usages of class attributes

from pyTooling.Attributes import Attribute

for cls in Attribute.GetClasses():
  pass

Find attribute usages of method attributes

from pyTooling.Attributes import Attribute

for method in Attribute.GetMethods():
  pass

Find attribute usages of function attributes

from pyTooling.Attributes import Attribute

for function in Attribute.GetFunctions():
  pass

Condensed definition of class Attribute

@export
class Attribute:

  def __init_subclass__(cls, scope: Nullable[AttributeScope] = None, **kwargs: Any) -> None:
    ...

  def __call__(self, entity: Entity) -> Entity:
    ...

  @readonly
  def Scope(self) -> AttributeScope:
    ...

  @classmethod
  def GetFunctions(cls, scope: Nullable[type | ModuleType] = None) -> Generator[TAttr, None, None]:
    ...

  @classmethod
  def GetClasses(
    cls,
    scope: Nullable[type | ModuleType] = None,
    subclassOf: Nullable[type] = None,
  ) -> Generator[TAttr, None, None]:
    ...

  @classmethod
  def GetMethods(cls, scope: Nullable[type] = None) -> Generator[TAttr, None, None]:
    ...

  @classmethod
  def GetAttributes(
    cls,
    method: MethodType,
    includeSubClasses: bool = True,
  ) -> tuple[Attribute, ...]:
    ...

Planned Features

Allow attributes to be applied only once per kind.
Allow limitation of attributes to classes, methods or functions, so an attribute meant for methods can’t be applied to a function or class.
Allow filtering attribute with a predicate function, so values of an attribute instance can be checked too.

SimpleAttribute

The SimpleAttribute class accepts any positional and any keyword arguments as data. That data is made available via Args and KwArgs properties.

from pyTooling.Attributes import SimpleAttribute

@SimpleAttribute(kind="testsuite")
class MyClass:
  @SimpleAttribute(kind="testcase", id=1, description="Test and operator")
  def test_and(self):
    ...

  @SimpleAttribute(kind="testcase", id=2, description="Test xor operator")
  def test_xor(self):
    ...

Condensed definition of class SimpleAttribute:

class SimpleAttribute(Attribute):
   def __init__(self, *args, **kwargs) -> None:
      ...

   @readonly
   def Args(self) -> Tuple[Any, ...]:
      ...

   @readonly
   def KwArgs(self) -> Dict[str, Any]:
      ...

User-Defined Attributes

It’s recommended to derive user-defined attributes from Attribute, so the __init__ method can be overriden to accept a well defined parameter list including type hints.

The example defines an Annotation attribute, which accepts a single string parameter. When the attribute is applied, the parameter is stored in an instance. The inner field is then accessible via readonly Annotation property.

Find attribute usages of class attributes

@export
class SimpleAttribute(Attribute):

  def __init__(self, *args: Any, **kwargs: Any) -> None:
    ...

  @readonly
  def Args(self) -> tuple[Any, ...]:
    ...

  @readonly
  def KwArgs(self) -> dict[str, Any]:
    ...

Find attribute usages of class attributes

from pyTooling.Attributes import Attribute

class Annotation(Attribute):
  _annotation: str

  def __init__(self, annotation: str):
    self._annotation = annotation

  @readonly
  def Annotation(self) -> str:
    return self._annotation

Searching Attributes

An attribute is searched from either end: from the entity, to ask what was attached to it, or from the attribute class, to ask what it was attached to. The second direction is the one a framework needs, and it is what makes an attribute more than a decorator that stores a value.

From the entity - what is attached here?

GetAttributes() reads the __pyattr__ field of a single method and returns the attributes of this kind attached to it.

from pyTooling.Attributes  import Attribute
from pyTooling.MetaClasses import ExtendedType

class Command(Attribute):
  pass

class Program(metaclass=ExtendedType):
  @Command()
  def Version(self) -> None:
    pass

attributes = Command.GetAttributes(Program.Version)

From the attribute - where was I used?

GetClasses(), GetMethods() and GetFunctions() return generators over every entity the attribute class was attached to - no module has to be imported or walked to find them.

for method in Command.GetMethods():
  print(method)

No scan is involved in either direction. Each attribute class keeps three registries - _classes, _methods and _functions - and the decorator appends to them while the module is being imported, so a query is a list traversal at run time rather than a search. The cost is paid once, at class creation time, which is the design goal above.

Every derived attribute class gets its own registries. __init_subclass__ assigns fresh lists per subclass, so Command.GetMethods() never reports a method that only carries a Flag, although both derive from Attribute.

Attention

A method is only in the method registry if the meta-class put it there. In a class body, a method is still a plain function when the decorator runs, so the attribute registers it as a function. ExtendedType moves it to the method registry while it builds the class. A class built without that meta-class therefore answers GetFunctions(), not GetMethods() - which is why the example above states it.

Attention

An entity is only registered once its defining module has been imported. An attribute query in a plugin architecture therefore answers for the plugins loaded so far, not for the plugins installed.

A registry holds one entry per application, so an entity carrying the attribute twice is yielded twice. Wrap the result in a set - or in dict.fromkeys(), to keep the order - where each entity should be handled once.

Filtering Attributes

The first filter is the class the query is sent to. Command.GetMethods() returns the methods carrying a Command; Attribute.GetMethods() returns nothing, because the base-class was never attached to anything. A hierarchy of attribute classes is therefore a hierarchy of queries, which is the design goal filter attributes based on their class hierarchy.

Subclasses count as the attribute they derive from - but only in the entity direction. GetAttributes() matches with isinstance(), so Command.GetAttributes(method) also returns a SpecialCommand attached to that method. The registries behind GetMethods() are per class, so that query does not see the subclass.

The remaining filters narrow where an entity may be declared:

Method	Parameter	Accepts
`GetClasses()`	`scope`	A module - the class has to be defined in it - or a class, and then the class has to be nested in it.
`GetClasses()`	`subclassOf`	A class the annotated class has to derive from.
`GetMethods()`	`scope`	The class the method has to be defined in.
`GetFunctions()`	`scope`	A module the function has to be defined in.

# every annotated class in this module
Command.GetClasses(scope=sys.modules[__name__])

# ... that is also a 'Handler'
Command.GetClasses(scope=sys.modules[__name__], subclassOf=Handler)

Attention

GetFunctions() accepts a module as its scope. Passing a class raises NotImplementedError rather than returning an empty generator.

Hint

A class built by the ExtendedType meta-class offers the filter the other way round: GetMethodsWithAttributes asks one class which of its methods carry attributes, and its predicate parameter takes an attribute class, an iterable of attribute classes, or None for all of them. Use it when the class is known and the attributes are not; use the queries above when the attribute is known and the classes are not.

Grouping Attributes

A set of attributes that always appear together can be applied as one attribute. Because every attribute class is callable, its __call__ is free to attach more than itself, and _AppendAttribute() is what attaches one attribute to an entity without going through the decorator syntax.

The group attribute stays a normal attribute: it is registered on the entity like the ones it applies, so the entity can afterwards be found through the group or through any of its members.

from pyTooling.Attributes import Attribute, SimpleAttribute

class GroupAttribute(Attribute):
   _id: str

   def __init__(self, id: str):
      self._id = id

   def __call__(self, entity: Entity) -> Entity:
      self._AppendAttribute(entity, SimpleAttribute(3, 4, id=self._id, name="attr1"))
      self._AppendAttribute(entity, SimpleAttribute(5, 6, id=self._id, name="attr2"))

      return entity


class Grouped(TestCase):
   def test_Group_Simple(self) -> None:
      @SimpleAttribute(1, 2, id="my", name="Class1")
      @GroupAttribute("grp")
      class MyClass1:
         pass

Implementation Details

An attribute is stored twice, once per direction of the search described above.

On the entity

The annotated data is stored in an additional __dict__ entry on each annotated class, method or function. The field is named by ATTRIBUTES_MEMBER_NAME, which is __pyattr__, and it holds a list, so multiple attributes can be applied to the same entity.

New attributes are inserted at the front of that list. Decorators are applied bottom-up, so inserting rather than appending gives back the order the decorators were written in:

@Command(cmd="version")     # __pyattr__[0]
@Flag(param="verbose")      # __pyattr__[1]
def Version(self) -> None:
  pass

On the attribute class

The same attribute instance is appended to _classes, _methods or _functions of its own class, chosen by what the entity is - MethodType, FunctionType or type. Anything else raises TypeError; class fields and module variables are not supported.

Scope

AttributeScope is an IntFlag naming the language entities an attribute can be applied to - Class, Method, Function, and Any as their union. A derived attribute states it as class keyword argument scope, and an attribute instance reports it through Scope:

class Hook(Attribute, scope=AttributeScope.Method):
  pass

class AsyncHook(Hook):                 # inherits the scope 'Method'
  pass

The keyword sets _scope, which a class without it inherits. Overriding _scope in the class body still works; the keyword takes precedence. A value other than an AttributeScope raises TypeError.

Applying an attribute to an entity its scope doesn’t allow raises AttributeScopeError, naming the attribute and the entity. A function defined in a class body counts as a method, although it is still a plain function while the decorator runs: its qualified name is <Class>.<name>, where a module’s function is named <name> and a nested function <function>.<locals>.<name>.

It is what makes attribute inheritance possible: when the ExtendedType meta-class builds a class, it walks the base-classes’ __pyattr__ and re-attaches every attribute whose scope contains Class to the derived class. An attribute scoped to methods is not inherited this way, because it was never a statement about the class.

Consumers

This abstraction layer is used by:

✅ Declarative definition of ArgParse parser rules.
pyTooling.Attributes.ArgParse

Competing Solutions

Python has no language feature for attributes: a decorator can attach data to a class, method or function, but nothing records where it was attached. pyTooling.Attributes adds that registry, so an attribute class is asked for the entities carrying it. The solutions below either attach data without this reverse query, or find it by scanning modules.

The model is C#’s attributes: an attribute is a class derived from System.Attribute, and [AttributeUsage] names its targets as AttributeScope does. There, the compiler checks an attribute’s targets and whether it may be applied more than once - pyTooling.Attributes checks neither -, and attributes apply to fields, parameters, return values and assemblies too; but a program finds the uses of an attribute by reflecting over the types of the loaded assemblies.

venusian

Source: venusian (3.1.1, released 2024-12-02), of the Pylons project.

Disadvantages

A decorator only attaches a callback. The decorated objects are found by a Scanner, which imports every module of a package and inspects every object at module scope.
Callbacks are selected by a category string, not by a hierarchy of attribute classes, and the attached data has no type of its own.

Advantages

The scan imports the modules itself, so it finds entities in modules nobody imported yet - a query of pyTooling.Attributes answers for the modules imported so far.
A decorator is passive until a scan, so importing a module has no side effect.

typing.Annotated

Source: PEP 593, typing.Annotated in the standard library.

Disadvantages

It annotates a type - of a parameter, a variable or a return value - not a class, method or function.
Metadata is read from one object, by typing.get_type_hints() with include_extras=True; there is no query for the objects carrying a given metadata class.

Standoff

Complementary: metadata about a value’s type belongs into an annotation, metadata about an entity into an attribute.

Advantages

Part of the standard library and understood by type checkers.

Decorators with a Registry

A decorator that appends the decorated object to a module-level list or dictionary is the usual hand-written solution.

Disadvantages

Every framework writes its own registry, and filtering by a hierarchy of attribute classes has to be written too.
Nothing is attached to the entity, so the entity can’t be asked what was applied to it.

Advantages

A few lines of code and no dependency.