Sourcecode on GitHub Code license Documentation - Read Now! Documentation License
PyPI - Tag PyPI - Status PyPI - Python Version
GitHub Workflow - Build and Test Status Libraries.io status for latest release Codacy - Quality Codacy - Line Coverage Codecov - Branch Coverage

pyTooling Documentation

pyTooling is a powerful collection of arbitrary useful abstract data models, lacking classes, decorators, a new performance boosting meta-class and enhanced exceptions. It also provides lots of helper functions e.g. to ease the handling of package descriptions or to unify multiple existing APIs into a single API.

It’s useful ‒ if not even essential ‒ for any Python-base project independent if it’s a library, framework, CLI tool or just a “script”.

In addition, pyTooling provides a collection of CI job templates for GitHub Actions. This drastically simplifies GHA-based CI pipelines for Python projects.

Package Details

The following descriptions and code examples give peak onto pyTooling’s highlights. But be ensured, there is more to explore, which can’t be highlighted on the main landing page.

Attributes

The pyTooling.Attributes module offers the base implementation of .NET-like attributes realized with Python decorators. The annotated and declarative data is stored as instances of Attribute classes in an additional __pyattr__ field per class, method or function.

The annotation syntax (decorator syntax) allows users to attach any structured data to classes, methods or functions. In many cases, a user will derive a custom attribute from Attribute and override the __init__ method, so user-defined parameters can be accepted when the attribute is constructed.

Later, classes, methods or functions can be searched for by querying the attribute class for attribute instance usage locations (see example to the right). Another option for class and method attributes is declaring a classes using pyTooling’s ExtendedType meta-class. Here the class itself offers helper methods for discovering annotated methods.

A SimpleAttribute class is offered accepting any positional and keyword parameters. In a more advanced use case, users are encouraged to derive their own attribute class hierarchy from Attribute.

from pyTooling.Attributes import Attribute

class Command(Attribute):
  def __init__(self, cmd: str, help: str = ""):
    pass

class Flag(Attribute):
  def __init__(self, param: str, short: str = None, long: str = None, help: str = ""):
    pass

@Command(cmd="version", help="Print version information.")
@Flag(param="verbose", short="-v", long="--verbose", help="Default handler.")
def Handler(self, args):
  pass

for function in Command.GetFunctions():
  pass

from pyTooling.Attributes import Attribute
from pyTooling.MetaClasses import ExtendedType

class TestCase(Attribute):
  def __init__(self, name: str):
    pass

class Program(metaclass=ExtendedType):
   @TestCase(name="Handler routine")
   def Handler(self, args):
     pass



prog = Program()
for method, attributes in prog.GetMethodsWithAttributes(predicate=TestCase):
  pass

from pyTooling.Attributes import Attribute
from pyTooling.MetaClasses import ExtendedType

class TestSuite(Attribute):
  def __init__(self, name: str):
    pass

@TestSuite(name="Command line interface tests")
class Program(metaclass=ExtendedType):
   def Handler(self, args):
     pass

prog = Program()


for testsuite in TestSuite.GetClasses():
  pass

ArgParse

Defining commands, arguments or flags for a command line argument parser like argparse is done imperatively. This means code executed in-order defines how the parser will accept inputs. Then more user-defined code is needed to dispatch the collected and type-converted arguments to handler routines. See an example to the right as “Traditional argparse”.

In contrast, pyTooling.Attributes.ArgParse allows the definition of commands, arguments or flags as declarative code attached to handler routines using pyTooling’s attributes. This allow a cleaner and more readable coding style. Also maintainability is improved, as arguments are defined using clear attribute names attached to the matching handler routine. Thus parser and handler code is not separated.

If the command line interface uses many commands, handlers and their arguments can be spread across mixin classes. Later, the whole CLI is assembled by using multiple inheritance. In case handlers use shared argument sets, arguments can be grouped and shared by defining grouping attributes.

class Program:
  def __init__(self):
    mainParser = argparse.ArgumentParser()
              mainParser.set_defaults(func=self.HandleDefault)
    mainParser.add_argument("-v", "--verbose")
    subParsers = mainParser.add_subparsers()

    newUserParser = subParsers.add_parser("new-user", help="Add a new user.")
    newUserParser.add_argument(dest="username", metaName="username", help="Name of the new user.")
    newUserParser.add_argument("--quota", dest="quota", help="Max usable disk space.")
    newUserParser.set_defaults(func=self.NewUserHandler)

    deleteUserParser = subParsers.add_parser("delete-user", help="Delete a user.")
    deleteUserParser.add_argument(dest="username", metaName="username", help="Name of the user.")
    deleteUserParser.add_argument("-f", "--force", dest="force", help="Ignore internal checks.")
    deleteUserParser.set_defaults(func=self.DeleteUserHandler)

    listUserParser = subParsers.add_parser("list-user", help="List all users.")
    listUserParser.set_defaults(func=self.ListUserHandler)

  def HandleDefault(self, args) -> None:
    pass

  def NewUserHandler(self, args) -> None:
    pass

  def DeleteUserHandler(self, args) -> None:
    pass

  def ListUserHandler(self, args) -> None:
    pass

class Program:
  @DefaultHandler()
  @FlagArgument(short="-v", long="--verbose", dest="verbose", help="Show verbose messages.")
  def HandleDefault(self, args) -> None:
    pass

  @CommandHandler("new-user", help="Add a new user.")
  @StringArgument(dest="username", metaName="username", help="Name of the new user.")
  @LongValuedFlag("--quota", dest="quota", help="Max usable disk space.")
  def NewUserHandler(self, args) -> None:
    pass

  @CommandHandler("delete-user", help="Delete a user.")
  @StringArgument(dest="username", metaName="username", help="Name of the user.")
  @FlagArgument(short="-f", long="--force", dest="force", help="Ignore internal checks.")
  def DeleteUserHandler(self, args) -> None:
    pass

  @CommandHandler("list-user", help="List all users.")
  def ListUserHandler(self, args) -> None:
    pass

CLI Abstraction

pyTooling.CLIAbstraction offers an abstraction layer for command line programs, so they can be used easily in Python. There is no need for manually assembling parameter lists or considering the order of parameters. All parameters like -v or --value=42 are described using nested classes on a Program class. Each nested class derived from predefined argument classes knows about the correct formatting pattern, character escaping, and if needed about necessary type conversions.

Such an instance of a program can be converted to an argument list suitable for subprocess.Popen. In stead of deriving from Program, abstracted command line tools can derive from Executable which offers embedded Popen behavior.

class Git(Executable):
  def __new__(cls, *args: Tuple[Any, ...], **kwargs: Dict[str, Any]):
    cls._executableNames = {
      "Darwin": "git",
      "Linux": "git",
      "Windows": "git.exe"
    }
    return super().__new__(cls)

  @CLIArgument()
  class FlagHelp(ShortFlag, name="h"): ...

  @CLIArgument()
  class FlagVersion(LongFlag, name="version"): ...

  @CLIArgument()
  class CommandHelp(CommandArgument, name="help"): ...

  @CLIArgument()
  class CommandCommit(CommandArgument, name="commit"): ...

  @CLIArgument()
  class ValueCommitMessage(ShortTupleFlag, name="m"): ...

tool = Git()
tool[tool.FlagVersion] = True

tool.StartProcess()

Common Helper Functions

This is a set of useful helper functions:

def myFunction(condition: bool) -> Iterable:
  myList = [3, 21, 5, 7]
  if condition:
    return myList[0:2]
  else
    return myList[1:3]

beginOfSequence = myFunction(True)
first = firstItem(beginOfSequence)


# 3

from pyTooling.Common import mergedicts

dictA = {"a": 11, "b": 12}
dictB = {"x": 21, "y": 22}

for key, value in mergedicts(dictA, dictB):
  pass

# ("a", 11)
# ("b", 12)
# ("x", 21)
# ("y", 22)

from pyTooling.Common import zipdicts

dictA = {"a": 11, "b": 12, "c": 13}
dictB = {"a": 21, "b": 22, "c": 23}

for key, valueA, valueB in zipdicts(dictA, dictB):
  pass


# ("a", 11, 21)
# ("a", 12, 22)
# ("a", 13, 23)

Common Classes

from pytest import mark
from unittest import TestCase

from pyTooling.Common import CurrentPlatform

class MyTests(TestCase):
  @mark.skipif(not CurrentPlatform.IsNativeWindows, reason="Skipped, if platform isn't native Windows.")
  def test_OnlyNativeWindows(self):
    pass

  @mark.skipif(not CurrentPlatform.IsMinGW64OnWindows, reason="Skipped, if platform isn't MinGW64.")
  def test_OnlyMinGW64(self):
    pass

  @mark.skipif(CurrentPlatform.IsPyPy, reason="getsizeof: not supported on PyPy")
  def test_ObjectSize(self):
    pass

from pyTooling.Versioning import SemanticVersion

version = SemanticVersion("2.5.4")

Todo

Needs example code

Configuration

Various file formats suitable for configuration information share the same features supporting: key-value pairs (dictionaries), sequences (lists), and simple types like string, integer and float. pyTooling provides an abstract configuration file data model supporting these features. Moreover, concrete configuration file format reader implementations are provided as well.

Todo

Needs example code

pass

Todo

Needs example code

pass

Todo

Needs example code

pass

Todo

Needs example code

pass

Data Structures

pyTooling also provides fast and powerful data structures offering object-oriented APIs:

Todo

Needs example code

pass

Todo

Needs example code

pass

Todo

Needs example code

pass

Graph

%%{init: { "flowchart": { "nodeSpacing": 15, "rankSpacing": 30, "curve": "linear", "useMaxWidth": false } } }%% graph LR A(A); B(B); C(C); D(D); E(E); F(F) ; G(G); H(H); I(I) A --> B --> E G --> F A --> C --> G --> H --> D D -.-> A D & F -.-> B I ---> E --> F --> D classDef node fill:#eee,stroke:#777,font-size:smaller; classDef node fill:#eee,stroke:#777,font-size:smaller; classDef node fill:#eee,stroke:#777,font-size:smaller;

A directed graph with backward-edges denoted by dotted vertex relations.

Statemachine

%%{init: { "flowchart": { "nodeSpacing": 15, "rankSpacing": 30, "curve": "linear", "useMaxWidth": false } } }%% graph TD A(Idle); B(Check); C(Prepare); D(Read); E(Finished); F(Write) ; G(Retry); H(WriteWait); I(ReadWait) A:::mark1 --> B --> C --> F F --> H --> E:::cur B --> G --> B G -.-> A --> C D -.-> A C ---> D --> I --> E -.-> A classDef node fill:#eee,stroke:#777,font-size:smaller; classDef cur fill:#9e9,stroke:#6e6; classDef mark1 fill:#69f,stroke:#37f,color:#eee;

A statemachine graph.

Tree

%%{init: { "flowchart": { "nodeSpacing": 15, "rankSpacing": 30, "curve": "linear", "useMaxWidth": false } } }%% graph TD R(Root) A(...) BL(Node); B(GrandParent); BR(Node) CL(Uncle); C(Parent); CR(Aunt) DL(Sibling); D(Node); DR(Sibling) ELN1(Niece); ELN2(Nephew) EL(Child); E(Child); ER(Child); ERN1(Niece);ERN2(Nephew) F1(GrandChild); F2(GrandChild) R:::mark1 --> A A --> BL & B & BR B --> CL & C & CR C --> DL & D & DR DL --> ELN1 & ELN2 D:::cur --> EL & E & ER DR --> ERN1 & ERN2 E --> F1 & F2 classDef node fill:#eee,stroke:#777,font-size:smaller; classDef cur fill:#9e9,stroke:#6e6; classDef mark1 fill:#69f,stroke:#37f,color:#eee;

Root of the current node are marked in blue.

Decorators

  • Abstract Method

    • @abstractmethod: Methods marked with @abstractmethod are abstract and need to be overwritten in a derived class.
      An abstract method might be called from the overwriting method.

    • @mustoverride: Methods marked with @mustoverride are abstract and need to be overridden in a derived class.
      It’s not allowed to call a mustoverride method.

  • Data Access

    • @readonly: Methods marked with @readonly get transformed into a read-only property.

    • ⚠BROKEN⚠: Methods with @classproperty decorator transform methods to class-properties.

  • Documentation

    • @export: Register a given function or class as publicly accessible in a module.
      Functions and classes exposed like this are also used by Sphinx extensions to (auto-)document public module members.

    • @InheritDocString: The decorator copies the doc-string from a given base-class to the annotated method.

  • Performance

    • @slotted: Classes marked with @slotted get transformed into classes using __slots__.
      This is achieve by exchanging the meta-class to ExtendedType.

    • @mixin: Classes marked with @mixin do not store their fields in __slots__.
      When such a mixin-class is inherited by a class using slots, the fields of the mixin become slots.

    • @singleton: Classes marked with @singleton get transformed into singleton classes.
      This is achieve by exchanging the meta-class to ExtendedType.

  • Miscellaneous

Todo

Needs example code

pass

Exceptions

Meta-Classes

pyTooling provides an enhanced meta-class called ExtendedType to replace the default meta-class type. It combines features like using slots, abstract methods and creating singletons by applying a single meta-class. In comparison, Python’s approach in to provide multiple specific meta-classes (see abc) that can’t be combined e.g. to a singleton using slots.

ExtendedType allows to implement slotted types, mixins, abstract and override methods and singletons, and combinations thereof. Exception messages in case of errors have been improved too.

Slotted types significantly reduce the memory footprint by 4x and decrease the class field access time by 10..25%. While setting up slotted types needed a lot of manual coding, this is now fully automated by this meta-class. It assumes, annotated fields are going to be slots. Moreover, it also takes care deferred slots in multiple-inheritance scenarios by marking secondary base-classes as mixins. This defers slot creation until a mixin is inherited.

class MyClass(metaclass=ExtendedType):

A class definition using the ExtendedType meta-class. I can now implement abstract methods using the decorators @abstractmethod or @mustoverride.

class MyClass(metaclass=ExtendedType, singleton=True):

A class defined with enabled singleton behavior allows only a single instance of that class to exist. If another instance is going to be created, a previously cached instance of that class will be returned.

class MyClass(metaclass=ExtendedType, slots=True):

A class defined with enabled slots behavior stores instance fields in slots. The meta-class, translates all type-annotated fields in the class definition to slots. Slots allow a more efficient field storage and access compared to dynamically stored and accessed fields hosted in __dict__. This improves the memory footprint as well as the field access performance of all class instances. This behavior is automatically inherited to all derived classes.

class MyClass(metaclass=ExtendedType, slots=True, mixin=True):

A class defined with enabled mixin behavior collects type-annotated instance fields so they can be added to slots in an inherited class. Thus, slots are not created for mixin-classes but deferred in the inheritance hierarchy.

class MyClass(SlottedObject):

A class definition deriving from SlottedObject will bring the slotted type behavior to that class and all its derived classes.

Todo

Needs example code

def

Todo

Needs example code

def

Todo

Needs example code

def

Packaging

A set of helper functions to describe a Python package for setuptools.

Todo

Needs example code

Terminal

The pyTooling.TerminalUI package offers a set of helpers to implement a text user interface (TUI) in a terminal. It’s designed on the idea that command line programs emit one line of text per message. Each message can be categorized as normal text, warnings, errors, and many more.

Therefore, this package offers a LineTerminal implementation, derived from a basic Terminal class. Of cause, it also includes colored outputs based on colorama.

Todo

Terminal helpers.

Todo

Needs example code

Timer

A Timer class to measure and accumulate code execution times.

Todo

Needs example code

Contributors

License

This Python package (source code) is licensed under Apache License 2.0.
The accompanying documentation is licensed under Creative Commons - Attribution 4.0 (CC-BY 4.0).