Overview

CLIAbstraction offers an abstraction layer for command line programs, so they can be used easily in Python. There is no need for manually assembling parameter lists or considering the order of parameters. All parameters like -v or --value=42 are described as CommandLineArgument instances on a Program class. Each argument class like ShortFlag or PathArgument knows about the correct formatting pattern, and if needed about necessary type conversions. A program instance can be converted to an argument list suitable for subprocess.Popen, which passes each argument to the program without a shell - so no argument needs escaping.

While a user-defined command line program abstraction derived from Program only takes care of maintaining and assembling parameter lists, a more advanced base-class, called Executable, is offered with embedded Popen behavior.

Design Goals

The main design goals are:

Offer access to CLI programs as Python classes.
Abstract CLI arguments (a.k.a. parameter, option, flag, …) as members on such a Python class.
Abstract differences in operating systems like argument pattern (POSIX: -h vs. Windows: /h), path delimiter signs (POSIX: / vs. Windows: \) or executable names.
Derive program variants from existing programs.
Assemble parameters as list for handover to subprocess.Popen, in the order the program declares them.
Launch a program with Popen and hide the complexity of Popen.
Get a generator object for line-by-line output reading to enable postprocessing of outputs.

Example

The following example implements a portion of the git program and its commit sub-command.

A new class Git is derived from pyTooling.CLIAbstraction.Executable.
A class variable _executableNames is set, to specify different executable names based on the operating system.
Nested classes are used to describe arguments and flags for the Git program.
These nested classes are annotated with the @CLIArgument attribute, which is used to register the nested classes in an ordered lookup structure. This declaration order is also used to order arguments when converting to a list for Popen.

Usage of Git

# Create a program instance and set common parameters.
git = Git()
git[git.FlagVerbose] = True

# Derive a variant of that pre-configured program.
commit = git.GetCommitTool("Bumped dependencies.", amend=True)

# Launch the program and parse outputs line-by-line.
commit.StartProcess()
for line in commit.GetLineReader():
  print(line)

Declaration of Git

from pyTooling.CLIAbstraction import CLIArgument, Executable
from pyTooling.CLIAbstraction.Argument import PathListArgument
from pyTooling.CLIAbstraction.Command import CommandArgument
from pyTooling.CLIAbstraction.Flag import LongFlag
from pyTooling.CLIAbstraction.ValuedTupleFlag import ShortTupleFlag

class Git(Executable):
  _executableNames: ClassVar[Dict[str, str]] = {
    "Darwin":  "git",
    "FreeBSD": "git",
    "Linux":   "git",
    "Windows": "git.exe"
  }

  @CLIArgument()
  class FlagVerbose(LongFlag, name="verbose"):
    """Print verbose messages."""

  @CLIArgument()
  class CommandCommit(CommandArgument, name="commit"):
    """Command to commit staged files."""

  @CLIArgument()
  class FlagAmend(LongFlag, name="amend"):
    """Replace the tip of the current branch."""

  @CLIArgument()
  class ValueCommitMessage(ShortTupleFlag, name="m"):
    """Specify the commit message."""

  @CLIArgument()
  class ArgumentPaths(PathListArgument):
    """Files to commit."""

  def GetCommitTool(
    self,
    message: str,
    amend: bool = False,
    paths: Nullable[Iterable[Path]] = None
  ) -> "Git":
    """Derive a commit command from this program."""
    tool = self.__class__(executablePath=self._executablePath)
    self._CopyParameters(tool)

    tool[tool.CommandCommit] = True
    tool[tool.ValueCommitMessage] = message
    if amend:
      tool[tool.FlagAmend] = True
    if paths is not None:
      tool[tool.ArgumentPaths] = paths

    return tool

Programm API

Condensed definition of class Program:

@export
class Program(metaclass=ExtendedType, slots=True):

  def __init_subclass__(cls, *args: Any, **kwargs: Any) -> None:
    ...

  def __init__(
    self,
    executablePath: Nullable[Path] = None,
    binaryDirectoryPath: Nullable[Path] = None,
    dryRun: bool = False,
  ) -> None:
    ...

  @readonly
  def DryRunMessages(self) -> list[str]:
    ...

  def LogDryRun(self, message: str) -> None:
    ...

  def __getitem__(self, key: type[CommandLineArgument]) -> CommandLineArgument:
    ...

  def __setitem__(self, key: type[CommandLineArgument], value: CommandLineArgument) -> None:
    ...

  @readonly
  def Path(self) -> Path:
    ...

  def ToArgumentList(self) -> list[str]:
    ...

  def __repr__(self) -> str:
    ...

  def __str__(self) -> str:
    ...

Executable API

Condensed definition of class Executable:

@export
class Executable(Program):

  def __init__(
    self,
    executablePath: Nullable[Path] = None,
    binaryDirectoryPath: Nullable[Path] = None,
    workingDirectory: Nullable[Path] = None,
    environment: Nullable[Environment] = None,
    dryRun: bool = False,
  ) -> None:
    ...

  def StartProcess(self, environment: Nullable[Environment] = None) -> None:
    ...

  def Send(self, line: str, end: str = "\n") -> None:
    ...

  def GetLineReader(self) -> Generator[str, None, None]:
    ...

  def Wait(self, timeout: Nullable[float] = None, kill: bool = False) -> Nullable[int]:
    ...

  def Terminate(self) -> Nullable[int]:
    ...

  @readonly
  def ExitCode(self) -> int:
    ...

Competing Solutions

The packages below run a program from Python, but describe its command line as strings. pyTooling describes it as classes: every argument of a program is a nested class of its Program, whose base class - ShortFlag, LongValuedFlag, WindowsFlag, … - formats it as -v, --value=42 or /v. The arguments are listed in the order they are declared, and the executable’s name is chosen per platform.

subprocess

Source: the standard library’s subprocess.

Disadvantages

A command line is a list of strings, assembled by the caller in the right order and in each program’s syntax.

Standoff

ToArgumentList() returns such a list, and Executable starts it with Popen.

Advantages

No dependency, and every option of Popen is available.

plumbum

Source: plumbum, on PyPI as plumbum.

Disadvantages

Arguments are bound as strings - local["ls"]["-l"] - so a flag’s syntax is written at every call.

Advantages

Pipelines (|), redirection (<, >), background execution, and commands run on a remote machine over SSH.
A toolkit for writing command line applications.

sh

Source: sh, on PyPI as sh.

Disadvantages

Windows is not supported.
Keyword arguments become flags by one rule - one letter -o value, more letters --name - so a program using another syntax, like /flag or -flag=value, is called with strings.

Advantages

A program is called like a function, sh.git.commit(m="message"), without declaring it first.

invoke

Source: invoke, on PyPI as invoke.

Disadvantages

A command is one string run by a shell, so quoting and the shell’s syntax are the caller’s.

Standoff

A task runner - tasks with their own command line - more than a program abstraction.

Consumers

This abstraction layer is used by:

✅ Wrap command line interfaces of EDA tools (Electronic Design Automation) in Python classes.
pyEDAA.CLITool