Filesystem

The pyTooling.Filesystem package provides fast and simple access to directory statistics like file sizes, accumulated directory sizes, symlinks, hardlinks, etc.

Features

  • Scan a directory and its subdirectories for files and create a in-memory filesystem view (directories, files, symbolic links, hard links).

  • Identify filenames pointing to the same file (a.k.a hard links).

  • Compute directory sizes by aggregating file sizes.

Missing Features

  • tbd

Planned Features

  • tbd

Out of Scope

  • tbd

By Feature

Danger

Accessing internal fields of a node is strongly not recommended for users, as it might lead to a corrupted tree data structure. If a power-user wants to access these fields, feel free to use them for achieving a higher performance, but you got warned 😉.

Root Reference

Every element of a scanned filesystem - a Directory, a Filename, a SymbolicLink - knows the Root it belongs to, through Root. The root knows itself, so the reference is never None within a scan and needs no special case at the top:

from pathlib import Path
from pyTooling.Filesystem import Root

root = Root(Path("/home/user/project"))

root.Root is root                        # True

source = next(root.Subdirectories)
source.Root is root                      # True
next(source.RegularFiles).Root is root   # True

The reference exists because the interesting questions are asked of the root and answered for the whole scan: which symbolic links are broken, how many distinct files there are, how much space would be needed without hard links. An element deep in the tree can reach those answers without being handed the root separately.

On the root

Answers

BrokenSymbolicLinks

the links whose target doesn’t exist

UnconnectedSymbolicLinks

the links whose target lies outside the scanned tree

TotalUniqueFileCount

distinct file storage objects, hard links counted once

TotalHardLinkCount

directory entries pointing at multiply-linked files

Attention

Root is writable, and assigning it is how the scan wires an element into its tree - not something user code should do. Assigning None raises ValueError and assigning anything that is not a Root raises TypeError, but neither protects a tree from being re-parented into an inconsistent state.

Parent Reference

Parent is the containing directory, so the tree can be walked upwards as well as downwards. Path is built from that chain, which is why an element knows its full path without storing one:

source = next(root.Subdirectories)

source.Parent is root      # True
source.Name                # 'source'
source.Path                # PosixPath('/home/user/project/source')

file = next(source.RegularFiles)
file.Parent is source      # True

Walking downwards is done with the generators on Directory - Subdirectories, Files, RegularFiles, SymbolicLinks for one level, and IterateDirectories() / IterateFiles() for the whole subtree.

Hint

A File is the storage object and a Filename is a directory entry naming it. That is the distinction hard links are built on: one File with several Parents. So an element’s parent is its directory, while a file’s Parents are the names it is reachable by.

Size

Size is in bytes, and a directory’s size is the sum of everything below it. AggregateSizes() computes those sums for the whole tree:

root = Root(Path("/home/user/project"))
root.AggregateSizes()

root.Size                  # every directory entry, hard links counted each time
next(root.Subdirectories).Size

Hard links make “the size of a directory” ambiguous, so three properties answer three different questions rather than one of them pretending to be the answer. For a tree of a.txt (100 bytes), b.txt (200 bytes), c.txt (50 bytes), a second directory entry hard-linked to a.txt, and a symbolic link to b.txt:

Property

Value

Question it answers

Size

450

How much do the directory entries add up to? The hard link is counted again - 100 + 200 + 50 + 100.

Size2

100

How much of that is hard-linked content, counted once per file?

Size3

200

How much would the hard-linked content cost on a filesystem without hard links? 100 × 2 entries.

So Size - Size2 is what hard-linking saves, and the counts alongside them tell the same story in files: TotalFileCount is 5, TotalUniqueFileCount is 3, and TotalHardLinkCount is 2.

Hint

ScanDuration and AggregateDuration report what the two phases cost, measured with Stopwatch. On a large tree the scan dominates, because it is the part that touches the filesystem.

Competing Solutions

Directory Tree

Source: Directory Tree

Todo

FILESYS::Directory-Tree write comparison here.

Disadvantages

  • …

Standoff

  • …

Advantages

  • …

folderstats

Source: folderstats

Todo

FILESYS::folderstats write comparison here.

Disadvantages

  • …

Standoff

  • …

Advantages

  • …

dutree

Source: dutree

Todo

FILESYS::dutree write comparison here.

Disadvantages

  • …

Standoff

  • …

Advantages

  • …