Filesystem
The pyTooling.Filesystem package provides fast and simple access to directory statistics like file sizes,
accumulated directory sizes, symlinks, hardlinks, etc.
Features
Scan a directory and its subdirectories for files and create a in-memory filesystem view (directories, files, symbolic links, hard links).
Identify filenames pointing to the same file (a.k.a hard links).
Compute directory sizes by aggregating file sizes.
Delete a scanned tree no longer needed, so its memory is freed without waiting for the garbage collector.
Missing Features
tbd
Planned Features
tbd
Out of Scope
tbd
By Feature
Danger
Accessing internal fields of a node is strongly not recommended for users, as it might lead to a corrupted tree data structure. If a power-user wants to access these fields, feel free to use them for achieving a higher performance, but you got warned 😉.
Root Reference
Every element of a scanned filesystem - a Directory, a
Filename, a SymbolicLink - knows the
Root it belongs to, through Root. The root knows
itself, so the reference is never None within a scan and needs no special case at the top:
from pathlib import Path
from pyTooling.Filesystem import Root
root = Root(Path("/home/user/project"))
root.Root is root # True
source = next(root.Subdirectories)
source.Root is root # True
next(source.RegularFiles).Root is root # True
The reference exists because the interesting questions are asked of the root and answered for the whole scan: which symbolic links are broken, how many distinct files there are, how much space would be needed without hard links. An element deep in the tree can reach those answers without being handed the root separately.
On the root |
Answers |
|---|---|
the links whose target doesn’t exist |
|
the links whose target lies outside the scanned tree |
|
distinct file storage objects, hard links counted once |
|
directory entries pointing at multiply-linked files |
Attention
Root is writable, and assigning it is how the scan wires an element into its
tree - not something user code should do. Assigning None raises ValueError and assigning anything that
is not a Root raises TypeError, but neither protects a tree from being
re-parented into an inconsistent state.
Parent Reference
Parent is the containing directory, so the tree can be walked upwards as
well as downwards. Path is built from that chain, which is why an element
knows its full path without storing one:
source = next(root.Subdirectories)
source.Parent is root # True
source.Name # 'source'
source.Path # PosixPath('/home/user/project/source')
file = next(source.RegularFiles)
file.Parent is source # True
Walking downwards is done with the generators on Directory -
Subdirectories, Files,
RegularFiles, SymbolicLinks for one
level, and IterateDirectories() /
IterateFiles() for the whole subtree.
Size
Size is in bytes, and a directory’s size is the sum of everything below it.
AggregateSizes() computes those sums for the whole tree:
root = Root(Path("/home/user/project"))
root.AggregateSizes()
root.Size # every directory entry, hard links counted each time
next(root.Subdirectories).Size
Hard links make “the size of a directory” ambiguous, so three properties answer three different questions rather
than one of them pretending to be the answer. For a tree of a.txt (100 bytes), b.txt (200 bytes),
c.txt (50 bytes), a second directory entry hard-linked to a.txt, and a symbolic link to b.txt:
Property |
Value |
Question it answers |
|---|---|---|
|
How much do the directory entries add up to? The hard link is counted again - |
|
|
How much of that is hard-linked content, counted once per file? |
|
|
How much would the hard-linked content cost on a filesystem without hard links? |
So Size - Size2 is what hard-linking saves, and the counts alongside them tell the same story in files:
TotalFileCount is 5,
TotalUniqueFileCount is 3, and
TotalHardLinkCount is 2.
Hint
ScanDuration and
AggregateDuration report what the two phases cost, measured with
Stopwatch. On a large tree the scan dominates, because it is the part that touches
the filesystem.
Deleting a Scope
The elements of a scanned tree reference each other: a directory its subdirectories, filenames and symbolic links,
each of them its parent and the root, a filename its file object and the file object all its filenames, and the root
every file object. These are reference cycles, so when the last reference to a Root
is dropped, Python’s reference counting can’t free the tree. The memory is freed only by the cyclic garbage collector
(gc), which runs from time to time, or when a program calls gc.collect(). A full collection visits every
object of the program, not only the tree - and a scan creates one object per directory entry.
Delete() disconnects all elements: no element has a root, a parent, subelements or
link sources any more, no filename refers to a file object and no file object to a filename. This includes the
elements Collapse() removed from their directories, which the root still reaches
through the file objects. Afterwards, nothing but the program’s own references keeps the elements alive, so reference
counting frees them as soon as these are dropped.
root = Root(Path("/home/user/project"))
root.AggregateSizes()
print(root.Size)
root.Delete()
del root # freed now, not at the next garbage collection
Deleting pays off where a program scans directory trees again while it keeps running.
Competing Solutions
Directory Tree
Source: Directory Tree
Todo
FILESYS::Directory-Tree write comparison here.
Disadvantages
…
Standoff
…
Advantages
…
folderstats
Source: folderstats
Todo
FILESYS::folderstats write comparison here.
Disadvantages
…
Standoff
…
Advantages
…
dutree
Source: dutree
Todo
FILESYS::dutree write comparison here.
Disadvantages
…
Standoff
…
Advantages
…