Uniform Resource Identifier (URI) and Locator (URL)
pyTooling.URI implements a Uniform Resource Identifier (URI, RFC 3986) and, derived from it, a Uniform
Resource Locator (URL) - a URI with an authority (//host) and a hierarchical path - and a Uniform Resource Name
(URN, RFC 8141).
url = URL.Parse("http://paebbels:xxx@semaphore.plc2.de:5000/api/v1/semaphore?name=Riviera&foo=bar#page2")
print(url.Scheme) # Protocols.HTTP
print(url.User) # paebbels
print(url.Password) # xxx
print(url.Host) # semaphore.plc2.de:5000
print(url.Path) # /api/v1/semaphore
print(url.Query["name"].Value) # Riviera
print(url.Fragment) # page2
URIs
A URI is a scheme, :, a scheme-specific part, an optional query and an optional fragment -
the generic syntax every URI follows, whatever its scheme: mailto:user@example.org, tel:+49-761-123456,
urn:isbn:0451450523. URI.Parse reads every scheme; the scheme is stored in lower
case, as schemes are case-insensitive.
A URI whose scheme-specific part starts with // (an authority) and whose scheme is one of
Protocols is a URL: URI.Parse reads it by
URL.Parse and returns a URL, which derives from
URI and splits the scheme-specific part into user, password, host and path.
uri = URI.Parse("mailto:user@example.org?subject=Hi")
uri.Scheme # 'mailto'
uri.SchemeSpecificPart # 'user@example.org'
uri.Query["subject"].Value # 'Hi'
url = URI.Parse("https://example.org/api")
isinstance(url, URL) # True
url.Scheme # Protocols.HTTPS
url.SchemeSpecificPart # '//example.org/api'
URI.Scheme is the scheme’s name; a URL answers the
Protocols member instead, which also tells whether the scheme is secured.
URNs
A URN (Uniform Resource Name, RFC 8141) names a resource without locating it:
urn:<namespace-id>:<namespace-specific-string>[?+r-component][?=q-component][#f-component]
The namespace identifier (isbn, ietf, uuid, …) has 2 to 32 letters, digits or - and decides the
structure of the namespace-specific string; a : in it separates parts only by that namespace’s rules, so a URN is no
hierarchical path. The r-component carries parameters for a resolver, the q-component parameters for the named
resource - read as key=value pairs into Query, as a URL’s query -, and the f-component is
a fragment. URN.Parse reads a URN, and so does
URI.Parse for the scheme urn.
urn = URI.Parse("urn:ietf:rfc:3986")
urn.NamespaceID # 'ietf'
urn.NamespaceSpecificString # 'rfc:3986'
urn = URN.Parse("urn:example:weather?+lang=de?=time=now#temperature")
urn.Resolution # 'lang=de'
urn.Query["time"].Value # 'now'
urn.Fragment # 'temperature'
Parsing
URL.Parse reads a URL as RFC 3986 writes an absolute URI with an
authority: a scheme followed by ://, a host with an optional user, password and port, then a path starting with a
slash, a query and a fragment. Only a file or unix URL may leave the host empty, and then names an absolute
path, e.g. file:///etc/hosts.
scheme://[user[:password]@]host[:port][/path/to/file][?query][#fragment]
A string that isn’t such a URL raises URLError: github.com names no scheme,
http:///path no host. A relative reference like things/4711 is no URL either; it is appended to one with
URL.__truediv__.
What is typed into a browser’s address bar often has no scheme. URL.Parse(text, defaultScheme=Protocols.HTTPS)
prepends one to a URL without scheme://, as a browser does: www.example.org/path is read as
https://www.example.org/path, //example.org/path as https://example.org/path. An error found in the
completed URL gets a note naming the prepended scheme. Without defaultScheme, parsing stays strict.
A port is a number after the host’s :. http://host:/x (no port) and ssh://git@github.com:org/repo.git (a
path instead of a port) raise URLError, and its note gives the URL as it was
probably meant: http://host/x, ssh://git@github.com/org/repo.git. Git’s and scp’s SCP-like address
git@github.com:org/repo.git names no scheme; the notes offer SCPAddress and give
its URL ssh://git@github.com/~/org/repo.git. In file://C:/Windows, the drive letter stands where
the host is; the note gives file:///C:/Windows.
Internationalized Resource Identifiers (IRI)
A URL of RFC 3986 holds ASCII characters only. An Internationalized Resource Identifier (IRI, RFC 3987) is
the same syntax with Unicode characters allowed, and an internationalized domain name (IDN) in its host is carried as
A-labels, the ASCII form IDNA 2008 defines: xn-- followed by the Punycode of the label.
URL.Parse(url, iri=True) reads an IRI and maps it to a URL, as
RFC 3987, section 3.1 describes:
A Unicode host is converted to its A-labels by IDNA 2008 (RFC 5890, RFC 5891), after the mapping of UTS #46.
Every other non-ASCII character - in the user information, the path, the query and the fragment - is percent-encoded as UTF-8.
So the parsed URL is an RFC 3986 URL either way, and str() renders it in
ASCII. Host.UnicodeHostname decodes the A-labels again, for a
user to read.
Example
from pyTooling.URI import URL
url = URL.Parse("https://www.müller.de/", iri=True)
print(url) # https://www.xn--mller-kva.de/
print(url.Host.Hostname) # www.xn--mller-kva.de
print(url.Host.UnicodeHostname) # www.müller.de
url = URL.Parse("https://de.wikipedia.org/wiki/Straße", iri=True)
print(url) # https://de.wikipedia.org/wiki/Stra%C3%9Fe
url = URL.Parse("https://www.müller.de/")
# URLError: Syntax error when parsing URL 'https://www.müller.de/'.
# A URL holds ASCII characters only. Read an IRI, which holds Unicode characters, with 'iri=True'.
The host is mapped the way browsers map it. The WHATWG URL Standard
converts a host with domain to ASCII, which is UTS #46 processing: case folding (MÜLLER.de →
xn--mller-kva.de), width mapping of full-width letters and the ideographic full stop (。) as a dot, then IDNA
2008. It processes non-transitionally, as IDNA 2008 does: ß stays ß, so straße.de is
xn--strae-oqa.de - the domain DENIC registers -, where IDNA 2003 and Python’s built-in idna codec map it to
strasse.de, a different domain. A character IDNA 2008 disallows, e.g. a symbol as in ☃.net which IDNA 2003
accepted, raises URLError with IDNA’s reason as note.
Specification |
What it decides here |
|---|---|
The syntax of a URL (URI), in ASCII. |
|
The IRI and its mapping to a URI: percent-encode UTF-8, convert the host with IDNA. |
|
IDNA 2008: A-labels and U-labels, and which characters a label may hold. |
|
The mapping before IDNA 2008: case folding, width mapping, label separators; non-transitional. |
|
What browsers do: domain to ASCII with UTS #46, which |
The conversion needs the idna package, installed by the extra idna
(pip install pyTooling[idna]). Without it, converting a Unicode host or decoding an A-label raises
MissingDependencyError; an IRI whose host is ASCII, and an ASCII host without A-labels,
need no package.
Query
RFC 3986 puts no structure into a query. The common form are key=value pairs separated by &, and a key may
repeat: a form with a multi-select or a group of checkboxes sends tag=a&tag=b.
URL.Query holds the parameters by key, each a
QueryParameter:
A key written once is a
KeyValuePair.A key written several times is a
KeySequencePairholding every value in order.
Both answer Value and
Values, so a reader needn’t know which one it got.
Value of a repeated key is its last value - what a dictionary
of the pairs would keep -, and Values holds all of them; for a key
written once, it holds the one value.
url = URL.Parse("https://example.org/search?tag=python&tag=vhdl&page=2&verbose")
url.Query["page"].Value # '2'
url.Query["tag"].Value # 'vhdl'
url.Query["tag"].Values # ('python', 'vhdl')
url.Query["verbose"].Value # None - a parameter without '='
A parameter written without = has the value None, one with an empty value (?verbose=) the value "". A
URL with an empty query (https://example.org/?) has an empty dictionary, a URL without a query None.
str() writes every value back. The values of a key are written
together, at the position of the key’s first occurrence: ?tag=a&page=2&tag=b is written
?tag=a&tag=b&page=2. A URL built by hand takes its parameters as an iterable; a key may occur once:
url = URL(
Protocols.HTTPS,
Path.Parse("/search"),
Host("example.org"),
query=(KeySequencePair("tag", ("python", "vhdl")), KeyValuePair("page", "2"))
)
# https://example.org/search?tag=python&tag=vhdl&page=2
SCP-like Addresses
scp, rsync and Git also write a remote file or repository as [user@]host:[path], e.g.
git@github.com:org/repo.git (scp(1), Git’s scp-like syntax). That is no URL:
it names no scheme, no port, no query and no fragment, and a relative path starts in the user’s home directory,
where a URL’s path starts at the root.
SCPAddress.Parse reads it into an
SCPAddress - User,
Host and Path -, and
ToURL() converts it to the ssh:// URL naming the same location.
address = SCPAddress.Parse("git@github.com:org/repo.git")
address.User # 'git'
str(address.Host) # 'github.com'
str(address.Path) # 'org/repo.git' - relative: in the home directory
str(address.ToURL()) # 'ssh://git@github.com/~/org/repo.git'
str(SCPAddress.Parse("deploy@server:/srv/www").ToURL()) # 'ssh://deploy@server/srv/www'
A URL can’t express a path in the home directory; ToURL() writes it the way
Git does, as /~/, and a path starting with ~user keeps it. Not every SSH server expands ~; Git does.
As in Git, the first : separates the host, so the part before it holds no /: ./a:b is no SCP-like address.
A single letter before the : is a Windows drive letter (C:/Windows) and raises
URLError, as does an address with a scheme - a URL is read by
URL.Parse.