Overview

pyTooling.REST reads and writes JSON resources of a REST API, with the standard library and nothing else.

from pyTooling.REST import RESTClient

client = RESTClient("https://api.example.org", token, headers={"Accept": "application/json"})

thing = client.GetJSONObject("things/4711")[0]
client.PatchJSONObject("things/4711", {"state": "closed"})

Every REST API is read the same way, and none of that is knowledge about a particular service: a request carrying a bearer token, an answer that has to be a JSON object, a transient failure that is worth another attempt, and a collection that arrives one page at a time.

Resources

A client is constructed with the API’s base URL, and every request names a resource path below it - the API is stated once, not in every call. The two are composed by URL.__truediv__, which puts a query the resource path carries where a query belongs instead of leaving it inside a path element.

The base URL is a URL, so a value that names no scheme or no host is refused where it is given rather than where it is requested. It may be given as a URL, which is then kept as it is, or as a string, which is parsed once; either way a trailing slash is removed, so https://example.org/api/v3/ and https://example.org/api/v3 are the same client.

https://example.org/api// is refused, though. RFC 3986 allows an empty path element - segment = *pchar - so it is a valid URL and Parse() reads it, and normalization doesn’t collapse one either. As a base URL it is a typo, and every request would carry it as a double slash.

Method

What it does

GetJSONObject()

Read a resource, and say where its next page is.

PostJSONObject()

Create a resource in a collection.

PutJSONObject()

Replace a resource.

PatchJSONObject()

Change some fields of a resource.

DeleteResource()

Delete a resource.

A write sends its JSON object as the body and answers with the API’s answer, or None where the API answers with no body - which is what a deletion usually does. The body’s Content-Type is a member of MediaType, which is its own media type, so it is written where a header’s value is expected. The method itself is a member of http.HTTPMethod, the standard library’s own enumeration of what RFC 9110 defines.

An answer that isn’t a JSON object - an array, an HTML error page, a truncated body - raises a RESTError naming the URL, rather than being handed on as something a caller has to check. The answer’s media type is checked first, by MediaType.Matches: it ignores the header’s parameters, compares the name case-insensitively, and counts the RFC 6839 structured syntax suffix, so application/vnd.github+json is application/json.

Pagination

GetJSONObject() answers with the resource path of the next page, or None when the answer names none, so a collection is read by a loop that ends by itself:

resourcePath = "things?per_page=100"
while resourcePath is not None:
  page, resourcePath = client.GetJSONObject(resourcePath)
  for thing in page["things"]:
    print(thing["name"])

The next page is read from the RFC 8288 Link header, which is how a paginated REST API says where its collection continues. A Link header pointing outside APIURL is rejected instead of followed: a token is only sent to the API it was given for, and a next page elsewhere would take it along.

Retries

A request failing transiently is tried again: a status in TRANSIENT_HTTP_STATUS (429, 500, 502, 503 and 504), a timeout, or an API that can’t be reached. The pause grows exponentially - RetryDelay doubled for every earlier attempt - or lasts as long as a Retry-After header demands, if that is longer, but never longer than MAXIMUM_RETRY_AFTER. RFC 9110 writes that header as a non-negative number of seconds or as an HTTP-date; only the first is honored, because a date says when to try again and not how long a backoff should be.

A request failing with any other HTTP status, like 401, 403 or 404, isn’t tried again, because another attempt can’t succeed. Its error carries what the API said: the message field of the answer’s body, and the number of attempts when there was more than one.

Important

A request that isn’t idempotent is never tried again. A POST the API did carry out, but whose answer was lost on the way back, would create the resource a second time; RFC 9110 doesn’t call PATCH idempotent either. GET, PUT and DELETE are retried, POST and PATCH are sent once.

An instance holds no state beyond what it was constructed with, and every request builds its own headers, so one client can serve several threads. A pause before a retry blocks only the thread waiting for that answer.

A client for one API

A derived class is what turns the generic client into a reader of one service: it fixes the base URL and the headers that service expects, and adds the requests it offers as methods.

_AddErrorNotes() is the hook for what an API states about itself: what a status means there - a 404 from an API that reads repositories is worth a different sentence than a 404 from one that reads invoices.

The Authorization header carries the bearer scheme of RFC 6750, which is what a token-based API expects and what an OAuth 2.0 flow’s access token is used with once the flow handed one out. An API expecting something else - the basic scheme of RFC 7617, say - overrides _RequestHeaders(), which is where every header of a request is decided.

class ExampleClient(RESTClient):
  def __init__(self, token = None) -> None:
    super().__init__("https://api.example.org", token, headers={"Accept": "application/json"})

  def ReadThing(self, name: str) -> JSONObject:
    return self.GetJSONObject(f"things/{name}")[0]

  def _AddErrorNotes(self, error: RESTError, status: int) -> None:
    if status in (401, 403, 404):
      error.add_note("Check the thing's name, and that the token may read it.")

The workflow run reader of pyTooling.GitHub is such a reader.

Competing Solutions

pyTooling.REST is no general HTTP client: it reads and writes JSON objects of one API, synchronously, and opens a connection per request. What it adds to the standard library is what every REST API needs - retries of transient failures, pagination by the Link header, and an error naming the URL. The packages below are general HTTP clients, so a caller builds these parts on top of them.

requests

Source: requests, on PyPI as requests.

Disadvantages

  • Four dependencies: charset_normalizer, idna, urllib3 and certifi.

  • By default, a failed request isn’t tried again, and a retry never follows an HTTP status. Retrying on a status needs urllib3’s Retry mounted on an HTTPAdapter.

  • Response.json() returns whatever JSON the answer holds; checking it is a JSON object is left to the caller. Response.links parses the Link header, but following the next page is the caller’s loop.

Advantages

  • Sessions reuse connections and keep cookies, and authentication schemes, file uploads and streamed bodies are included.

  • Any media type, not just JSON objects.

urllib3

Source: urllib3, on PyPI as urllib3.

Disadvantages

  • Retry retries on an HTTP status only if the status is listed in status_forcelist, or if the answer to 413, 429 or 503 carries a Retry-After header.

  • Reading JSON, checking the media type and pagination are the caller’s.

Standoff

  • No dependencies either.

  • The same rule for which requests are tried again: by default DELETE, GET, HEAD, OPTIONS, PUT and TRACE - not POST or PATCH. The backoff doubles too, capped at 120 seconds.

Advantages

  • Thread-safe connection pools, and a random jitter for the backoff.

HTTPX

Source: httpx, on PyPI as httpx.

Disadvantages

  • Four dependencies: anyio, certifi, httpcore and idna.

  • The transport’s retries retries a failed connection only - ConnectError and ConnectTimeout. For a 503 Service Unavailable, its documentation suggests a general retry package like tenacity.

Advantages

  • One API for synchronous and asynchronous code (Client, AsyncClient), and HTTP/2 with http2=True.

aiohttp

Source: aiohttp, on PyPI as aiohttp.

Disadvantages

  • For asyncio only, with seven dependencies (nine before Python 3.13).

Advantages

  • Many requests in flight at once on one thread, and a server besides the client.