Python: Calling Web APIs with urllib

Many services provide data through a web API. Python can make simple HTTP requests using the standard library, which is useful on a small Debian machine where keeping dependencies to a minimum matters.

Make a GET Request

import json
from urllib.request import Request, urlopen

url = "https://api.github.com/repos/python/cpython"
request = Request(
    url,
    headers={"User-Agent": "python-learning-example"},
)

with urlopen(request, timeout=10) as response:
    data = json.load(response)

print(data["full_name"])
print(data["stargazers_count"])

The server may reject requests without a user agent, and every network request should have a timeout so a disconnected field node does not wait forever.

Handle Network Errors

from urllib.error import HTTPError, URLError

try:
    with urlopen(request, timeout=10) as response:
        data = json.load(response)
except HTTPError as error:
    print(f"Server returned HTTP {error.code}")
except URLError as error:
    print(f"Could not reach the server: {error.reason}")

Do not hide every exception with a blanket except Exception. Handling the errors you expect makes failures easier to diagnose.

Query Parameters

from urllib.parse import urlencode

params = urlencode({"q": "raspberry pi", "page": 1})
url = "https://example.invalid/search?" + params
print(url)

Use urlencode rather than placing user input directly into a URL. It correctly escapes spaces and other special characters.

Respect the Service

  • Read the API documentation before writing a client.
  • Use timeouts and handle temporary failures.
  • Do not poll more often than the service allows.
  • Cache data when it does not need to be fresh every second.
  • Never put passwords or API keys directly in source code.

Decode Only What You Need

API responses can contain much more data than the program needs. Check that the fields you depend on are present and keep the conversion close to the request:

def repository_summary(data):
    return {
        "name": data.get("full_name", "unknown"),
        "stars": int(data.get("stargazers_count", 0)),
        "language": data.get("language") or "not listed",
    }

Using get() is appropriate for optional fields. For a required field, raise a clear error instead of quietly producing an incomplete report.

Retry Temporary Failures Carefully

import time
from urllib.error import URLError

for attempt in range(3):
    try:
        with urlopen(request, timeout=10) as response:
            data = json.load(response)
        break
    except URLError:
        if attempt == 2:
            raise
        time.sleep(2 ** attempt)

Retries are for temporary network failures, not for invalid credentials or a malformed request. Keep the number of attempts small so an offline Pi does not spend all day retrying.

Keep Secrets Out of Source

If an API needs a token, read it from an environment variable or a protected configuration file:

import os

token = os.environ.get("SERVICE_TOKEN")
if not token:
    raise RuntimeError("SERVICE_TOKEN is not configured")

Do not put real tokens in examples, Git repositories, shell history, or messages sent over the mesh.

References