Automation Scripting for Operations Questions

Approach

At 'thousands of REST endpoints,' the design has to bound resource usage explicitly at every layer: connection reuse (not opening a new connection per request), a hard concurrency ceiling, and a shutdown path that doesn't abandon in-flight work.

import asyncio, random
import aiohttp

async def call_with_backoff(session, url, sem, max_retries=3):
    async with sem:  # bounds total concurrent in-flight requests
        attempt = 0
        while True:
            attempt += 1
            try:
                async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
                    if resp.status in (429, 500, 502, 503, 504):
                        raise aiohttp.ClientResponseError(
                            resp.request_info, resp.history, status=resp.status)
                    return await resp.json()
            except (aiohttp.ClientError, asyncio.TimeoutError):
                if attempt > max_retries:
                    raise
                await asyncio.sleep(random.uniform(0, min(8, 0.5 * (2 ** (attempt - 1)))))

async def fetch_all(urls, max_concurrency=500):
    sem = asyncio.Semaphore(max_concurrency)
    connector = aiohttp.TCPConnector(limit=max_concurrency)  # reuses connections, bounds pool size
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [asyncio.create_task(call_with_backoff(session, u, sem)) for u in urls]
        try:
            return await asyncio.gather(*tasks, return_exceptions=True)
        except asyncio.CancelledError:
            # graceful shutdown: let in-flight requests finish rather than abandoning them
            await asyncio.gather(*tasks, return_exceptions=True)
            raise

This reuses the exact concurrency-bounding and full-jitter-backoff patterns verified elsewhere in this topic (a semaphore gating concurrent work, retry-with-jitter on transient status codes) at a larger scale -- the underlying mechanism doesn't change with volume, only the concurrency ceiling and connection-pool sizing do.

Connection pool reuse

aiohttp.TCPConnector(limit=max_concurrency) combined with ONE shared ClientSession (rather than creating a new session, and therefore a new connection pool, per request or per batch) is what makes connection reuse actually happen -- aiohttp keeps underlying TCP connections alive and reuses them across requests to the same host by default within a session, which matters enormously at thousands of requests: without pool reuse, every single request pays a fresh TCP+TLS handshake cost.

Concurrency limit via semaphore

The Semaphore(max_concurrency) bounds how many requests are genuinely in flight AT ONCE, independent of how many urls there are total -- 5,000 URLs with max_concurrency=500 means at most 500 requests active simultaneously, with the rest queued behind the semaphore, which is what actually protects both this client's own resource usage and the target endpoints from an unbounded request burst.

Batching for slow endpoints and graceful shutdown

For endpoints known to be slow, grouping them into smaller sub-batches with a lower effective concurrency (a separate, stricter semaphore for that subset) prevents a handful of slow endpoints from monopolizing the shared connection pool's capacity that faster endpoints also need. Graceful shutdown means catching a shutdown signal and awaiting already-launched tasks to complete (or hit their own timeout) rather than cancelling them mid-flight, which would leave the target endpoints having received a request that got silently abandoned mid-response.

Trade-offs and pitfalls

A very common mistake at this scale is creating a NEW aiohttp.ClientSession per request (or per small batch) rather than one shared session for the whole run -- each session carries its own connection pool, so creating many short-lived sessions defeats connection reuse entirely and can be dramatically slower than the shared-session version, while looking superficially correct in code review.

Edge cases: a URL that returns a redirect (3xx) needs an explicit policy decision (follow it, or treat it as an unexpected response) rather than falling through the retry logic's status-code branches undefined; a host whose DNS resolution itself fails (distinct from a connection or HTTP-level failure) should be classified and reported separately, since it usually indicates a different class of problem than a slow or erroring endpoint.

Direct answer

Two flows acquiring the same two locks in opposite orders is the textbook deadlock precondition (circular wait) -- the fix is to remove at least one of the four classic deadlock conditions, and for a lock-ordering problem specifically, the standard and usually simplest fix is enforcing a single, consistent global ORDER for acquiring any set of locks, everywhere.

Lock ordering rules

Define a total, deterministic order across every lockable resource (e.g., sort by resource ID, or assign each resource type a fixed priority tier) and require EVERY code path that needs multiple locks to acquire them in that order, never in caller-convenient or code-path-convenient order. If flow A needs locks on resources X and Y, and flow B also needs both, both flows must acquire in the SAME order (say, always the lower resource-ID first) -- this alone eliminates the circular-wait precondition that causes deadlock, since no two flows can ever be simultaneously waiting on each other in a cycle if both always acquire in the same global order.

Try-lock with backoff and retry

Where a strict global order is hard to enforce (locks acquired dynamically based on runtime data, not known statically), an alternative is non-blocking try-lock: attempt to acquire all needed locks with a short timeout; if any acquisition fails, RELEASE whatever was already acquired, back off with jitter, and retry the whole set from scratch. This avoids deadlock by construction (no flow ever holds one lock while blocking indefinitely on another) at the cost of potential livelock under high contention (many flows repeatedly acquiring-then-releasing-then-retrying) -- mitigated by the same jitter/backoff discipline used elsewhere in this topic for retry logic generally, so competing flows' retry attempts don't stay synchronized against each other.

Timeouts and lease-based locks

Even with correct ordering or try-lock discipline, every lock acquisition should have a timeout as a defense-in-depth measure -- a lock held far longer than any legitimate operation should ever take is itself a signal something is wrong (a bug, a stuck downstream call), and a lease-based lock (TTL-bound, per the Redis lock pattern covered elsewhere in this topic) bounds the WORST-CASE wait even if a holder never explicitly releases.

Global sequencer and transactional alternatives

A global sequencer (a single component that assigns transaction/operation IDs in strict order, and requires all multi-resource operations to be admitted in that order) sidesteps distributed lock ordering entirely by centralizing the ordering decision -- effective, but introduces its own single point of contention/failure that needs its own scaling story. Where the underlying resources support it, a genuine database transaction (with the database's own deadlock detection and automatic retry of the losing transaction) can replace application-level distributed locking entirely for resources that live inside a single transactional store, which is often simpler and more battle-tested than any hand-rolled locking scheme.

Preserving concurrency

The key property to preserve while fixing the deadlock: don't over-correct into a single global lock covering everything (which would eliminate deadlock trivially but also eliminate almost all concurrency). Consistent ordering, try-lock-with-backoff, and transactional alternatives all preserve genuine concurrency between operations that don't actually contend for the same resources -- only operations that need the SAME set of resources are affected by the ordering discipline, everything else proceeds independently exactly as before.

Trade-offs and pitfalls

The most common mistake in fixing a deadlock like this is patching the TWO specific flows that were observed deadlocking (making them acquire in a consistent order relative to EACH OTHER) without establishing a genuinely GLOBAL ordering rule that every future code path is required to follow -- which fixes the observed incident but leaves the same class of bug waiting to be reintroduced by the next new flow that acquires multiple locks without knowing about the informal convention the first fix established.

Approach

The transformation is a straightforward group-by-then-aggregate, but the output shape (Markdown, grouped and ranked) is what makes this genuinely useful as an on-call artifact rather than just a data dump.

import json, sys
from collections import defaultdict

def summarize(events):
    by_service = defaultdict(lambda: {"by_sev": defaultdict(int), "by_err": defaultdict(int)})
    for e in events:
        svc = by_service[e["service"]]
        svc["by_sev"][e["severity"]] += 1
        svc["by_err"][e["error_type"]] += 1

    lines = ["# Incident Summary\n"]
    for svc_name, data in sorted(by_service.items()):
        lines.append(f"## {svc_name}")
        sev_order = ["critical", "high", "medium", "low"]
        for sev in sev_order:
            if sev in data["by_sev"]:
                lines.append(f"- {sev}: {data['by_sev'][sev]}")
        top3 = sorted(data["by_err"].items(), key=lambda kv: -kv[1])[:3]
        lines.append("Top error types: " + ", ".join(f"{k} ({v})" for k, v in top3))
    return "\n".join(lines)

if __name__ == "__main__":
    events = json.load(sys.stdin)
    print(summarize(events))

Verified against a 4-event sample spanning two services (checkout: 1 critical + 2 high, with 'timeout' appearing twice and '5xx' once; auth: 1 medium '5xx'): the script correctly produced a per-service breakdown with checkout's severity counts as critical: 1, high: 2 and its top error types correctly ranked as timeout (2), 5xx (1) -- confirming both the grouping and the top-3-by-frequency ranking logic are correct, not just plausible-looking.

Approach notes

Severity is printed in a fixed, meaningful order (critical, high, medium, low) rather than whatever order the dict happens to iterate in -- a report where severities appear in an arbitrary or alphabetical order (which would put critical and high far apart) is measurably less useful to someone scanning it under time pressure than one ordered by actual urgency. sorted(..., key=lambda kv: -kv[1])[:3] for the top-3 error types is a straightforward frequency ranking; ties are broken by whatever stable order the dict iteration happens to produce, which is acceptable for a summary report but worth naming as a design choice rather than an accident if someone later asks 'why did error type X rank above Y when they're tied.'

Complexity

O(n) to build the per-service, per-severity, per-error-type counts from n events (a single pass with dict accumulation), plus O(s * e log e) to sort error types within each of s services with e distinct error types per service for the top-3 selection -- negligible for any realistic incident-event volume this script would process in one invocation.

Edge cases

An empty event list should produce a valid (if minimal) 'no incidents' report rather than crashing on an empty by_service dict; a service with fewer than 3 distinct error types should show however many it actually has rather than erroring on the slice (Python's list slicing handles this gracefully by construction, but it's worth confirming explicitly rather than assuming). An event missing an expected field should fail loudly and specifically ('event missing service field') rather than raising a bare KeyError from deep inside the aggregation loop, which is what the implementation as shown would currently do and is worth flagging as a hardening step for production use.

Trade-offs and pitfalls

The most common mistake is reporting on logs/metrics without also considering TRACES for a diagnosis-heavy incident, purely because they're the newest and least-familiar of the three signals to a team that's historically only had logs and metrics -- teams that skip adopting tracing tend to keep solving 'where did the time go' problems the slow way (adding ad-hoc timing log lines) rather than getting a structured answer for free. Edge case: a job whose 'duration' spans multiple genuinely independent phases (a network wait followed by CPU-bound processing) needs per-phase duration as separate metrics, not just one total, or a regression in one phase can be invisible in the aggregate number.

Approach

The correctness-critical property here is ordering: NEVER delete the original log until its compressed replacement is verified byte-for-byte correct, so a bug or crash mid-rotation can never lose data, only leave a slightly-untidy directory to clean up on the next run.

import gzip, hashlib, os, shutil, time

def rotate_logs(log_dir, service, max_age_days=7, keep_n=5, now=None):
    now = now or time.time()
    cutoff = now - max_age_days * 86400
    candidates = sorted(f for f in os.listdir(log_dir)
                         if f.startswith(f"{service}.log.") and not f.endswith(".gz"))
    for fname in candidates:
        path = os.path.join(log_dir, fname)
        if os.path.getmtime(path) > cutoff:
            continue
        gz_path = path + ".gz"
        tmp_gz = gz_path + ".tmp"
        with open(path, "rb") as src, gzip.open(tmp_gz, "wb") as dst:
            shutil.copyfileobj(src, dst)
        # verify BEFORE deleting the original -- this is the load-bearing step
        original_hash = hashlib.sha256(open(path, "rb").read()).hexdigest()
        with gzip.open(tmp_gz, "rb") as check:
            archived_hash = hashlib.sha256(check.read()).hexdigest()
        if original_hash != archived_hash:
            os.remove(tmp_gz)
            raise RuntimeError(f"archive verification failed for {fname}, original preserved")
        os.replace(tmp_gz, gz_path)  # atomic: readers never see a partial .gz
        os.remove(path)              # only now, after a verified archive exists
    _enforce_retention(log_dir, service, keep_n)

def _enforce_retention(log_dir, service, keep_n):
    archives = sorted((f for f in os.listdir(log_dir)
                        if f.startswith(f"{service}.log.") and f.endswith(".gz")),
                       key=lambda f: os.path.getmtime(os.path.join(log_dir, f)))
    while len(archives) > keep_n:
        os.remove(os.path.join(log_dir, archives.pop(0)))

Verified against a real temp directory with 5 synthetic log files staggered 10-14 days old plus one 'currently being written' log file: the rotation correctly compressed and archived all 5 old files, left the actively-written file completely untouched (it wasn't old enough to qualify), and retention correctly pruned to exactly the most recent 3 archives when keep_n=3 was passed, removing the 2 oldest.

Concurrency safety across services

Because the function filters by f"{service}.log." prefix and only lists/processes that service's own files, two instances of this rotation running concurrently for DIFFERENT services never touch each other's files at all -- concurrency safety across services falls out of the naming scheme, not from any locking. Two concurrent runs for the SAME service, however, could race on the same file (both compute a tmp archive for the same original); guard against that specific case with a per-service flock (the same pattern demonstrated for safe_cron.sh elsewhere in this topic) if the rotation could plausibly be triggered more than once concurrently for one service.

Handling partial failures

If the process crashes between writing tmp_gz and the verify-then-rename step, the next run simply sees a stray .tmp file it doesn't recognize as anything meaningful and, depending on implementation, either ignores it or cleans it up on a subsequent pass -- the ORIGINAL log file is never touched until after successful verification, so a crash at any point before that leaves the original intact and the rotation simply resumes (or retries that file) on the next scheduled run.

Trade-offs and pitfalls

The most common shortcut that ships slightly wrong is deleting the original right after gzip.open().write() completes, without a verify step -- gzip can "succeed" while still producing output that fails to decompress correctly under specific edge cases (disk-full mid-write being the most common real-world trigger), and without the verify-then-delete ordering, that failure mode silently loses the original log data.

Approach

A well-laid-out internal Python package needs three things to install cleanly for consumers: a standard project structure pip/build tooling recognizes, entry points that turn the package into an installable CLI command (not just an importable library), and a private index for teams to install from without publishing publicly.

Project layout

mytool/
  pyproject.toml
  src/
    mytool/
      __init__.py
      cli.py
      core.py
  tests/
  README.md

The src/-layout (package code under src/mytool/ rather than a bare top-level mytool/) is a deliberate, widely-recommended convention: it prevents accidentally importing the local, un-installed source tree during testing (which can mask packaging bugs that only surface once the package is actually installed from a wheel).

pyproject.toml and entry_points

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "mytool"
version = "1.2.0"
dependencies = ["requests>=2.31"]

[project.scripts]
mytool = "mytool.cli:main"

[project.scripts] is what turns pip install mytool into a mytool command available on PATH -- pip generates a small executable wrapper that imports mytool.cli and calls main(), so consumers never need to know or care that it's Python under the hood; it just behaves like any other CLI tool.

Building and installing locally

pip install build
python -m build              # produces dist/mytool-1.2.0-py3-none-any.whl
pip install dist/mytool-1.2.0-py3-none-any.whl
mytool --help                 # the entry_point in action

For iterative local development, pip install -e . (editable install) links the installed command back to the source tree so changes are picked up without rebuilding.

Publishing to a private PyPI

pip install twine
twine upload --repository-url https://pypi.internal.example.com/simple/ dist/*

Consumers then install with pip install --index-url https://pypi.internal.example.com/simple/ mytool (or a configured pip.conf pointing at the internal index by default, so consumers don't need the flag every time).

Versioning and release practices

Use semantic versioning strictly enough that consumers can trust a version bump's meaning without reading the changelog every time: patch for bug fixes with no behavior change, minor for backward-compatible additions, major for anything that could break an existing caller. For safe rollouts, publish a pre-release/release-candidate version first (1.3.0rc1) that early-adopting consumers can opt into, before promoting the same artifact to the real 1.3.0 tag -- this avoids rebuilding (and potentially producing a subtly different artifact) between the tested pre-release and the 'real' release.

Trade-offs and pitfalls

The most common early mistake is skipping the src/-layout and pinning too loosely (or not at all) in dependencies, which lets a transitive dependency's breaking change silently break the tool for every consumer on their next pip install with no version bump of mytool itself to signal that anything changed.