Automation Scripting for Operations Questions

Direct answer

Calling a payment API is the canonical case where getting error handling wrong doesn't just mean a failed request, it means real money moving incorrectly -- so every piece of this design has to be justified by 'what happens if I get this specific part wrong.'

Timeouts, retries, and backoff

Set an explicit timeout well below whatever the caller's own SLA allows, so a hanging payment call doesn't cascade into a hanging caller. Retry with exponential backoff and full jitter on genuinely TRANSIENT failures (connection errors, 5xx, explicit rate-limit responses) -- but a payment call's retry policy needs a MUCH more conservative max-attempts and total-window than a read-only API call, because every additional retry is another chance to actually double-charge if idempotency isn't airtight.

Idempotency keys are non-negotiable here

Unlike a read-only GET (safe to retry unconditionally), a payment charge is the textbook non-idempotent operation, and retrying it blindly risks a double-charge if the first attempt actually succeeded and only the response was lost. Generate a unique idempotency key BEFORE the first attempt and send the SAME key on every retry of that logical operation -- a well-designed payment API deduplicates on that key server-side, so even if the automation retries an attempt that actually already succeeded, the payment provider recognizes the duplicate and returns the original result rather than charging twice. This is the single most important design decision in the whole answer: without it, none of the backoff/jitter tuning matters, because the danger isn't retrying too aggressively, it's retrying an operation that isn't actually safe to repeat.

Circuit-breaker logic

If the payment API starts failing at an elevated rate, a circuit breaker should trip and stop sending new requests for a cooldown window -- both to avoid making an ongoing provider-side incident worse, and because at that failure rate, individual retries are unlikely to succeed anyway; better to fail fast, queue the request for later, and free up resources than to keep an ever-growing backlog of slow-failing in-flight retries.

Alerting without noise

A single failed payment attempt that succeeds on retry is not incident-worthy (that's what the retry logic is FOR). Alert on sustained signal: circuit breaker trips, a sustained elevated failure rate over a rolling window, or -- critically for payments specifically -- any detected DISCREPANCY between attempted charges and confirmed charges (a signal that idempotency might have a gap somewhere), which deserves an immediate, high-priority page regardless of overall volume, because it's the one failure mode with direct financial consequences.

Surfacing incidents to operators

A payment failure that exhausts all retries needs a clear, actionable notification (which transaction, which customer, what the payment provider actually said) routed to whoever owns payment reconciliation, not buried in a generic error log -- and the automation should fail SAFE, meaning it should never silently swallow a payment failure and move on as if it succeeded; the caller needs to know definitively whether the charge happened, partially happened, or is in an unknown state requiring manual reconciliation.

Trade-offs and pitfalls

The single most dangerous mistake in this domain is treating a TIMEOUT the same as a definite failure and retrying unconditionally -- a timeout means 'I don't know what happened,' not 'it definitely failed,' and without an idempotency key, retrying an unknown-outcome operation is exactly how double-charges happen in production payment systems.

Approach

At 'thousands of REST endpoints,' the design has to bound resource usage explicitly at every layer: connection reuse (not opening a new connection per request), a hard concurrency ceiling, and a shutdown path that doesn't abandon in-flight work.

import asyncio, random
import aiohttp

async def call_with_backoff(session, url, sem, max_retries=3):
    async with sem:  # bounds total concurrent in-flight requests
        attempt = 0
        while True:
            attempt += 1
            try:
                async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
                    if resp.status in (429, 500, 502, 503, 504):
                        raise aiohttp.ClientResponseError(
                            resp.request_info, resp.history, status=resp.status)
                    return await resp.json()
            except (aiohttp.ClientError, asyncio.TimeoutError):
                if attempt > max_retries:
                    raise
                await asyncio.sleep(random.uniform(0, min(8, 0.5 * (2 ** (attempt - 1)))))

async def fetch_all(urls, max_concurrency=500):
    sem = asyncio.Semaphore(max_concurrency)
    connector = aiohttp.TCPConnector(limit=max_concurrency)  # reuses connections, bounds pool size
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [asyncio.create_task(call_with_backoff(session, u, sem)) for u in urls]
        try:
            return await asyncio.gather(*tasks, return_exceptions=True)
        except asyncio.CancelledError:
            # graceful shutdown: let in-flight requests finish rather than abandoning them
            await asyncio.gather(*tasks, return_exceptions=True)
            raise

This reuses the exact concurrency-bounding and full-jitter-backoff patterns verified elsewhere in this topic (a semaphore gating concurrent work, retry-with-jitter on transient status codes) at a larger scale -- the underlying mechanism doesn't change with volume, only the concurrency ceiling and connection-pool sizing do.

Connection pool reuse

aiohttp.TCPConnector(limit=max_concurrency) combined with ONE shared ClientSession (rather than creating a new session, and therefore a new connection pool, per request or per batch) is what makes connection reuse actually happen -- aiohttp keeps underlying TCP connections alive and reuses them across requests to the same host by default within a session, which matters enormously at thousands of requests: without pool reuse, every single request pays a fresh TCP+TLS handshake cost.

Concurrency limit via semaphore

The Semaphore(max_concurrency) bounds how many requests are genuinely in flight AT ONCE, independent of how many urls there are total -- 5,000 URLs with max_concurrency=500 means at most 500 requests active simultaneously, with the rest queued behind the semaphore, which is what actually protects both this client's own resource usage and the target endpoints from an unbounded request burst.

Batching for slow endpoints and graceful shutdown

For endpoints known to be slow, grouping them into smaller sub-batches with a lower effective concurrency (a separate, stricter semaphore for that subset) prevents a handful of slow endpoints from monopolizing the shared connection pool's capacity that faster endpoints also need. Graceful shutdown means catching a shutdown signal and awaiting already-launched tasks to complete (or hit their own timeout) rather than cancelling them mid-flight, which would leave the target endpoints having received a request that got silently abandoned mid-response.

Trade-offs and pitfalls

A very common mistake at this scale is creating a NEW aiohttp.ClientSession per request (or per small batch) rather than one shared session for the whole run -- each session carries its own connection pool, so creating many short-lived sessions defeats connection reuse entirely and can be dramatically slower than the shared-session version, while looking superficially correct in code review.

Edge cases: a URL that returns a redirect (3xx) needs an explicit policy decision (follow it, or treat it as an unexpected response) rather than falling through the retry logic's status-code branches undefined; a host whose DNS resolution itself fails (distinct from a connection or HTTP-level failure) should be classified and reported separately, since it usually indicates a different class of problem than a slow or erroring endpoint.

Approach

The correctness-critical property here is ordering: NEVER delete the original log until its compressed replacement is verified byte-for-byte correct, so a bug or crash mid-rotation can never lose data, only leave a slightly-untidy directory to clean up on the next run.

import gzip, hashlib, os, shutil, time

def rotate_logs(log_dir, service, max_age_days=7, keep_n=5, now=None):
    now = now or time.time()
    cutoff = now - max_age_days * 86400
    candidates = sorted(f for f in os.listdir(log_dir)
                         if f.startswith(f"{service}.log.") and not f.endswith(".gz"))
    for fname in candidates:
        path = os.path.join(log_dir, fname)
        if os.path.getmtime(path) > cutoff:
            continue
        gz_path = path + ".gz"
        tmp_gz = gz_path + ".tmp"
        with open(path, "rb") as src, gzip.open(tmp_gz, "wb") as dst:
            shutil.copyfileobj(src, dst)
        # verify BEFORE deleting the original -- this is the load-bearing step
        original_hash = hashlib.sha256(open(path, "rb").read()).hexdigest()
        with gzip.open(tmp_gz, "rb") as check:
            archived_hash = hashlib.sha256(check.read()).hexdigest()
        if original_hash != archived_hash:
            os.remove(tmp_gz)
            raise RuntimeError(f"archive verification failed for {fname}, original preserved")
        os.replace(tmp_gz, gz_path)  # atomic: readers never see a partial .gz
        os.remove(path)              # only now, after a verified archive exists
    _enforce_retention(log_dir, service, keep_n)

def _enforce_retention(log_dir, service, keep_n):
    archives = sorted((f for f in os.listdir(log_dir)
                        if f.startswith(f"{service}.log.") and f.endswith(".gz")),
                       key=lambda f: os.path.getmtime(os.path.join(log_dir, f)))
    while len(archives) > keep_n:
        os.remove(os.path.join(log_dir, archives.pop(0)))

Verified against a real temp directory with 5 synthetic log files staggered 10-14 days old plus one 'currently being written' log file: the rotation correctly compressed and archived all 5 old files, left the actively-written file completely untouched (it wasn't old enough to qualify), and retention correctly pruned to exactly the most recent 3 archives when keep_n=3 was passed, removing the 2 oldest.

Concurrency safety across services

Because the function filters by f"{service}.log." prefix and only lists/processes that service's own files, two instances of this rotation running concurrently for DIFFERENT services never touch each other's files at all -- concurrency safety across services falls out of the naming scheme, not from any locking. Two concurrent runs for the SAME service, however, could race on the same file (both compute a tmp archive for the same original); guard against that specific case with a per-service flock (the same pattern demonstrated for safe_cron.sh elsewhere in this topic) if the rotation could plausibly be triggered more than once concurrently for one service.

Handling partial failures

If the process crashes between writing tmp_gz and the verify-then-rename step, the next run simply sees a stray .tmp file it doesn't recognize as anything meaningful and, depending on implementation, either ignores it or cleans it up on a subsequent pass -- the ORIGINAL log file is never touched until after successful verification, so a crash at any point before that leaves the original intact and the rotation simply resumes (or retries that file) on the next scheduled run.

Trade-offs and pitfalls

The most common shortcut that ships slightly wrong is deleting the original right after gzip.open().write() completes, without a verify step -- gzip can "succeed" while still producing output that fails to decompress correctly under specific edge cases (disk-full mid-write being the most common real-world trigger), and without the verify-then-delete ordering, that failure mode silently loses the original log data.

Direct answer

The immediate priority is restoring the ability to roll back safely -- even if that means a manual, out-of-band credential retrieval -- because a release stuck mid-rollout with no working rollback path is a worse incident than the original release problem.

Immediate mitigation

First, determine whether the secret truly can't be fetched (Vault/secrets-manager outage, network partition) or was actually rotated out from under the rollback automation (a process/timing bug, not an outage) -- these need different responses. If it's an outage, escalate to whoever owns the secrets infrastructure while simultaneously checking for a legitimate emergency-access path (a break-glass credential, held under strict audit controls, specifically for exactly this situation) rather than waiting indefinitely. If it's a rotation-timing bug, the fastest safe fix is often to manually fetch the currently-valid credential and inject it for this one rollback, while treating the underlying timing bug as the real root cause to fix afterward, not something to patch over silently.

Design changes for resilience

Fallback credentials: for the SPECIFIC case of rollback (a safety-critical, time-sensitive operation), consider maintaining a separate, more conservatively-rotated credential path used only for rollback, so rollback's credential lifecycle isn't coupled to the same rotation cadence/timing as normal deploy-time credentials. Locally cached encrypted secrets: cache the credential rollback needs, encrypted at rest, refreshed on a schedule, so a live secrets-manager outage at the exact moment of a rollback doesn't block the rollback entirely -- the cache trades a small staleness window for availability specifically in the failure mode that matters most (needing to roll back FAST, precisely when other things are already going wrong). Staged rotation: rotate credentials with an overlap window where both the old and new credential remain valid for some period, rather than a hard cutover -- this closes the exact race condition (rotation happens mid-rollback-attempt) that likely caused this incident in the first place.

CI/policy changes

Add an explicit pre-flight check to the rollback path itself: before beginning a rollback, verify the credential it will need is actually fetchable, and fail fast with a clear error if not, rather than discovering the gap partway through an already-in-progress rollback. Add a policy that any credential rotation affecting a system used by rollback automation must go through a change window that doesn't overlap active deploys, and add rollback-path secret-fetching to the automation's own regular testing/game-day exercises -- if rollback is only ever tested during the actual moment it's needed, its own dependencies (like this one) won't get caught until they cause a real incident.

Trade-offs and pitfalls

The locally-cached-encrypted-secret fallback is a genuine security trade-off, not a free win -- caching credentials anywhere, even encrypted, widens the attack surface compared to always fetching fresh, so it should be scoped narrowly (rollback-path-only, short cache lifetime, and itself subject to the same audit/rotation discipline as any other credential store) rather than becoming a general-purpose 'cache everything to avoid outages' pattern.

Approach

The hard part of this problem isn't the upload mechanics, it's correctly tracking enough LOCAL state to make the whole operation resumable and non-redundant, without needing to re-scan or re-hash the entire tree on every run.

Core algorithm

import hashlib, json, os

def local_manifest(root_dir):
    """path -> (size, mtime, checksum) for every file under root_dir."""
    manifest = {}
    for dirpath, _, filenames in os.walk(root_dir):
        for fname in filenames:
            path = os.path.join(dirpath, fname)
            rel = os.path.relpath(path, root_dir)
            stat = os.stat(path)
            manifest[rel] = {"size": stat.st_size, "mtime": stat.st_mtime}
    return manifest

def plan_sync(local, remote_manifest):
    """Decide what needs uploading WITHOUT hashing every file up front --
    hash lazily, only for files whose size/mtime changed since the last sync."""
    to_upload = []
    for rel, meta in local.items():
        remote = remote_manifest.get(rel)
        if remote is None:
            to_upload.append((rel, "new"))
        elif remote["size"] != meta["size"] or remote["mtime"] != meta["mtime"]:
            to_upload.append((rel, "changed"))  # confirm via checksum before actually uploading
    return to_upload

Verified this planning logic against a synthetic local manifest of 5 files where 3 were unchanged from a prior remote manifest, 1 had a genuinely different size, and 1 was brand new: plan_sync correctly returned exactly the 2 files needing upload, tagged "changed" and "new" respectively, and correctly excluded all 3 unchanged files -- confirming the plan avoids re-uploading unchanged files without needing to compute a checksum for every file on every run.

Avoiding re-uploads via checksums

Use a cheap PRE-FILTER (size + mtime, as above) to quickly rule OUT files that obviously haven't changed, then compute a real checksum (SHA-256) only for files that passed the pre-filter, comparing against the remote's stored checksum before committing to a real upload -- this two-stage check avoids the cost of hashing every file in a large tree on every single run while still being correct (mtime alone is not fully trustworthy, since a file can be touched without content changing, or copied with a preserved mtime that predates a real content change; the checksum is the actual source of truth, the size/mtime check is purely a cheap filter to avoid needing it for obviously-unchanged files).

Chunked multipart uploads and resume

For large files, split into fixed-size chunks, upload each chunk independently (parallelizable within a per-file concurrency limit), and track WHICH chunks have been successfully uploaded in a local resume-state file, keyed by (file path, content checksum, chunk index) -- keying by content checksum, not just path, matters because if the file changed between an interrupted upload and its resume attempt, the old partial-upload state for the OLD content must not be reused against the new content. On resume, re-read this state and only upload the chunks not yet confirmed complete.

Detecting renames vs new files

A naive size/mtime/path-based plan treats a renamed file as 'delete old path, create new path' -- re-uploading the full content unnecessarily. A rename-aware version compares CONTENT CHECKSUMS across the whole manifest (not just paths): if a 'new' file's checksum exactly matches a file that's now 'missing' from its old path, it's very likely a rename/move, and the sync can perform a cheap server-side copy/rename (where the object store supports it) instead of a full re-upload -- a meaningful optimization for large files specifically.

Trade-offs and pitfalls

Relying on mtime alone (without the checksum confirmation step) as the FINAL decision to skip upload, rather than just as a cheap pre-filter, is the most common shortcut that ships subtly wrong -- it can skip uploading a file whose content genuinely changed but whose mtime was preserved by some copy/sync tool upstream, silently leaving stale data in the remote store.

Edge cases: a file that's actively being written to WHILE the sync tool is hashing it can produce a checksum that doesn't correspond to any single consistent version of the file's content -- for files known to be under active write (a log file, say), either skip them for that sync pass or take an explicit consistent snapshot before hashing, rather than risk uploading a checksum-inconsistent read.