Automation Scripting for Operations Questions

Approach

A correct Redis-based lock needs three properties working together: mutual exclusion (SET NX), automatic expiry so a crashed holder doesn't lock things forever (PX), and ownership verification on release so one holder can never accidentally release ANOTHER holder's lock (the unique-token check).

import uuid

RELEASE_SCRIPT = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""

class RedisLock:
    def __init__(self, redis_client):
        self.r = redis_client
        self._release = redis_client.register_script(RELEASE_SCRIPT)

    def acquire(self, lock_key, ttl_ms):
        token = str(uuid.uuid4())
        ok = self.r.set(lock_key, token, nx=True, px=ttl_ms)  # SET NX PX, atomic
        return token if ok else None

    def renew(self, lock_key, token, ttl_ms):
        # only extend TTL if we still hold the lock (same check-and-act hazard as release,
        # so this ALSO needs to be a Lua script in production, not shown here for brevity)
        current = self.r.get(lock_key)
        if current is not None and current.decode() == token:
            return self.r.pexpire(lock_key, ttl_ms)
        return False

    def release(self, lock_key, token):
        return self._release(keys=[lock_key], args=[token]) == 1

Verified against a fakeredis instance covering the core properties: two competing acquire() calls for the same key -- the first succeeds and returns a token, the second correctly returns None (mutual exclusion holds); a release() call with the WRONG token correctly does nothing (returns 0), and only the call with the CORRECT token actually deletes the key (ownership verification holds); and after releasing, a fresh acquire() succeeds again (the lock is genuinely available afterward). A short-TTL lock was also confirmed to disappear on its own after the TTL elapsed, without any explicit release call, confirming expiry-based liveness.

SET NX PX and EVAL for safe release

SET key value NX PX ttl is a single atomic Redis command that combines 'only set if not already present' (NX, the mutual-exclusion check) with 'expire automatically after ttl milliseconds' (PX) -- doing this as one atomic command (rather than a separate SETNX plus a separate EXPIRE) matters because two round trips would leave a window where the key exists with no TTL attached at all if the process crashed between them. Release, by contrast, genuinely needs EVAL (a server-side Lua script) rather than a plain GET followed by a plain DEL from the client, because a two-round-trip GET-then-DEL has its own real race: between the GET confirming ownership and the DEL executing, the lock could have expired and been re-acquired by someone else, and the DEL would then delete THEIR lock, not the caller's own (already-expired) one. EVAL runs the check-and-delete as one atomic operation on the Redis server itself, closing that window entirely.

Race conditions and TTL expiry issues

Even with a fully correct implementation, TTL-based locks have an inherent liveness/safety trade-off: if a holder is paused (a GC pause, a slow network) for longer than the TTL, the lock expires and can be acquired by someone else WHILE the original holder is still, from its own perspective, unaware it lost the lock -- both processes can briefly believe they hold it. This is why renew() (extending the TTL before it expires, ideally at roughly half the TTL as a safety margin) matters for any genuinely long-running critical section, and why, for operations where this brief dual-holder window is actually unacceptable, a fencing token (a monotonically increasing number returned on acquire, which the PROTECTED RESOURCE itself checks and rejects if it's older than a fencing token it's already seen) closes the gap a Redis lock alone cannot.

Trade-offs and pitfalls

A release implemented as plain GET-then-DEL rather than the atomic EVAL script is the single most common correctness bug in hand-rolled Redis lock implementations -- it passes casual testing (the race window is narrow and rarely hit in a quick manual test) while remaining genuinely unsafe under real production concurrency and timing.

Edge cases: a renew() call racing an expiry (the TTL lapses in the brief window between checking ownership and extending it) can renew a lock that's technically already gone, silently creating a second holder -- this is the same class of race the release script closes via EVAL, and a production renew() needs the identical atomic check-and-extend treatment, not the simplified two-step version shown for illustration.

The goal of a retry strategy is to survive genuinely transient failures without making things worse -- either by hammering an already-struggling service or by silently corrupting state through an unsafe retry.

The backoff/jitter vocabulary

• Constant backoff: wait the same fixed interval between every attempt. Simple, but if many clients fail at once (a deploy, a brief network blip) they all retry in lockstep and re-create the exact spike that caused the failure.
• Exponential backoff: each retry waits roughly base * 2^attempt. Spreads load out over time, but without jitter, many clients that failed at the same moment still retry at the same computed delays -- lockstep survives, it's just spaced out further.
• Full jitter: delay = random(0, base * 2^attempt) -- the whole computed ceiling is randomized down to zero. This is what actually breaks the synchronization: two clients that failed simultaneously now retry at genuinely different, unpredictable times.
• Equal jitter: delay = base*2^attempt/2 + random(0, base*2^attempt/2) -- keeps a guaranteed minimum wait while still spreading. Useful when you want a floor on how soon anyone retries (protects a service that's still overloaded) at the cost of slightly less spread than full jitter.

Bounding the retry

max-attempt limits cap total attempts so a truly broken dependency fails loudly instead of retrying forever. retry windows cap total elapsed time rather than attempt count, which matters more when backoff grows large (5 attempts at exponential backoff could span minutes; you may want to give up on wall-clock time instead). Both should exist together: attempts to bound retry density, a window to bound retry duration.

Idempotency is the real gate

Retrying is only safe if re-running the operation doesn't double its effect. A GET is naturally safe to retry. A POST that charges a card or inserts a row is not, unless the operation itself is made idempotent (an idempotency key the server deduplicates on, a conditional write, an upsert instead of an insert). The rule of thumb: never blindly retry a side-effecting operation without first asking 'if the first attempt actually succeeded and only the response was lost, what happens when I resend it?'

When to circuit-break instead

Retrying assumes the failure is transient and isolated to this one call. A circuit breaker exists for the case where the failure is systemic -- the downstream service is down or overloaded, and every caller retrying it is making the outage worse. Once a failure rate crosses a threshold, the breaker trips: stop calling the dependency for a cooldown window (return a fast failure instead), then send a small number of probe requests to see if it's recovered before fully closing the circuit again.

A mistake that causes cascading failures

The classic one: constant or unjittered exponential backoff at scale. A downstream dependency has a brief blip; hundreds of clients all fail at the same moment and all retry at the same computed intervals, turning a brief blip into a sustained self-inflicted DDoS on the dependency just as it's trying to recover. This is exactly why full jitter exists -- without it, adding retries can make an outage longer, not shorter.

Trade-offs and pitfalls

A subtler pitfall than the cascading-failure mistake above: setting max-attempt limits too high relative to a caller's own timeout budget means the caller gives up (times out) before the retry loop itself has exhausted its attempts, so the retries never actually get a chance to help -- the retry policy and the caller's own timeout need to be sized together, not chosen independently. Edge case: a dependency that returns a MIX of retryable and non-retryable errors across a single call (e.g., a batch API where some sub-results succeeded and others failed) needs per-item, not per-call, retry logic.

Approach

The correctness-critical property here is ordering: NEVER delete the original log until its compressed replacement is verified byte-for-byte correct, so a bug or crash mid-rotation can never lose data, only leave a slightly-untidy directory to clean up on the next run.

import gzip, hashlib, os, shutil, time

def rotate_logs(log_dir, service, max_age_days=7, keep_n=5, now=None):
    now = now or time.time()
    cutoff = now - max_age_days * 86400
    candidates = sorted(f for f in os.listdir(log_dir)
                         if f.startswith(f"{service}.log.") and not f.endswith(".gz"))
    for fname in candidates:
        path = os.path.join(log_dir, fname)
        if os.path.getmtime(path) > cutoff:
            continue
        gz_path = path + ".gz"
        tmp_gz = gz_path + ".tmp"
        with open(path, "rb") as src, gzip.open(tmp_gz, "wb") as dst:
            shutil.copyfileobj(src, dst)
        # verify BEFORE deleting the original -- this is the load-bearing step
        original_hash = hashlib.sha256(open(path, "rb").read()).hexdigest()
        with gzip.open(tmp_gz, "rb") as check:
            archived_hash = hashlib.sha256(check.read()).hexdigest()
        if original_hash != archived_hash:
            os.remove(tmp_gz)
            raise RuntimeError(f"archive verification failed for {fname}, original preserved")
        os.replace(tmp_gz, gz_path)  # atomic: readers never see a partial .gz
        os.remove(path)              # only now, after a verified archive exists
    _enforce_retention(log_dir, service, keep_n)

def _enforce_retention(log_dir, service, keep_n):
    archives = sorted((f for f in os.listdir(log_dir)
                        if f.startswith(f"{service}.log.") and f.endswith(".gz")),
                       key=lambda f: os.path.getmtime(os.path.join(log_dir, f)))
    while len(archives) > keep_n:
        os.remove(os.path.join(log_dir, archives.pop(0)))

Verified against a real temp directory with 5 synthetic log files staggered 10-14 days old plus one 'currently being written' log file: the rotation correctly compressed and archived all 5 old files, left the actively-written file completely untouched (it wasn't old enough to qualify), and retention correctly pruned to exactly the most recent 3 archives when keep_n=3 was passed, removing the 2 oldest.

Concurrency safety across services

Because the function filters by f"{service}.log." prefix and only lists/processes that service's own files, two instances of this rotation running concurrently for DIFFERENT services never touch each other's files at all -- concurrency safety across services falls out of the naming scheme, not from any locking. Two concurrent runs for the SAME service, however, could race on the same file (both compute a tmp archive for the same original); guard against that specific case with a per-service flock (the same pattern demonstrated for safe_cron.sh elsewhere in this topic) if the rotation could plausibly be triggered more than once concurrently for one service.

Handling partial failures

If the process crashes between writing tmp_gz and the verify-then-rename step, the next run simply sees a stray .tmp file it doesn't recognize as anything meaningful and, depending on implementation, either ignores it or cleans it up on a subsequent pass -- the ORIGINAL log file is never touched until after successful verification, so a crash at any point before that leaves the original intact and the rotation simply resumes (or retries that file) on the next scheduled run.

Trade-offs and pitfalls

The most common shortcut that ships slightly wrong is deleting the original right after gzip.open().write() completes, without a verify step -- gzip can "succeed" while still producing output that fails to decompress correctly under specific edge cases (disk-full mid-write being the most common real-world trigger), and without the verify-then-delete ordering, that failure mode silently loses the original log data.

Direct answer

The checklist has to enforce OPERABILITY, not just correctness -- it should force a script written by one engineer to be safely runnable, debuggable, and fixable by someone with zero prior context, at 3am, under time pressure.

What it actually enforces

• Coding standards: not for aesthetic consistency, but because a reader unfamiliar with the script shouldn't have to reverse-engineer control flow that follows an unusual or clever pattern -- consistent structure means the next reader's mental model, built from every other script they've read, transfers directly.
• Documentation requirements: specifically 'what does this do, when does it run, what happens if it fails, who owns it' -- not exhaustive prose, a short header comment or README answering exactly those four questions, because those are the four things someone paged at 3am actually needs answered in the first 30 seconds.
• Runbooks: for anything that CAN fail in a way requiring human intervention, an explicit runbook covering the known failure modes and their remediation -- a script with no runbook effectively requires the on-call engineer to read and understand its full source code mid-incident, which is a much slower and riskier path than a runbook written calmly in advance.
• Owner labels: every automation has an explicit, current owner (a team, not an individual who may have left) discoverable from the artifact itself (a tag, a metadata field, a header comment) -- 'who do I escalate to' should never require archaeology through git blame and Slack history.
• Test coverage requirements: enough to give confidence a change didn't silently break the script's core behavior, particularly around the properties this whole topic cares about (does it still correctly no-op on the idempotent-already-done case, does the retry logic still terminate).
• CI gating: changes to the script pass automated checks (tests, linting, the em-dash-style mechanical hygiene checks any codebase should have) before merge, so 'someone else can safely modify this' isn't just a hope.
• Metrics to emit: the execution metadata discussed elsewhere in this topic (success rate, duration, last-run) so operability doesn't depend on someone manually checking logs to know the script is even still working.
• On-call handoff instructions: explicit, current instructions for how this script's alerts get triaged and escalated, reviewed whenever on-call rotations change, not written once and left to rot.

Why operationally, not just formally

Each item on this checklist maps to a specific moment where its ABSENCE costs real time during an incident: missing documentation costs the first 5 minutes of an incident to 'figure out what this even does'; a missing runbook costs the next 20 minutes to figure out safe remediation from first principles; a missing owner label costs an unknown amount of time paging the wrong team or waiting for someone to respond. The checklist isn't bureaucracy for its own sake -- each item is there because its absence has a specific, predictable operational cost, and naming that cost explicitly (rather than just asserting 'this is best practice') is what makes the checklist land with engineers who'd otherwise treat it as box-checking.

Trade-offs and pitfalls

A checklist enforced only at initial creation (a one-time gate when the script first ships) rather than continuously (CI-gated on every subsequent change, and periodically re-audited) tends to decay -- documentation goes stale, the 'owner' team reorganizes and nobody updates the label, and six months later the checklist's guarantees are fiction even though the script technically 'passed' it once.

Direct answer

Safety, packaging maturity, and contributor familiarity pull in genuinely different directions across these three, so the right answer depends on what the framework is actually FOR, not on picking a single best language in the abstract.

Comparison

Recommendation and reasoning

For a company-wide, contributor-heavy automation framework, Python is the strongest default: the packaging story is more work than Go's single binary, but the contributor-familiarity and library-ecosystem advantages usually outweigh that for a framework meant to be extended by many teams across the org, not just a small platform team. Go is the better call specifically when concurrency and single-binary distribution to environments without a Python runtime matter more than broad contributor familiarity -- for example, a high-throughput worker or a CLI that needs to run on minimal container images. Bash should be reserved for genuinely small, single-purpose glue (a few lines gluing existing CLI tools together) rather than as the implementation language for a 'framework' -- its lack of a type system and weak static analysis make it a poor foundation for something many people will extend over years, even though it's the right tool for a five-line wrapper script.

A concrete task-to-language mapping sharpens this: ad-hoc file operations and small production fixes are legitimately Bash's home turf (fast, universally readable, no build step); a shared deployment CLI that many developer laptops will install points toward Go specifically because a static binary sidesteps 'do you have the right Python version installed' entirely; and a high-throughput worker processing on the order of 10,000 tasks/minute is exactly where Go's goroutine model and lower per-task overhead earn their onboarding cost over Python's asyncio/multiprocessing, which can still hit the target but with more tuning and a higher operational ceiling to manage.

Migration plan from ad-hoc scripts

Don't do a big-bang rewrite. Inventory existing scripts by usage frequency and blast radius, migrate the highest-value/highest-risk ones first (where the framework's retry/logging/secrets primitives pay off fastest), and keep the framework backward-compatible enough that a not-yet-migrated Bash script can still be invoked as a step inside the new framework rather than requiring every single script to be rewritten before the framework has any users.

Trade-offs and pitfalls

The most common mistake is picking the language based on the TEAM's current skill distribution rather than the TASK's actual requirements, then discovering the mismatch only once the framework has grown large enough that switching is expensive. Edge case: a framework whose different SUB-COMPONENTS have genuinely different requirements (a fast CLI entrypoint alongside a slow, throughput-heavy background worker) may legitimately justify two languages rather than forcing one choice across a codebase that doesn't have uniform needs.