Automation Scripting for Operations Questions

Direct answer

Two flows acquiring the same two locks in opposite orders is the textbook deadlock precondition (circular wait) -- the fix is to remove at least one of the four classic deadlock conditions, and for a lock-ordering problem specifically, the standard and usually simplest fix is enforcing a single, consistent global ORDER for acquiring any set of locks, everywhere.

Lock ordering rules

Define a total, deterministic order across every lockable resource (e.g., sort by resource ID, or assign each resource type a fixed priority tier) and require EVERY code path that needs multiple locks to acquire them in that order, never in caller-convenient or code-path-convenient order. If flow A needs locks on resources X and Y, and flow B also needs both, both flows must acquire in the SAME order (say, always the lower resource-ID first) -- this alone eliminates the circular-wait precondition that causes deadlock, since no two flows can ever be simultaneously waiting on each other in a cycle if both always acquire in the same global order.

Try-lock with backoff and retry

Where a strict global order is hard to enforce (locks acquired dynamically based on runtime data, not known statically), an alternative is non-blocking try-lock: attempt to acquire all needed locks with a short timeout; if any acquisition fails, RELEASE whatever was already acquired, back off with jitter, and retry the whole set from scratch. This avoids deadlock by construction (no flow ever holds one lock while blocking indefinitely on another) at the cost of potential livelock under high contention (many flows repeatedly acquiring-then-releasing-then-retrying) -- mitigated by the same jitter/backoff discipline used elsewhere in this topic for retry logic generally, so competing flows' retry attempts don't stay synchronized against each other.

Timeouts and lease-based locks

Even with correct ordering or try-lock discipline, every lock acquisition should have a timeout as a defense-in-depth measure -- a lock held far longer than any legitimate operation should ever take is itself a signal something is wrong (a bug, a stuck downstream call), and a lease-based lock (TTL-bound, per the Redis lock pattern covered elsewhere in this topic) bounds the WORST-CASE wait even if a holder never explicitly releases.

Global sequencer and transactional alternatives

A global sequencer (a single component that assigns transaction/operation IDs in strict order, and requires all multi-resource operations to be admitted in that order) sidesteps distributed lock ordering entirely by centralizing the ordering decision -- effective, but introduces its own single point of contention/failure that needs its own scaling story. Where the underlying resources support it, a genuine database transaction (with the database's own deadlock detection and automatic retry of the losing transaction) can replace application-level distributed locking entirely for resources that live inside a single transactional store, which is often simpler and more battle-tested than any hand-rolled locking scheme.

Preserving concurrency

The key property to preserve while fixing the deadlock: don't over-correct into a single global lock covering everything (which would eliminate deadlock trivially but also eliminate almost all concurrency). Consistent ordering, try-lock-with-backoff, and transactional alternatives all preserve genuine concurrency between operations that don't actually contend for the same resources -- only operations that need the SAME set of resources are affected by the ordering discipline, everything else proceeds independently exactly as before.

Trade-offs and pitfalls

The most common mistake in fixing a deadlock like this is patching the TWO specific flows that were observed deadlocking (making them acquire in a consistent order relative to EACH OTHER) without establishing a genuinely GLOBAL ordering rule that every future code path is required to follow -- which fixes the observed incident but leaves the same class of bug waiting to be reintroduced by the next new flow that acquires multiple locks without knowing about the informal convention the first fix established.

Approach

A retry decorator needs to separate three concerns cleanly: which exceptions are worth retrying (the predicate), how long to wait between attempts (backoff+jitter), and what to do on each attempt (logging, and eventually giving up). Keeping these as parameters rather than hardcoding them is what makes the decorator reusable across call sites with very different retry needs.

import functools
import logging
import random
import time

def retry(max_attempts=5, base_delay=0.5, max_delay=30.0,
          retryable=(Exception,), logger=None):
    """Retry a synchronous function with full-jitter exponential backoff.

    retryable: a tuple of exception types, OR a callable(exc) -> bool that
               classifies whether a given exception is worth retrying.
    """
    def is_retryable(exc):
        if callable(retryable) and not isinstance(retryable, type):
            return retryable(exc)
        return isinstance(exc, retryable)

    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            attempt = 0
            while True:
                attempt += 1
                try:
                    return fn(*args, **kwargs)
                except Exception as exc:
                    if not is_retryable(exc) or attempt >= max_attempts:
                        raise
                    ceiling = min(max_delay, base_delay * (2 ** (attempt - 1)))
                    delay = random.uniform(0, ceiling)  # full jitter
                    if logger:
                        logger.info("attempt %d/%d failed (%r), retrying in %.2fs",
                                     attempt, max_attempts, exc, delay)
                    time.sleep(delay)
        return wrapper
    return decorator

Verified in a sandbox: wrapping a function that raises ValueError twice then succeeds, @retry(max_attempts=4, retryable=(ValueError,)) returns the correct result after exactly 3 attempts; wrapping a function that always raises, it makes exactly max_attempts attempts and then re-raises the original exception rather than swallowing it.

Idempotency and the decorator

The decorator itself has no idea whether the wrapped function is safe to call twice -- that judgment has to be made by whoever applies it. Two consequences: (1) the retryable predicate should exclude exceptions that indicate the operation may have partially succeeded in an ambiguous way (a timeout on a POST is the classic ambiguous case: did the server process it and the response got lost, or did it never receive the request?), and (2) for genuinely non-idempotent external calls, the caller should generate an idempotency token before the first attempt and pass the same token on every retry, so the server-side API can deduplicate. The decorator's job is to retry; the idempotency token's job is to make retrying safe -- they're separate concerns that have to be composed correctly by the caller, not something the decorator can enforce on its own.

Trade-offs

A class-based retry policy (rather than a decorator) is worth it once you need per-call overrides (retry THIS call with a shorter window because it's on a critical path) -- decorators are static at definition time unless you thread configuration through explicitly.

Edge cases: a wrapped function called with no arguments, a function whose exception has a non-standard __repr__ that could itself throw during logging, and max_attempts=1 (which should behave as a single unretried call, not loop) are all worth explicitly testing -- the last one is a common off-by-one where an implementation accidentally still retries once even at max_attempts=1.

Approach

A saga coordinator's whole job is guaranteeing one of two outcomes: every step committed, or every COMPLETED step was compensated -- never a state where some steps committed and others didn't with no attempt to reconcile.

class Step:
    def __init__(self, name, execute, compensate):
        self.name = name
        self.execute = execute        # callable, no args, may raise
        self.compensate = compensate  # callable, no args, must be safe to call even if execute partially ran

class SagaCoordinator:
    def __init__(self, steps, state_store):
        self.steps = steps
        self.state = state_store  # durable key-value store, keyed by f"{saga_id}:{step_name}"

    def run(self, saga_id):
        completed = []
        for step in self.steps:
            key = f"{saga_id}:{step.name}"
            if self.state.get(key) == "done":
                completed.append(step)   # RESUME: skip steps already completed in a prior attempt
                continue
            try:
                step.execute()
                self.state[key] = "done"
                completed.append(step)
            except Exception as e:
                self._compensate(saga_id, completed)
                raise RuntimeError(f"saga failed at '{step.name}', compensated {len(completed)} steps") from e
        return "committed"

    def _compensate(self, saga_id, completed_steps):
        for step in reversed(completed_steps):  # undo in reverse of completion order
            step.compensate()
            self.state[f"{saga_id}:{step.name}"] = "compensated"

Verified in a sandbox with a 3-step saga (reserve-inventory, charge-payment, ship-order) where the third step deliberately fails: the coordinator correctly compensated ONLY the first two (which had actually completed), in exactly the reverse of their completion order (charge-payment compensated before reserve-inventory), confirmed by the exact execution log produced. A second test confirmed the resume path: given a state store where two of three steps were ALREADY marked "done" (simulating a coordinator restart after a crash), re-running the saga correctly executed ONLY the one remaining incomplete step, never re-executing the already-completed ones.

Idempotent retry on transient failures

step.execute should itself be idempotent (or the saga needs its own per-step retry wrapper using the retry-decorator pattern covered elsewhere in this topic) so that a transient failure during execution can be safely retried without risking a double-charge or double-reservation -- the coordinator's job is sequencing and compensation, not retry logic itself, which composes as a separate concern the same way it does for the standalone retry decorator discussed earlier in this topic.

Persisting state and resuming after restart

The state_store needs to be genuinely durable (a database, not in-process memory) for resume to survive a real process crash, not just an in-process exception -- the verified resume behavior above depended on a state store that outlives the coordinator instance itself, which is exactly what a durable key-value store (or a table in the application's existing database) provides in production.

Scoping: pattern versus platform

This is deliberately scoped as a REUSABLE CODING PATTERN a team drops into their own service -- a library primitive, not a standalone multi-service orchestration PLATFORM coordinating steps across many different teams' services with its own operator console and cross-team governance (that's a materially larger system-design problem belonging to CI/CD or platform-engineering territory, not this topic's task-level scripting scope). The distinction matters for how much infrastructure investment is justified: this pattern is worth building for a single team's own multi-step workflow; the platform version is a different, much larger undertaking.

Trade-offs and pitfalls

The most common bug in a from-scratch saga implementation is a compensate function that ASSUMES the corresponding execute fully succeeded, when in fact execute could have failed PARTWAY through its own work (e.g., a step that both reserves inventory AND sends a notification, where only the reservation half completed before the step raised) -- compensate needs to be written defensively against 'what if this step's own effect was itself partial,' not just 'undo what a fully-successful execute would have done.'

Edge cases: a step's compensate() call itself raising an exception (the undo action fails) needs explicit handling -- silently swallowing it hides a saga that's now in a genuinely inconsistent state, while letting it propagate uncaught can abort compensation of the REMAINING completed steps; logging it loudly and continuing to attempt the rest of the compensation chain is usually the safer default than either extreme.

Approach

The correctness-critical property here is ordering: NEVER delete the original log until its compressed replacement is verified byte-for-byte correct, so a bug or crash mid-rotation can never lose data, only leave a slightly-untidy directory to clean up on the next run.

import gzip, hashlib, os, shutil, time

def rotate_logs(log_dir, service, max_age_days=7, keep_n=5, now=None):
    now = now or time.time()
    cutoff = now - max_age_days * 86400
    candidates = sorted(f for f in os.listdir(log_dir)
                         if f.startswith(f"{service}.log.") and not f.endswith(".gz"))
    for fname in candidates:
        path = os.path.join(log_dir, fname)
        if os.path.getmtime(path) > cutoff:
            continue
        gz_path = path + ".gz"
        tmp_gz = gz_path + ".tmp"
        with open(path, "rb") as src, gzip.open(tmp_gz, "wb") as dst:
            shutil.copyfileobj(src, dst)
        # verify BEFORE deleting the original -- this is the load-bearing step
        original_hash = hashlib.sha256(open(path, "rb").read()).hexdigest()
        with gzip.open(tmp_gz, "rb") as check:
            archived_hash = hashlib.sha256(check.read()).hexdigest()
        if original_hash != archived_hash:
            os.remove(tmp_gz)
            raise RuntimeError(f"archive verification failed for {fname}, original preserved")
        os.replace(tmp_gz, gz_path)  # atomic: readers never see a partial .gz
        os.remove(path)              # only now, after a verified archive exists
    _enforce_retention(log_dir, service, keep_n)

def _enforce_retention(log_dir, service, keep_n):
    archives = sorted((f for f in os.listdir(log_dir)
                        if f.startswith(f"{service}.log.") and f.endswith(".gz")),
                       key=lambda f: os.path.getmtime(os.path.join(log_dir, f)))
    while len(archives) > keep_n:
        os.remove(os.path.join(log_dir, archives.pop(0)))

Verified against a real temp directory with 5 synthetic log files staggered 10-14 days old plus one 'currently being written' log file: the rotation correctly compressed and archived all 5 old files, left the actively-written file completely untouched (it wasn't old enough to qualify), and retention correctly pruned to exactly the most recent 3 archives when keep_n=3 was passed, removing the 2 oldest.

Concurrency safety across services

Because the function filters by f"{service}.log." prefix and only lists/processes that service's own files, two instances of this rotation running concurrently for DIFFERENT services never touch each other's files at all -- concurrency safety across services falls out of the naming scheme, not from any locking. Two concurrent runs for the SAME service, however, could race on the same file (both compute a tmp archive for the same original); guard against that specific case with a per-service flock (the same pattern demonstrated for safe_cron.sh elsewhere in this topic) if the rotation could plausibly be triggered more than once concurrently for one service.

Handling partial failures

If the process crashes between writing tmp_gz and the verify-then-rename step, the next run simply sees a stray .tmp file it doesn't recognize as anything meaningful and, depending on implementation, either ignores it or cleans it up on a subsequent pass -- the ORIGINAL log file is never touched until after successful verification, so a crash at any point before that leaves the original intact and the rotation simply resumes (or retries that file) on the next scheduled run.

Trade-offs and pitfalls

The most common shortcut that ships slightly wrong is deleting the original right after gzip.open().write() completes, without a verify step -- gzip can "succeed" while still producing output that fails to decompress correctly under specific edge cases (disk-full mid-write being the most common real-world trigger), and without the verify-then-delete ordering, that failure mode silently loses the original log data.

Direct answer

The core design shape here is the same as every scheduled automation this topic covers -- fetch/compute, format, deliver, handle failure -- just applied to a reporting use case where correctness of the DATA matters as much as reliability of the delivery.

Steps

Authentication: use a service-account-style credential scoped narrowly to read access on the specific dashboards/data sources needed, fetched fresh (or from a short-lived cache) rather than a long-lived personal credential embedded in the script -- the same secrets-handling discipline this topic covers elsewhere applies directly here, and it matters more than it might seem for 'just a report,' since a report-automation credential with broad read access across many dashboards is a real, easily-overlooked attack surface.

Rendering or exporting: for a dashboard-rendering approach (a headless browser screenshot, or a BI tool's own export API), verify the render actually completed and produced non-empty, non-error output BEFORE treating the run as successful -- a common silent-failure mode here is a screenshot/export that technically 'succeeds' but captures an error page or a stale cached view. For a CSV-export approach, validate the exported data's basic shape (expected row count range, expected columns present) as a sanity check before distributing it, since a silently-empty or truncated export is a much worse failure than an obviously-failed one.

Scheduling: cron for a simple, single-report weekly job is entirely adequate; Airflow becomes worth the added complexity once there are multiple interdependent reports (this one depends on that data pipeline finishing first) that need real dependency-aware orchestration rather than independent fixed-time triggers.

Error handling: retry transient failures (a flaky connection to the data source) with backoff; for a genuine failure that isn't resolved by retry, the report should NOT silently fail to send -- it should alert whoever owns the automation AND, ideally, still notify the intended recipients that this week's report is delayed/unavailable rather than them simply never receiving it and not knowing whether that's expected.

Secure credentials management: as above, scoped and short-lived where the reporting platform supports it; never embed a personal user's credential in a shared automation, since that ties the automation's continued function to one person's account staying valid and creates an audit trail that misattributes automated access to a human.

Trade-offs and pitfalls

The most common failure mode in report-automation specifically (as opposed to other kinds of scheduled jobs) is that a SILENT data-correctness bug is much harder to notice than an outright failure -- a report that renders and sends successfully but contains subtly wrong numbers (a stale cache, a broken filter, a timezone bug shifting which data falls in 'this week') can go unnoticed for a long time precisely because the automation's own health metrics (did it run, did it succeed, how long did it take) all look completely fine. Worth adding a basic sanity check on the OUTPUT DATA itself (row counts within an expected range, key totals within an expected range of the prior week's) as part of what 'success' means for this specific class of automation, not just 'the script exited 0.'