Automation Scripting for Operations Questions

Approach

The two hard requirements -- dependency-aware parallelism and reverse-order rollback -- both come down to correctly tracking which steps actually COMPLETED, not just which steps exist in the YAML.

import concurrent.futures as cf

def run_step(name, executed_log, fail_on=None):
    if fail_on == name:
        raise RuntimeError(f"{name} failed")
    executed_log.append(name)

def run_dag(steps, fail_on=None):
    by_name = {s["name"]: s for s in steps}
    done, completed_order, failure = set(), [], None
    remaining = list(steps)
    with cf.ThreadPoolExecutor(max_workers=4) as pool:
        while remaining and failure is None:
            batch = [s for s in remaining if all(d in done for d in s["deps"])]
            if not batch:
                break  # nothing ready: either done, or a dependency cycle
            futures = {pool.submit(run_step, s["name"], [], fail_on): s for s in batch}
            for fut in cf.as_completed(futures):
                s = futures[fut]
                try:
                    fut.result()
                    done.add(s["name"]); completed_order.append(s["name"]); remaining.remove(s)
                except Exception as e:
                    failure = (s["name"], e)
    return completed_order, failure

def rollback(steps, completed_order):
    by_name = {s["name"]: s for s in steps}
    return [by_name[n]["rollback"] for n in reversed(completed_order) if by_name[n].get("rollback")]

Verified in a sandbox against a 4-step DAG (create-network and create-db with no dependencies, create-app depending on both, smoke-test depending on create-app): the success path correctly ran create-network/create-db before create-app, and create-app before smoke-test, confirming dependency ordering was respected even though network and db ran concurrently in the same batch. A second run forced create-app to fail: only create-network and create-db had completed, and the rollback function produced exactly their rollback actions (delete-network, drop-db) in the REVERSE of whichever order they actually finished in -- confirming rollback correctly scopes to what genuinely completed, not the full step list, and correctly reverses actual completion order rather than YAML declaration order.

Dependency-based parallelism

Steps become 'ready' the moment every dependency in their deps list is in the done set -- this naturally lets independent steps (no shared dependencies) run concurrently within a ThreadPoolExecutor, while dependent steps wait. The loop re-evaluates 'what's ready now' after every completed batch, so as soon as create-network and create-db both finish, create-app becomes ready in the very next iteration without needing to wait for anything else.

Rollback ordering

The critical detail, confirmed by the test above: rollback must reverse the ACTUAL completion order (completed_order, tracked as steps genuinely finish) not the declared YAML order and not a fixed dependency-graph traversal order -- because with concurrent execution, which of two independent steps finishes first is not deterministic, and rollback has to undo whatever the real, as-it-happened execution order was.

Trade-offs and pitfalls

A step with NO rollback field (like smoke-test in the original spec) is correctly skipped during rollback rather than raising an error -- not every step needs to be reversible (a read-only smoke test has nothing to undo), and requiring every step to define one would force teams to write meaningless no-op rollbacks just to satisfy the schema.

Edge cases: a step declared with a dependency on a step NAME that doesn't exist in the YAML (a typo) should fail parsing loudly before any execution starts, not silently treat the dependency as already-satisfied; a step with a SELF-dependency or a genuine cycle in the dependency graph must be detected and rejected before scheduling, since the ready-check loop above would otherwise simply stop making progress with no clear error explaining why.

The goal of a retry strategy is to survive genuinely transient failures without making things worse -- either by hammering an already-struggling service or by silently corrupting state through an unsafe retry.

The backoff/jitter vocabulary

• Constant backoff: wait the same fixed interval between every attempt. Simple, but if many clients fail at once (a deploy, a brief network blip) they all retry in lockstep and re-create the exact spike that caused the failure.
• Exponential backoff: each retry waits roughly base * 2^attempt. Spreads load out over time, but without jitter, many clients that failed at the same moment still retry at the same computed delays -- lockstep survives, it's just spaced out further.
• Full jitter: delay = random(0, base * 2^attempt) -- the whole computed ceiling is randomized down to zero. This is what actually breaks the synchronization: two clients that failed simultaneously now retry at genuinely different, unpredictable times.
• Equal jitter: delay = base*2^attempt/2 + random(0, base*2^attempt/2) -- keeps a guaranteed minimum wait while still spreading. Useful when you want a floor on how soon anyone retries (protects a service that's still overloaded) at the cost of slightly less spread than full jitter.

Bounding the retry

max-attempt limits cap total attempts so a truly broken dependency fails loudly instead of retrying forever. retry windows cap total elapsed time rather than attempt count, which matters more when backoff grows large (5 attempts at exponential backoff could span minutes; you may want to give up on wall-clock time instead). Both should exist together: attempts to bound retry density, a window to bound retry duration.

Idempotency is the real gate

Retrying is only safe if re-running the operation doesn't double its effect. A GET is naturally safe to retry. A POST that charges a card or inserts a row is not, unless the operation itself is made idempotent (an idempotency key the server deduplicates on, a conditional write, an upsert instead of an insert). The rule of thumb: never blindly retry a side-effecting operation without first asking 'if the first attempt actually succeeded and only the response was lost, what happens when I resend it?'

When to circuit-break instead

Retrying assumes the failure is transient and isolated to this one call. A circuit breaker exists for the case where the failure is systemic -- the downstream service is down or overloaded, and every caller retrying it is making the outage worse. Once a failure rate crosses a threshold, the breaker trips: stop calling the dependency for a cooldown window (return a fast failure instead), then send a small number of probe requests to see if it's recovered before fully closing the circuit again.

A mistake that causes cascading failures

The classic one: constant or unjittered exponential backoff at scale. A downstream dependency has a brief blip; hundreds of clients all fail at the same moment and all retry at the same computed intervals, turning a brief blip into a sustained self-inflicted DDoS on the dependency just as it's trying to recover. This is exactly why full jitter exists -- without it, adding retries can make an outage longer, not shorter.

Trade-offs and pitfalls

A subtler pitfall than the cascading-failure mistake above: setting max-attempt limits too high relative to a caller's own timeout budget means the caller gives up (times out) before the retry loop itself has exhausted its attempts, so the retries never actually get a chance to help -- the retry policy and the caller's own timeout need to be sized together, not chosen independently. Edge case: a dependency that returns a MIX of retryable and non-retryable errors across a single call (e.g., a batch API where some sub-results succeeded and others failed) needs per-item, not per-call, retry logic.

Approach

For a backup/restore CLI, argparse subparsers give you isolated flag namespaces per command plus free -h text, which is exactly what a multi-command tool needs.

import argparse, os, sys

def build_parser():
    p = argparse.ArgumentParser(prog="backuptool")
    p.add_argument("--verbose", action="store_true")
    p.add_argument("--dry-run", action="store_true")
    sub = p.add_subparsers(dest="command", required=True)

    backup = sub.add_parser("backup")
    backup.add_argument("--path", required=True)

    restore = sub.add_parser("restore")
    restore.add_argument("--version", required=True)
    return p

def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)
    try:
        if args.command == "backup":
            if not os.path.exists(args.path):
                print(f"error: path does not exist: {args.path}", file=sys.stderr)
                return 2  # usage error: bad input, nothing was attempted
            if args.dry_run:
                print(f"[dry-run] would back up {args.path}")
                return 0
            # ... real backup logic would go here ...
            return 0
        elif args.command == "restore":
            if not args.version.strip():
                print("error: --version must be non-empty", file=sys.stderr)
                return 2
            if args.dry_run:
                print(f"[dry-run] would restore version {args.version}")
                return 0
            # ... real restore logic would go here ...
            return 0
    except Exception as e:
        print(f"runtime error: {e}", file=sys.stderr)
        return 1  # the operation was attempted and failed

if __name__ == "__main__":
    sys.exit(main())

Key points

• add_subparsers(dest="command", required=True) makes argparse itself reject an invocation with no subcommand (exit code 2, argparse's own usage-error convention), rather than the script having to check for None manually.
• --path existence is validated explicitly and returns 2 (usage error, nothing attempted) rather than 1 -- the caller typed a bad path, the tool never touched anything.
• The runtime try/except around actual command dispatch is what earns exit code 1: something was ATTEMPTED and failed, as opposed to a bad invocation.
• --dry-run returns before any state-mutating call, and is checked identically in both subcommands so the pattern generalizes -- the same shape applies whether the tool grows a deploy --target subcommand or a run/plan/apply triad; the CLI skeleton doesn't change, only what's inside the dry-run branch does.

Complexity

Argument parsing itself is O(number of flags) with argparse's built-in machinery; not a meaningful cost. The design cost that matters is keeping validation (usage errors) and execution (runtime errors) cleanly separated so the two failure classes map to two different exit codes consistently across every subcommand.

Edge cases

restore --version "" (empty string passes required=True since the flag was technically supplied) must be checked explicitly, which the code above does. Unknown subcommands and missing required flags are handled by argparse itself, exiting 2 automatically. A --path that exists but isn't readable (permissions) should also surface as a runtime error (1), not a silent failure -- worth calling out even though it's not in the original spec, because it's the kind of edge case that ships broken in a first draft.

Edge cases: an empty --path/--version argument passed as a whitespace-only string technically satisfies required=True and needs its own explicit check (the code above already handles this for --version); a --path that exists but isn't readable due to permissions should surface as a runtime error (exit 1), not crash with an uncaught PermissionError traceback.

Trade-offs and pitfalls

A hand-rolled argparse skeleton like this trades a little boilerplate for full control over exit-code semantics; a higher-level framework (click, typer) would reduce the boilerplate at the cost of the framework choosing some of those conventions for you, which matters if the team needs exit codes to match an existing internal standard rather than the framework's own defaults.

Direct answer

The trade-off is really flexibility-and-control versus maintenance burden, and the honest answer is that mature tools should be the default, with custom Python tooling justified only when a specific, named gap makes the mature tool a genuinely bad fit.

What custom tooling costs you

Every custom automation you write, you also now own: security patching, keeping pace with the underlying APIs it wraps, documentation, onboarding new contributors to a bespoke tool instead of a widely-known one, and the ongoing maintenance burden of a codebase that's smaller and less battle-tested than Ansible/Terraform/Helm, which have years of edge cases already found and fixed by a much larger user base than any single team's tool will ever have.

What mature tools cost you

Mature tools trade flexibility for convention: Terraform's declarative model is excellent for infrastructure provisioning but awkward for genuinely imperative, conditional logic; Ansible's YAML-based approach is readable and has a huge module ecosystem but can become unwieldy for complex branching logic that's much more naturally expressed in a real programming language; Helm's templating (Go templates over YAML) is a well-known source of debugging pain once charts get complex. Adopting one also means accepting its opinions, release cadence, and occasionally working around a limitation rather than just fixing it yourself.

Decision criteria

Build custom when: the logic is genuinely imperative/conditional in a way that fights the mature tool's declarative model; you need tight integration with in-house systems that have no existing provider/module and writing one would be more work than the custom script; or the mature tool's abstraction genuinely doesn't fit the problem shape (e.g., orchestrating a sequence of application-level API calls with complex conditional branching, which Terraform's resource-graph model isn't designed for). Reach for the mature tool when: the task is squarely inside its designed use case (provisioning cloud infrastructure -> Terraform; configuration management across a fleet -> Ansible; deploying/templating Kubernetes manifests -> Helm), you want the security/compliance benefit of a widely-audited, actively-maintained tool rather than a bespoke one, or multiple teams need to reuse the same capability and a standard tool means shared knowledge across the org rather than tribal knowledge of one team's custom script.

Concrete examples

Justified custom Python: a script that reconciles state across three different internal systems with business-logic-heavy conditional rules no existing provider models, where the 'infrastructure' being managed isn't cloud resources at all but internal application state. Justified reliance on existing tools: provisioning a new VPC and its subnets (squarely Terraform's job), or ensuring a fleet of 500 servers has a consistent package set and config (squarely Ansible's job) -- writing a custom Python tool for either of these is very likely reinventing a solved problem worse and with less community-tested edge-case coverage.

Trade-offs and pitfalls

The realistic failure mode isn't picking custom-when-it-should-be-mature or vice versa in one big decision, it's SCOPE CREEP: a custom script starts as a genuinely justified small gap-filler and slowly grows into a large, unmaintained pseudo-framework because it was easier to keep extending than to stop and ask 'does a mature tool now cover most of what this has grown into.' Revisit the decision periodically, not just once at the start.

Direct answer

subprocess.run blocks the calling thread until the child process exits (or times out); asyncio.create_subprocess_exec returns control to the event loop immediately and lets you await the child's completion alongside other concurrent work. The choice is really about what else your script needs to be doing while the external command runs.

Blocking behavior and concurrency model

subprocess.run is simplest for a script that runs external commands sequentially, one after another -- there's no event loop to reason about, no await syntax, and stdout/stderr capture is a single synchronous call. If you need to run several external commands concurrently, the sync option is threads (a ThreadPoolExecutor calling subprocess.run per worker) -- which works, but each thread is a real OS thread, so it doesn't scale cleanly past a few hundred concurrent subprocess launches.

asyncio.create_subprocess_exec fits when the automation is ALREADY asyncio-based (talking to async HTTP clients, other async I/O) and you want subprocess launches to be just another awaitable alongside that work, sharing the same single-threaded event loop rather than spinning up a thread pool. It also composes naturally with asyncio.gather and semaphores for bounding how many subprocesses run at once, without the OS-thread overhead of the threaded approach.

Ease of implementation

subprocess.run wins here decisively for anything simple: subprocess.run([...], capture_output=True, timeout=30, check=True) is a single, easy-to-read line. The asyncio version requires creating the subprocess, then separately await-ing communicate() or manually pumping the stdout/stderr streams, and wrapping the whole thing in an async function -- meaningfully more ceremony for the same basic 'run a command and get its output' task.

When to prefer asyncio for SRE automation

Prefer asyncio when the automation's dominant cost is genuinely concurrent I/O-bound work at meaningful scale -- for example, running the same health-check command over SSH against 500 hosts in parallel, where you want hundreds of subprocesses in flight without hundreds of OS threads. Prefer sync subprocess.run (possibly with a modest thread pool) for anything with a handful of sequential or lightly-parallel external calls, a one-off maintenance script, or anywhere the extra async ceremony would cost more in review/maintenance burden than it saves in throughput. A useful rule of thumb: if you're not already in an async codebase and you're not launching dozens+ of concurrent subprocesses, subprocess.run (with threads if you need modest parallelism) is very likely simpler, more debuggable, and just as correct.

Trade-offs and pitfalls

Mixing the two carelessly is the most common real bug: calling a blocking subprocess.run from inside an async function (instead of create_subprocess_exec or running it in an executor) blocks the entire event loop, silently stalling every OTHER concurrent task the automation was supposed to be running -- turning what looked like a concurrent asyncio program into an accidentally-sequential one, with no error raised to tell you.