Automation Scripting for Operations Questions

Direct answer

subprocess.run blocks the calling thread until the child process exits (or times out); asyncio.create_subprocess_exec returns control to the event loop immediately and lets you await the child's completion alongside other concurrent work. The choice is really about what else your script needs to be doing while the external command runs.

Blocking behavior and concurrency model

subprocess.run is simplest for a script that runs external commands sequentially, one after another -- there's no event loop to reason about, no await syntax, and stdout/stderr capture is a single synchronous call. If you need to run several external commands concurrently, the sync option is threads (a ThreadPoolExecutor calling subprocess.run per worker) -- which works, but each thread is a real OS thread, so it doesn't scale cleanly past a few hundred concurrent subprocess launches.

asyncio.create_subprocess_exec fits when the automation is ALREADY asyncio-based (talking to async HTTP clients, other async I/O) and you want subprocess launches to be just another awaitable alongside that work, sharing the same single-threaded event loop rather than spinning up a thread pool. It also composes naturally with asyncio.gather and semaphores for bounding how many subprocesses run at once, without the OS-thread overhead of the threaded approach.

Ease of implementation

subprocess.run wins here decisively for anything simple: subprocess.run([...], capture_output=True, timeout=30, check=True) is a single, easy-to-read line. The asyncio version requires creating the subprocess, then separately await-ing communicate() or manually pumping the stdout/stderr streams, and wrapping the whole thing in an async function -- meaningfully more ceremony for the same basic 'run a command and get its output' task.

When to prefer asyncio for SRE automation

Prefer asyncio when the automation's dominant cost is genuinely concurrent I/O-bound work at meaningful scale -- for example, running the same health-check command over SSH against 500 hosts in parallel, where you want hundreds of subprocesses in flight without hundreds of OS threads. Prefer sync subprocess.run (possibly with a modest thread pool) for anything with a handful of sequential or lightly-parallel external calls, a one-off maintenance script, or anywhere the extra async ceremony would cost more in review/maintenance burden than it saves in throughput. A useful rule of thumb: if you're not already in an async codebase and you're not launching dozens+ of concurrent subprocesses, subprocess.run (with threads if you need modest parallelism) is very likely simpler, more debuggable, and just as correct.

Trade-offs and pitfalls

Mixing the two carelessly is the most common real bug: calling a blocking subprocess.run from inside an async function (instead of create_subprocess_exec or running it in an executor) blocks the entire event loop, silently stalling every OTHER concurrent task the automation was supposed to be running -- turning what looked like a concurrent asyncio program into an accidentally-sequential one, with no error raised to tell you.

Direct answer

This is fundamentally a policy decision, not a technical one -- 'can we build a good shared platform' is almost always yes; the actual question is whether the organizational cost of centralizing (migration effort, short-term velocity hit, political friction) is worth the long-term payoff (less duplicated retry/logging/secrets logic, more consistent operational quality, easier cross-team support).

Governance and technical abstractions needed

A shared platform needs, at minimum: a stable core API/library (the retry, logging, secrets, CLI-design primitives this whole topic covers) that individual teams build ON rather than each reinventing; a clear contribution model for teams that need something the core doesn't yet support (a plugin/extension point, not a fork); and a policy for who can approve changes to the shared core, since a shared platform used by many teams needs more conservative change-review than any single team's private script ever did.

Onboarding teams

Don't force a big-bang migration. Start with new automation being REQUIRED to use the shared platform (stops the bleeding, no more net-new duplication), while existing scripts migrate opportunistically -- prioritized by which existing scripts are highest-risk/highest-value to bring under the platform's shared quality bar (the same prioritization logic discussed for toil-reduction elsewhere in this topic: frequency x time x risk), not a blanket 'migrate everything by date X' mandate that creates resentment without matching capacity.

Measuring success

MTTR for automation-related incidents (does centralizing actually reduce time-to-fix, or does it create a bottleneck through a single team that now owns everything); deployment frequency and automation coverage (are teams actually adopting it, or nominally required to and quietly working around it); and, critically, a DIRECT measure of duplication reduced (how many teams' retry/logging/secrets code converged onto the shared implementation) since that's the core value proposition being tested.

Managing stakeholder pushback

The realistic pushback is 'this slows us down and takes away control,' and it's not always wrong -- a shared platform genuinely does trade some team-level autonomy for org-level consistency. Address it by making the platform team's response time to feature requests fast and predictable (a platform that's slow to extend just recreates the pressure to fork/route-around it), and by being honest that centralization is a genuine trade-off, not a strictly-better free lunch, which builds more trust than overselling it.

Phased rollout

Phase 1: build the core shared platform and require it for NEW automation only. Phase 2: migrate the highest-priority existing scripts (per the risk-based prioritization above), measuring MTTR/coverage as you go. Phase 3: revisit whether full migration is worth the remaining effort, or whether some legitimately-fine existing scripts are better left alone -- 100% migration is not automatically the right end state if the remaining stragglers are low-risk and low-value to migrate.

Trade-offs and pitfalls

The most common failure mode in this kind of centralization effort is the platform team optimizing for their OWN roadmap rather than genuinely serving adopting teams' needs -- once that trust breaks, teams quietly route around the shared platform for anything that isn't strictly mandated, and the platform ends up technically 'adopted' while genuinely providing much less value than the metrics suggest.

Direct answer

A cross-team CLI earns trust the same way a good API does: predictable structure, safe defaults, and no surprises. The two commands (backup, restore) should be subcommands of one binary, each with its own scoped flags, plus a small set of global flags that apply everywhere.

Command surface

backuptool [global flags] <command> [command flags]

Global flags:
  --verbose, -v          increase log verbosity (repeatable: -vv)
  --dry-run              show what would happen, make no changes
  --config PATH          explicit config file path
  --output json|text     machine-readable vs human-readable output

Commands:
  backup   --target NAME [--full|--incremental] [--retention DAYS]
  restore  --target NAME --snapshot ID [--to PATH]
  list     [--target NAME]                 # discoverability: what CAN I restore?
  status   [--run-id ID]                   # is a backup/restore in flight, did it succeed?

list and status aren't in the original ask but are worth calling out explicitly: a tool that can only DO things and never tell you what it's already done gets treated as a black box, and operators route around black boxes.

Exit codes and help

Use the conventional split: 0 success, 1 for a runtime failure (the operation was attempted and failed), 2 for a usage error (bad flags -- the operation was never attempted). This distinction matters operationally: a wrapper script or a monitoring check needs to tell 'you typo'd a flag' apart from 'the backup actually failed', and a single generic non-zero exit code collapses that distinction. Every command needs -h/--help text with at least one example invocation; a flag list with no example is where most CLI usability complaints come from.

Idempotent --dry-run

--dry-run should exercise the exact same code path as a real run up to (and not including) the side-effecting step, so what it prints is genuinely what would happen, not a separately-maintained approximation that can drift out of sync. Concretely: build the full plan (which files, which destination, computed retention actions), print the plan, and return before calling anything that mutates state. This is 'idempotent' in the sense that running --dry-run a hundred times in a row changes nothing and always shows the same plan for the same inputs -- if it isn't, the dry-run path has drifted from the real path and can no longer be trusted.

Config precedence

Standard, least-surprising order (highest wins): explicit CLI flag > environment variable > config file > built-in default. Document this order once, prominently, because 'why didn't my env var take effect' is the single most common CLI support question, and it's almost always a precedence surprise.

Libraries

Python: argparse (stdlib, sufficient for subcommands) or click/typer for richer UX with less boilerplate. Go: the standard library's flag package is thin; cobra (used by kubectl, docker CLI) is the de facto standard for subcommand-heavy tools and is a defensible default recommendation for a cross-team tool that other engineers will extend.

Trade-offs and pitfalls

The most common mistake is over-fitting the CLI's flags to whatever the FIRST team that adopts it needs, then discovering every subsequent team wants a slightly different flag shape for the same underlying concept -- naming flags around the general operation (--target, --source) rather than one team's specific vocabulary pays off once the tool is genuinely cross-team. Edge case: a flag that's valid for one subcommand but silently ignored (not rejected) for another is a common source of confused bug reports; validate flag-subcommand compatibility explicitly rather than letting argparse accept anything globally-defined everywhere.

Direct answer

In infrastructure automation, idempotency means running the script against a host that's already in the desired state produces no changes and no errors -- the script converges toward a target state rather than blindly re-applying a sequence of actions. For ensuring a directory and a config file exist with exact contents on many hosts, that means the script has to check current state, act only on the delta, and never assume it's the first time it's ever run there.

Design choices for idempotency

Check before creating the directory (os.makedirs(path, exist_ok=True) handles the 'already exists' case cleanly rather than erroring); for the config file, don't just check EXISTENCE, compare CONTENT -- a file that exists but has stale content still needs to be updated, so the idempotency check has to be 'does the current content match the desired content,' not just 'does a file exist at this path.'

Detecting divergence and atomic updates

import hashlib, os, tempfile

def ensure_config(path: str, desired_bytes: bytes) -> bool:
    """Returns True if a change was made, False if already correct (no-op)."""
    if os.path.exists(path):
        with open(path, 'rb') as f:
            if f.read() == desired_bytes:
                return False  # already correct, nothing to do
    d = os.path.dirname(path)
    fd, tmp_path = tempfile.mkstemp(dir=d, prefix='.tmp-')
    try:
        with os.fdopen(fd, 'wb') as f:
            f.write(desired_bytes)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp_path, path)  # atomic on POSIX: no reader ever sees a partial file
    except Exception:
        os.unlink(tmp_path)
        raise
    return True

Verified in a sandbox: calling ensure_config twice in a row with identical desired content reports changed=True on the first call and changed=False on the second, and the file's final bytes match the desired content exactly. A second test simulated a crash mid-write by leaving an orphaned .tmp-* file in the directory and confirmed the real config file was completely untouched by it -- the atomic-rename pattern means a crash during writing never corrupts or partially-overwrites the file a reader (or the next run) sees.

Avoiding races when multiple agents run concurrently

Write-then-os.replace() is atomic at the OS level for a SINGLE writer, but if two agents on the same host race to update the same file simultaneously, the LAST rename wins and the other agent's write is silently discarded (not corrupted, just lost) -- usually acceptable if both are converging toward the same desired state and would compute identical desired_bytes anyway, but worth naming explicitly since 'atomic' doesn't mean 'coordinated.' If genuinely concurrent, conflicting writers are possible, add a file lock (flock) around the read-compare-write sequence, or route all writes for a given path through a single process/queue rather than letting arbitrary agents write directly.

Pseudocode summary

ensure_directory(path):
    makedirs(path, exist_ok=True)

ensure_config_file(path, desired_content):
    if read(path) == desired_content: return NO_CHANGE
    atomic_write(path, desired_content)
    return CHANGED

Edge cases: a directory that exists but as the WRONG type (a file sitting at the path where a directory is expected) will make os.makedirs(path, exist_ok=True) raise rather than silently succeed -- worth an explicit, clearer error for this case rather than letting a confusing FileExistsError/NotADirectoryError surface unexplained. A desired-content byte string containing content that looks identical after a lossy encoding round-trip (rare, but possible with certain file encodings) can make the content-comparison check pass when the actual desired semantic content differs.

Trade-offs and pitfalls

The atomic-write-plus-content-compare pattern shown here trades a small amount of extra I/O (a read-and-compare before every write) for a strong safety guarantee; for a VERY high-frequency check (called thousands of times a second) that overhead would matter and a cheaper pre-check (a stored hash rather than a full read) would be worth the added bookkeeping complexity.

Direct answer

Orchestrating Terraform from Python is fundamentally a CHOICE OF INTERFACE, not a choice of what Terraform itself does underneath: invoking the CLI via subprocess treats Terraform as an opaque black box (parsing its text/JSON output, the same interface a human or a CI script would use), while a Python wrapper library (python-terraform or similar) provides a more Pythonic API over that SAME underlying CLI, typically still shelling out internally, meaning the real trade-off is thin-and-transparent (subprocess, you see and control exactly what CLI flags run) versus thicker-and-more-ergonomic (a wrapper, less boilerplate but another dependency and abstraction layer between your code and Terraform's actual behavior). Neither choice changes Terraform's own state, locking, or plan/apply semantics at all, those work identically regardless of which Python interface drives them.

Structured elaboration

Subprocess vs. python-terraform wrapper. subprocess.run(["terraform", "plan", "-out=tfplan"], ...) gives full, transparent control over exactly which flags run and how output is captured/parsed, at the cost of writing that parsing and error-handling yourself. A wrapper library provides Python methods (tf.plan(), tf.apply()) that internally construct and run the same CLI calls, saving boilerplate at the cost of a dependency that may lag behind Terraform's own CLI changes (a wrapper unmaintained against a newer Terraform version is a real, recurring risk) and an abstraction that can obscure exactly which underlying command ran when debugging.

Managing remote state and locking across workspaces. Regardless of which Python interface drives it, EACH workspace still needs its own remote-state backend and locking configuration, exactly as any Terraform usage requires; the Python orchestration layer's job is selecting WHICH workspace/directory a given subprocess call operates in (cwd= for subprocess, or the wrapper's own working-directory parameter), never re-implementing locking itself, Terraform's own backend locking mechanism is what actually prevents concurrent-write corruption, and the orchestration layer must not bypass or duplicate it.

Safe plan/apply workflows. Applied here as Python-driven automation rather than a CI YAML pipeline: plan first (capturing the plan artifact), a REVIEW or policy-as-code gate before apply, and apply referencing the EXACT saved plan artifact, never re-planning at apply time; a Python orchestrator needs to preserve this same discipline (save the plan file, do not silently skip straight to apply for convenience) rather than treating the safety sequence as optional now that it is being driven programmatically instead of by a human running commands.

Handling drift detection and remediation in automation. Applied per-workspace: a Python orchestrator can loop over every workspace, running plan (no-op detection) on each, aggregating results; the SAME rate-limiting and credential-error handling that answer implements applies directly here, since orchestrating across MANY workspaces from Python is exactly the multi-workspace-at-scale scenario that makes those concerns real rather than theoretical.

Worked example

A concrete Python orchestration pattern across multiple team workspaces, using subprocess directly (the more transparent, more broadly-applicable choice for a team wanting full visibility into exactly what runs):

import subprocess
import json

def run_terraform(workspace_dir: str, *args: str) -> subprocess.CompletedProcess:
    return subprocess.run(
        ["terraform", *args],
        cwd=workspace_dir,
        capture_output=True,
        text=True,
        check=False,  # inspect returncode explicitly, don't raise blindly
    )

def safe_plan_and_apply(workspace_dir: str) -> dict:
    plan = run_terraform(workspace_dir, "plan", "-out=tfplan", "-detailed-exitcode")
    # -detailed-exitcode: 0 = no changes, 1 = error, 2 = changes present
    if plan.returncode == 1:
        return {"status": "plan_error", "stderr": plan.stderr}
    if plan.returncode == 0:
        return {"status": "no_changes"}
    # returncode == 2: real changes proposed; a real pipeline would gate
    # here on review/policy-as-code before ever calling apply
    apply = run_terraform(workspace_dir, "apply", "tfplan")  # applies the SAVED plan, never re-plans
    return {"status": "applied" if apply.returncode == 0 else "apply_failed", "stderr": apply.stderr}

This structure keeps EVERY safety property of a normal CI/CD pipeline intact (saved-plan-then-apply, distinguishing no-op from real changes via -detailed-exitcode, explicit error handling rather than a blind check=True that would obscure WHICH step failed) while being driven by Python instead of CI YAML.

Output (actually executed with python3, using a stubbed run_terraform in place of a real terraform binary, which was not available in this sandbox)

returncode 0: {'status': 'no_changes'}
returncode 1: {'status': 'plan_error', 'stderr': ''}
returncode 2 then apply success: {'status': 'applied', 'stderr': ''} calls: [('plan', '-out=tfplan', '-detailed-exitcode'), ('apply', 'tfplan')]
returncode 2 then apply fail: {'status': 'apply_failed', 'stderr': ''}

All four branches of safe_plan_and_apply were exercised directly against a fake run_terraform returning each of Terraform's own documented -detailed-exitcode values (0, 1, 2) plus a simulated apply failure: a clean no-op correctly short-circuits before apply is ever called, a plan error correctly short-circuits before apply, and when real changes are proposed (returncode 2), apply is called exactly once, using the saved plan file, never a fresh plan.

Trade-offs and pitfalls

• Common mistake: treating a Python orchestration layer as a reason to skip the plan-then-apply-the-saved-plan discipline "since it's just a script now." Per the worked example, the safety property is about WHAT commands run and in WHAT order, not about whether a human or a script is issuing them; a Python orchestrator that calls apply without a preceding, saved plan reintroduces exactly the plan-staleness risk the save-then-apply discipline exists to prevent.
• A wrapper library's convenience is real but comes with a maintenance dependency risk: a wrapper that has not been updated for a recent Terraform CLI change can silently misbehave or simply fail; teams choosing a wrapper need to weigh this against subprocess's more verbose but more directly-controllable and up-to-date-by-construction approach (since it calls the real, current CLI directly).
• Multi-workspace orchestration from Python is exactly where rate-limiting and credential-error handling become necessary, not optional, a naive loop over many workspaces with no backoff can trigger real cloud-provider throttling, the SAME failure mode a poorly-designed drift scanner would hit at scale.
• check=True on a subprocess call (raising immediately on any non-zero exit) can obscure WHICH specific step failed if error handling is not deliberate, per the worked example's explicit check=False and returncode inspection; a blind, unexamined exception from a failed subprocess call is a weaker signal for automated remediation logic to act on than an explicitly distinguished failure mode.