Python Scripting for Infrastructure Automation Questions

Applied Python skills for building reusable, production grade automation for infrastructure and operations. Topics include designing modular automation code and libraries, using relevant third party libraries for systems administration and remote management, invoking and controlling subprocesses, interacting with application programming interfaces and cloud platform endpoints, robust error handling and structured logging, automated testing of scripts and modules, packaging and distributing tools for reuse, secure credential management, integration with configuration management and orchestration tooling, and designing multi step workflows and idempotent operations. Candidates should demonstrate experience with writing maintainable automation, reasoning about failure modes, and selecting appropriate abstractions and libraries for operational tasks.

EasyTechnical

70 practiced

You need to run a Python automation action across 500 hosts concurrently. Compare an approach building a custom runner using concurrent.futures + paramiko versus using an off-the-shelf orchestration tool like Ansible. Discuss scalability, inventory management, secrets distribution, retry behavior, idempotency, and how you would implement batching and failure isolation with the custom runner.

Sample Answer

Situation: I need to run a Python automation action across 500 hosts concurrently. I'll compare building a custom runner with concurrent.futures + Paramiko vs using Ansible, covering scalability, inventory, secrets, retries, idempotency, and how to batch/fail-isolate if I choose custom.

Short comparison (high level)- Ansible: mature, battle-tested. Handles inventory, parallelism, retries, idempotent modules, secrets via Vault, and built-in failure isolation/rolling updates. Less dev effort, better auditability and community-tested modules.- Custom runner (concurrent.futures + Paramiko): more control and flexibility, but you reimplement a lot (connection pooling, reconnect/backoff, secret handling, logging, idempotency, error classification). Good for very specific edge cases or tight integration with internal systems.

Scalability- Ansible: scales to hundreds/thousands using forks, async actions, AWX/Tower for orchestration and clustering. Handles connection concurrency limits.- Custom: concurrent.futures + Paramiko on a single controller can handle a few hundred concurrent SSH sessions but will hit CPU/network/file-descriptor limits. You must implement connection pooling, session reuse, and maybe horizontal runners (sharding hosts across controllers) to scale beyond that.

Inventory management- Ansible: dynamic inventory plugins (cloud, CMDB), group vars, host vars, and easy targeting.- Custom: you must code inventory loader and grouping; integrate with CMDB/consul/LDAP yourself and handle delta detection.

Secrets distribution- Ansible: integrates with Vault, manages vaulted files, and supports SSH agent, vault passwords, and credential stores in AWX.- Custom: implement secret retrieval (vault/KMS), ephemeral in-memory storage, and use SSH agent forwarding or temporary keys. Must ensure secrets never leak to logs or disk.

Retry behavior & backoff- Ansible: retryable failures can be automated (retries on transient network errors). Playbooks support serial and restart policies.- Custom: implement retry policies with exponential backoff, idempotency checks, and classify errors (transient vs permanent). Use paramiko exceptions to detect network blips and reattempt.

Idempotency- Ansible: many modules are idempotent (package installs, file templates). Playbooks are declarative.- Custom: ensure your Python action is idempotent (check-state -> change -> verify). Add safe guards (locks, leader election when mutating shared resources).

Batching and failure isolation with custom runner- Batching: shard hosts list and use a ThreadPoolExecutor with a controlled max_workers per batch, or process hosts in windows: - Example pattern: create batches of N (e.g., 50) and submit each batch, wait for completion, evaluate results, then proceed.- Failure isolation: run batches serially; if batch failure rate > threshold (e.g., 5%) stop rollout, alert, and roll back. For each host use per-host timeout, capture stdout/stderr/logs to per-host files, and classify failures. Optionally implement circuit breaker per task so repeated failures to a host are skipped.- Implementation sketch (python): use ThreadPoolExecutor for concurrency inside a batch, Paramiko SSH client reused per host, retrieve secrets from KMS just-in-time, maintain a per-host idempotency check (query service or file), and collect per-host result objects. Enforce global rate limits and file-descriptor limits.

When to pick which- Use Ansible for standard ops, predictable idempotent changes, auditing, and when leveraging inventory/secret integrations matters.- Use a custom runner when you need very specific flows, ultra-low-latency control, or when embedding into existing Python tooling — but expect to implement robust retry, secret handling, and scaling concerns.

Recommendation: Prefer Ansible (or AWX/Tower) for a 500-host run unless you have a compelling reason; if custom, implement batching (50–100 hosts), exponential backoff retries, per-host idempotency checks, secret retrieval from Vault/KMS at runtime, and stop-on-threshold failure isolation.

HardTechnical

57 practiced

During a release, rollbacks failed because automation couldn't fetch required secrets (secrets had been rotated or were missing). Describe immediate mitigation steps to restore rollbacks safely, and propose design changes to make rollback automation resilient to secret failures (fallback credentials, local cached encrypted secrets, staged rotation). Also propose CI/policy changes to prevent future secret-related rollback failures.

Sample Answer

Immediate mitigation (incident response, safe and fast):- Triage & containment: Pause further automated rollouts/rollbacks in CI/CD to prevent additional failures.- Fail-open manual path: Escalate to on-call and enable an authenticated, auditable manual rollback runbook that uses human-approved credentials. Use MFA + short-lived session tokens created by operators to execute rollback steps.- Restore minimal secrets: If rotation caused missing secrets, re-create or re-seed only the secrets required for rollback (read-only keys where possible) from the last-known-good encrypted backup in the vault. Use vault audit logs to confirm provenance.- Use emergency fallback credentials: If a delegated emergency key exists (see design below), enable it temporarily with strict TTL and enhanced logging, then perform rollback.- Verify & execute: Dry-run rollback steps in a staging-like environment or with canary subset; once validated, perform rollback and closely monitor metrics, logs, and error budgets.- Post-incident freeze: Lock further secret rotations until root cause and design fixes are applied.

Design changes for resilient rollback automation:- Fallback credentials: Provision a scoped, hardened emergency credential (least privilege, read-only for config/secrets) that lives in a separate secure vault with strict access controls and requires multi-party approval to use. Credential has short TTL and is only for rollback flows.- Local encrypted cache: Maintain a local fallback cache of critical secrets encrypted with a cluster key managed by KMS/HSM. Automation can decrypt with KMS-assisted access when primary vault is unreachable/rotated. Rotate cache keys with staged process.- Staged secret rotation: Implement phased rotation (canary -> group -> global). Rotation must be gated by automated health checks (ability to fetch secrets and run a simulated rollback) before promoting rotation.- Secret versioning & immutable backups: Keep immutable, versioned snapshots of secrets in the vault; rollback automation fetches an explicit version if latest fetch fails.- Health-check endpoints & feature-flagged fallbacks: Automation should probe secret-store health and switch to fallback modes under pre-defined conditions with full audit.

CI/policy changes to prevent recurrence:- Pre-rotation CI tests: Require automated pre-rotation tests in CI that deploy test rotations to staging and run automated rollback simulation against service mocks and canaries. Block rotation merge if tests fail.- Policy-as-code for secrets: Enforce IAM/policy rules that prohibit destructive rotations without passing canary tests and without updating dependent automation artifacts.- Pull-request gating: Secrets/rotation PRs must include a “rollback compatibility” checklist and signed-off runbook; CI validates that automation can retrieve both old and new secret versions.- Monitoring & alerts: Add SLOs for secret-store availability and automated alarms on failed secret fetch rates; alert runbooks specify immediate manual fallback activation.- Regular drills: Schedule chaos/DR drills around secret-store outages and practice using emergency credentials and cached secrets.

Why this approach:- Minimizes blast radius by using least-privilege emergency keys and short TTLs.- Ensures automation degrades predictably to audited manual paths.- Prevents future failures by testing rotations and enforcing policy gates, raising confidence in both rotation and rollback flows.

HardTechnical

52 practiced

You are the SRE lead asked to decide whether to centralize automation into a shared platform or let teams own their individual scripts. Create a migration plan that covers governance (APIs, linting, testing), onboarding steps, required APIs and abstractions, metrics to measure success (MTTR, deployment frequency, automation coverage), and risk mitigation strategies. Explain stakeholder engagement and phased rollout approach.

Sample Answer

Situation: Our org has fragmented automation—teams own scripts with duplication, inconsistent quality, and variable reliability. Goal: centralize automation into a shared Platform Automation Library (PAL) while allowing team autonomy.

Plan (high level, phased):1. Discovery (2–4 weeks)- Inventory scripts, categorize by scope/risk, identify high-value/critical automations.- Stakeholders: SREs, dev leads, security, compliance, QA.

2. Design & Governance (4–6 weeks)- Define platform vision, SLA/SLOs for platform services, and contributor model.- Governance items: - APIs: idempotent, versioned REST/gRPC endpoints for common ops (deploy, rollback, scale, run-checks, secrets access, alert ack). - Abstractions: job/task model, reusable playbooks, parameter schemas, RBAC hooks, audit trail. - Linting & style: enforceable linters (YAML/py/js) and IaC policies (OPA/Rego). - Testing: unit, integration, contract tests; local sandbox runner and CI gates; chaos/rollback tests. - Security: secret handling, least privilege, signed artifacts.

3. Build MVP (6–8 weeks)- Core APIs + CLI/SDKs, template library, CI pipelines, test harness, docs.

4. Onboarding & Migration (iterative)- Pilot with 2 product teams for 2 sprints: pair-migrate 1 critical script each, run dual-run for 1 week.- Create onboarding: templates, migration checklist, office hours, internal training, champion program.- Gradually onboard remaining teams in waves (by risk/criticality).

Metrics (measure weekly/monthly):- Reliability: MTTR for automated tasks, incident count caused by automation- Delivery: deployment frequency using platform vs legacy- Coverage: % of teams using platform, % of critical scripts migrated- Quality: CI pass rates, lint failures, security findings- ROI: time saved per month, duplicated scripts removed

Risk mitigation:- Dual-run mode (platform + legacy) to compare results before cutover- Feature flags and canary rollout for platform-driven actions- Fast rollback paths and runbooks kept in source until stabilization- Backout SLA and monitoring dashboards for automation health

Stakeholder engagement:- Steering committee (SRE lead, product eng managers, security) with weekly syncs- Developer champions in each team to drive adoption- Transparent docs, changelogs, and roadmap; collect feedback via retrospectives and surveys

Success criteria (90 days post-MVP):- Pilot teams migrated, MTTR ≤ previous baseline, deployment frequency increases or unchanged, >40% teams onboarded, reduction in duplicated scripts by 50%.

This balances centralization benefits (consistency, security, reuse) with team autonomy via SDKs, templates, and a contributor-friendly governance model.

EasyTechnical

52 practiced

Implement a Python function 'run_command(cmd: List[str], timeout: int) -> Tuple[int, str, str]' that executes an external command safely using the subprocess module. The function must: enforce the timeout, capture stdout and stderr without deadlocking on large outputs, return the process exit code and captured outputs as strings, and handle KeyboardInterrupt gracefully by terminating the child process. Assume Python 3.8+. Provide the function signature and explain major implementation choices.

Sample Answer

To execute external commands safely while avoiding deadlocks and enforcing timeouts, use subprocess.Popen with communicate() (not shell=True), handle TimeoutExpired to terminate/kill, and catch KeyboardInterrupt to ensure the child is cleaned up.

python

import subprocess
from typing import List, Tuple

def run_command(cmd: List[str], timeout: int) -> Tuple[int, str, str]:
    """
    Execute cmd (list of args), enforce timeout (seconds), capture stdout/stderr.
    Returns (returncode, stdout_str, stderr_str).
    """
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True  # returns str instead of bytes (Python 3.7+)
    )
    try:
        stdout, stderr = proc.communicate(timeout=timeout)
    except subprocess.TimeoutExpired:
        # Try graceful termination first, then force kill
        try:
            proc.terminate()
            stdout, stderr = proc.communicate(timeout=5)
        except subprocess.TimeoutExpired:
            proc.kill()
            stdout, stderr = proc.communicate()
    except KeyboardInterrupt:
        # Ensure child is not left as a zombie if user interrupts
        try:
            proc.terminate()
            proc.wait(timeout=5)
        except subprocess.TimeoutExpired:
            proc.kill()
            proc.wait()
        raise  # re-raise so caller knows we were interrupted
    return proc.returncode, stdout, stderr

Key implementation choices:- Use a list cmd and avoid shell=True to reduce injection risks.- Popen + communicate streams data in a controlled way to avoid deadlocks even for large outputs.- text=True decodes output to str automatically.- On timeout, terminate then kill if needed to avoid orphaned/ hung children.- On KeyboardInterrupt, ensure child termination and re-raise so the caller's interrupt handling works.

Edge cases:- Very large continuous output is handled by pipes + communicate; if streaming processing is required, use threads/asyncio to read incrementally.- If caller needs non-blocking behavior, return proc for external control instead.

EasyTechnical

62 practiced

Design a module and package layout for a reusable Python infrastructure automation library that provides: a core action API (create_vm, provision_network), pluggable cloud provider adapters, utilities for retries and authentication, a CLI entrypoint, and plugin hooks for custom provider code. Sketch directory layout, main modules and responsibilities, and how you would write tests and documentation for contributors.

Sample Answer

Requirements clarified:- Core action API (create_vm, provision_network)- Pluggable cloud providers + plugin hooks- Utilities: retries, auth- CLI entrypoint- Testability and contributor docs

Suggested layout (top-level package name: infra_kit):

infra_kit/ README.md # high-level overview, quickstart pyproject.toml / setup.cfg docs/ # contributor guide, API docs, examples infra_kit/ __init__.py cli.py # click/argparse CLI entrypoint -> calls infra_kit.api api.py # user-facing action API (create_vm, provision_network) core/ manager.py # Orchestrates actions, validates inputs, emits events exceptions.py retry.py # retry decorator/strategy (exponential, jitter) auth.py # Auth provider interface + common implementations hooks.py # Plugin hook registry (pre/post action hooks) providers/ base.py # ProviderAdapter abstract base class (required methods) aws_adapter.py gcp_adapter.py azure_adapter.py discovery.py # dynamic provider loading, entry-point based plugins/ # example plugin implementations (contrib) utils/ logging.py concurrency.py tests/ unit/ integration/ # mark slow; use recorded fixtures (vcrpy/localstack) conftest.py # pytest fixtures for providers, auth mocks

Key module responsibilities:- api.py: thin, stable surface for callers; converts DTOs to core.Manager calls- Manager: handles orchestration, retries, auth, emits hook events- ProviderAdapter: defines create_vm, delete_vm, configure_network, inventory methods- hooks.py: allow registration of lifecycle callbacks and plugin discovery via setuptools entry_points- retry/auth: pluggable strategies that can be swapped via DI

Plugin model:- Providers and plugins register via setuptools entry_points (infra_kit.providers, infra_kit.plugins)- hooks expose: before_action(action_name, context), after_action(...), on_error(...)

Testing strategy:- Unit tests with pytest: mock ProviderAdapter and assert Manager orchestration, retries, and hooks fired- Integration tests: use local emulators (localstack, google cloud emulator) or VCR recordings to avoid live charges; mark as integration- CI: run unit tests on PRs, nightly integration matrix- Coverage: target >85% for core modules

Documentation for contributors:- docs/CONTRIBUTING.md: coding style, testing instructions, releasing, semantic versioning policy for API stability- docs/PROVIDER_GUIDE.md: how to implement ProviderAdapter, required methods, example minimal adapter- docs/HOOKS.md: available hook points and examples- docs/CLI.md and examples/ showing common workflows- API docs generated with Sphinx/pytest-bdd for examples; include a quickstart and troubleshooting section

Design notes / trade-offs:- Keep api.py minimal to preserve backward compatibility- Use entry_points for flexible extensibility; fallback to explicit registration for env-restricted environments- Isolate provider side effects behind adapters to make testing fast and deterministic- Provide sane defaults for retry/auth but allow injection for advanced users

Unlock Full Question Bank

Get access to hundreds of Python Scripting for Infrastructure Automation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.