Python for Infrastructure Automation Questions

Use the Python programming language to build automation for infrastructure tasks. Relevant skills include file input and output, parsing and emitting JSON and YAML configuration files, invoking and managing subprocesses, error handling and retries, working with common HTTP and SSH libraries for automation, and processing and transforming infrastructure data structures. Candidates should be able to demonstrate writing reliable scripts and small programs for provisioning, configuration, orchestration, and integrating with automation tooling and APIs.

EasyTechnical

74 practiced

Describe idempotency in provisioning and configuration automation. Provide a concrete Python example where an operation is made idempotent (for example: user creation or package installation), and explain why idempotency matters for retries and safety in SRE workflows.

Sample Answer

Idempotency means running the same operation multiple times has the same effect as running it once. In provisioning/configuration automation this prevents accidental duplicates, inconsistent state, and makes retries safe.

Example: create a Linux user if it doesn't exist — idempotent because repeated runs leave system unchanged after first success.

python

import subprocess

def user_exists(username):
    return subprocess.call(['id', username], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0

def ensure_user(username, uid=None, shell='/bin/bash'):
    if user_exists(username):
        print(f"{username} already exists")
        return False  # no change
    cmd = ['useradd', '-m', '-s', shell]
    if uid:
        cmd += ['-u', str(uid)]
    cmd.append(username)
    subprocess.check_call(cmd)
    print(f"Created user {username}")
    return True  # changed

# Usage: safe to call repeatedly or after transient failures
ensure_user('deploy')

Why it matters:- Retries: automation can safely re-run after network/lock failures without creating duplicates or corrupting state.- Safety: minimizes blast radius in incidents; operations are predictable and reversible.- Observability & testing: idempotent tasks are easier to validate and reason about in CI/CD and runbooks.

Best practices: check current state, make operations declarative (ensure X), and return whether a change occurred for orchestration tools.

HardTechnical

133 practiced

Compare synchronous and asynchronous HTTP clients in Python with respect to connection pooling, DNS resolution, TLS handshake reuse, latency, and resource footprint. For an SRE task that must poll 100k endpoints every minute, recommend an approach and justify it with trade-offs.

Sample Answer

Brief comparison (sync vs async) across the requested dimensions:

- Connection pooling - Synchronous (requests/urllib3): pooling is per-Client/Session and typically tied to a blocking thread. To scale concurrency you spawn threads/processes; each may hold its own pool, increasing file-descriptor and memory usage. - Asynchronous (aiohttp/httpx async): event-loop–friendly pooled connections reused across coroutines in a single process. HTTP/2 clients (httpx with h2 or hyper) can multiplex many logical requests on few TCP connections, dramatically reducing connections.

- DNS resolution - Sync: uses system resolver (getaddrinfo) which blocks; often fine in small apps but problematic at high scale unless you offload to threaded resolver or local caching DNS. - Async: default resolvers may still block; use async DNS libraries (aiodns) or an external cache (local nscd/Unbound) to avoid blocking the event loop and reduce lookup latency.

- TLS handshake reuse - Sync: keep-alive reuses TCP/TLS sessions per pool; across threads/processes reuse limited unless shared session resumption configured. - Async: same benefits plus better support for TLS session resumption across many coroutines. HTTP/2/3 further reduces handshakes by multiplexing and QUIC (HTTP/3) removes TCP/TLS handshake cost in repeated requests.

- Latency - Sync: per-thread blocking causes higher tail latency when threads are exhausted. Good single-request latency but poor at massive concurrency. - Async: lower end-to-end latency at high concurrency because you avoid thread context-switching and can maintain many in-flight requests; HTTP/2/3 multiplexing reduces connection establishment latency.

- Resource footprint - Sync: high memory/CPU per thread/process, more file descriptors (many TCP connections) for high concurrency. - Async: lower memory/CPU overhead per concurrent request, fewer fds when using multiplexing.

Recommendation for polling 100k endpoints/min (~1,667 requests/sec):

Use an asynchronous, horizontally sharded approach:- Implement an async client (httpx.AsyncClient with HTTP/2 enabled or a specialized HTTP/2 client) running on multiple worker processes (e.g., 8–16 processes per host) using uvloop.- Enable connection pooling and HTTP/2 multiplexing so a single connection can serve many requests to the same host.- Use an async DNS resolver or (preferably) a local DNS cache (Unbound/nscd) to avoid blocking and to short-circuit lookups.- Ensure TLS session resumption and keep-alive are enabled; prefer HTTP/3 (QUIC) where supported for lower handshake cost.- Shard endpoints across multiple hosts (Kubernetes jobs or VM fleet). Estimate capacity: with average request RTT 200ms, one worker with concurrency 500 could handle ~2.5k req/s; pick N workers to meet target with margin.- Add backpressure, concurrency limits per-host, circuit-breakers, and exponential backoff to avoid overload.- Instrument heavily (latency, error rates, connection pool metrics, DNS latencies).

Trade-offs and justification:- Async reduces CPU/memory footprint and fd usage vs spinning thousands of threads/processes; HTTP/2/3 reduces connections and TLS handshakes.- Complexity: async code, DNS/TLS tuning, and HTTP/2 edge cases increase engineering effort and testing.- Operational: need horizontal scaling, careful tuning of concurrency, observability, and resilience (retries, rate-limits).- If endpoints are uniformly across many different hosts (no multiplexing benefit), connection count still large; combine async clients with multiple nodes and aggressive DNS/TLS caching.

Bottom line: choose an async HTTP client with HTTP/2/3 support, DNS caching, TLS session reuse, and horizontally scale processes/nodes. This yields the lowest latency and resource footprint for 100k polls/min while accepting higher implementation complexity.

MediumTechnical

126 practiced

Explain Python's Global Interpreter Lock (GIL). For SRE automation scripts that need concurrency, describe scenarios where threading is sufficient versus when you should use multiprocessing or asyncio. Give a brief example for each scenario.

Sample Answer

The GIL is a CPython mutex that allows only one native thread to execute Python bytecode at a time. It prevents true parallel execution of Python code on multiple CPU cores for CPU-bound work, but it does not block threads during blocking I/O or when native extensions release the GIL.

When to use which model (SRE context):

- Threading — sufficient for I/O-bound automation (SSH, HTTP calls, reading/writing files, waiting on subprocesses). Threads let you overlap blocking operations with minimal complexity.Example: concurrent SSH checks

python

import threading, requests

def check(url):
    r = requests.get(url, timeout=5)
    print(url, r.status_code)

threads = [threading.Thread(target=check, args=(u,)) for u in urls]
for t in threads: t.start()
for t in threads: t.join()

- Multiprocessing — use for CPU-bound tasks (log parsing, large data transforms, heavy regexes, compression) where you need parallel CPU utilization across cores because each process has its own GIL.Example: parallel log parsing

python

from multiprocessing import Pool

def parse(chunk): return heavy_parse(chunk)

with Pool() as p:
    results = p.map(parse, chunks)

- asyncio — ideal for massive numbers of concurrent, high-latency I/O ops (thousands of HTTP checks, long-lived websocket watchers) with low memory and context-switch cost. Use async libraries (aiohttp, asyncssh).Example: many concurrent HTTP checks

python

import asyncio, aiohttp

async def check(url):
    async with aiohttp.ClientSession() as s:
        async with s.get(url) as r:
            print(url, r.status)

asyncio.run(asyncio.gather(*(check(u) for u in urls)))

Trade-offs: threading simpler integration with blocking libraries; multiprocessing has IPC and memory overhead; asyncio requires async-aware libraries and different control flow. Choose based on whether workload is I/O-bound (threading/asyncio) or CPU-bound (multiprocessing) and on scale/complexity constraints.

MediumTechnical

87 practiced

You must handle API tokens and SSH keys in Python automation. Describe a secure strategy to manage credentials for automation scripts used by SREs, covering storage, access control, rotation, and avoidance of leakage (logs, exceptions, repos). Provide concrete practices.

Sample Answer

Start with a principle: never hard-code secrets; enforce least privilege, auditability, rotation, and defense-in-depth.

Storage- Use a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). Back secrets with KMS.- For SSH keys, store private keys in Vault or use an SSH CA/short-lived certs rather than long-lived private keys.

Access control- Use IAM policies or Vault ACLs granting scripts the minimal scopes. Give SRE automation service accounts a unique identity.- Use mTLS, instance profiles, or cloud metadata tokens (with short TTL) to authenticate to the secrets store.

Rotation and short-lived credentials- Prefer ephemeral credentials: Vault dynamic secrets, AWS STS, or signed SSH certificates with minutes-to-hours TTL.- Automate rotation: pipeline job that rotates long-lived secrets, notifies owners, updates consumers, runs smoke tests.

Avoid leakage- Never commit secrets to repos. Add pre-commit/githook checks (git-secrets, pre-commit) and CI scanning.- Filter logs and exceptions: use logging filters/formatters to redact secrets before emission.- Use process-level protections: restrict file permissions (600), run with least-privileged user, use ssh-agent/SSH_AUTH_SOCK for keys rather than raw files.- Avoid printing tokens; if a token must be passed to subprocess, pass via stdin or secure temp file with tight permissions.

Concrete Python practices- Retrieve secrets at runtime from the secrets manager and keep references minimal. Prefer using client libraries with TLS.- Example logging filter to redact obvious tokens:

python

import logging, re
class RedactFilter(logging.Filter):
    TOKEN_RE = re.compile(r'(?i)(token|secret|passwd|key)=\S+')
    def filter(self, record):
        record.msg = self.TOKEN_RE.sub(r'\1=[REDACTED]', str(record.msg))
        return True

logger = logging.getLogger()
logger.addFilter(RedactFilter())

- Use paramiko/asyncssh with keys loaded from ssh-agent or from in-memory bytes guarded by secure memory APIs if available.- In CI/CD, use the platform’s secret injection (not env files checked in). Use ephemeral CI tokens scoped and time-limited.

Operational controls- Enable audit logging on the secrets store; alert on suspicious reads.- Periodically run secrets inventory and rotate or revoke unused credentials.- Document ownership and run tabletop drills for key compromise.

This combination—centralized secrets, short-lived credentials, strict ACLs, logging protections, automated rotation, and developer/CI guardrails—keeps automation secure while remaining operable for SRE workflows.

MediumTechnical

96 practiced

Implement a Python decorator @retry_on(exception_types, max_attempts=3, base_delay=0.2) that retries the decorated function on specified exceptions with exponential backoff. The decorator should work for both synchronous and asynchronous (async def) functions transparently.

Sample Answer

Approach: use a decorator factory that accepts exception types and retry params, detect whether the wrapped function is async (inspect.iscoroutinefunction) and return an async or sync wrapper accordingly. Use exponential backoff (base_delay * 2**(attempt-1)). Use asyncio.sleep in async wrapper and time.sleep in sync wrapper. Preserve metadata with functools.wraps.

python

import asyncio
import functools
import inspect
import time
from typing import Tuple, Type, Union

def retry_on(exception_types: Union[Type[BaseException], Tuple[Type[BaseException], ...]],
             max_attempts: int = 3,
             base_delay: float = 0.2):
    if not isinstance(exception_types, tuple):
        exception_types = (exception_types,)

    def decorator(fn):
        is_async = inspect.iscoroutinefunction(fn)

        @functools.wraps(fn)
        async def async_wrapper(*args, **kwargs):
            attempt = 1
            while True:
                try:
                    return await fn(*args, **kwargs)
                except exception_types as e:
                    if attempt >= max_attempts:
                        raise
                    delay = base_delay * (2 ** (attempt - 1))
                    await asyncio.sleep(delay)
                    attempt += 1

        @functools.wraps(fn)
        def sync_wrapper(*args, **kwargs):
            attempt = 1
            while True:
                try:
                    return fn(*args, **kwargs)
                except exception_types as e:
                    if attempt >= max_attempts:
                        raise
                    delay = base_delay * (2 ** (attempt - 1))
                    time.sleep(delay)
                    attempt += 1

        return async_wrapper if is_async else sync_wrapper

    return decorator

Key points:- Works for both async and sync functions by returning an awaitable wrapper for coroutines.- Exponential backoff: base_delay, then doubled each retry.- Preserves function metadata with wraps.

Time/space:- Time: overhead equals sum of delays plus underlying function runtime; control dominated by retries.- Space: O(1) extra memory.

Edge cases / trade-offs:- Decorating sync functions called inside an asyncio event loop will block the loop because time.sleep is blocking. If needed, prefer async implementations or run sync work in an executor (asyncio.to_thread).- You may add jitter to avoid thundering herd, or accept-on-retry callback/logging for observability.- Consider limiting total backoff or adding timeout.

Unlock Full Question Bank

Get access to hundreds of Python for Infrastructure Automation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.