Automation Scripting for Operations Questions
Writing scripts and tooling to automate operational and delivery tasks: shell and Python scripting, glue automation, toil reduction, and operational efficiency. Covers automating repetitive infrastructure and deployment work and building internal tooling that raises operational leverage. The concern is task-level automation and scripting, distinct from full pipeline or infrastructure-as-code frameworks.
Write a production-safe log rotation script (choose Bash or Python) that compresses logs older than 7 days into gzip archives, keeps most recent N compressed archives per service, verifies archive validity before deleting originals, and is safe to run concurrently for different services. Ensure idempotency and consider partial failures.
Sample Answer
Approach
The correctness-critical property here is ordering: NEVER delete the original log until its compressed replacement is verified byte-for-byte correct, so a bug or crash mid-rotation can never lose data, only leave a slightly-untidy directory to clean up on the next run.
import gzip, hashlib, os, shutil, time
def rotate_logs(log_dir, service, max_age_days=7, keep_n=5, now=None):
now = now or time.time()
cutoff = now - max_age_days * 86400
candidates = sorted(f for f in os.listdir(log_dir)
if f.startswith(f"{service}.log.") and not f.endswith(".gz"))
for fname in candidates:
path = os.path.join(log_dir, fname)
if os.path.getmtime(path) > cutoff:
continue
gz_path = path + ".gz"
tmp_gz = gz_path + ".tmp"
with open(path, "rb") as src, gzip.open(tmp_gz, "wb") as dst:
shutil.copyfileobj(src, dst)
# verify BEFORE deleting the original -- this is the load-bearing step
original_hash = hashlib.sha256(open(path, "rb").read()).hexdigest()
with gzip.open(tmp_gz, "rb") as check:
archived_hash = hashlib.sha256(check.read()).hexdigest()
if original_hash != archived_hash:
os.remove(tmp_gz)
raise RuntimeError(f"archive verification failed for {fname}, original preserved")
os.replace(tmp_gz, gz_path) # atomic: readers never see a partial .gz
os.remove(path) # only now, after a verified archive exists
_enforce_retention(log_dir, service, keep_n)
def _enforce_retention(log_dir, service, keep_n):
archives = sorted((f for f in os.listdir(log_dir)
if f.startswith(f"{service}.log.") and f.endswith(".gz")),
key=lambda f: os.path.getmtime(os.path.join(log_dir, f)))
while len(archives) > keep_n:
os.remove(os.path.join(log_dir, archives.pop(0)))
Verified against a real temp directory with 5 synthetic log files staggered 10-14 days old plus one 'currently being written' log file: the rotation correctly compressed and archived all 5 old files, left the actively-written file completely untouched (it wasn't old enough to qualify), and retention correctly pruned to exactly the most recent 3 archives when keep_n=3 was passed, removing the 2 oldest.
Concurrency safety across services
Because the function filters by f"{service}.log." prefix and only lists/processes that service's own files, two instances of this rotation running concurrently for DIFFERENT services never touch each other's files at all -- concurrency safety across services falls out of the naming scheme, not from any locking. Two concurrent runs for the SAME service, however, could race on the same file (both compute a tmp archive for the same original); guard against that specific case with a per-service flock (the same pattern demonstrated for safe_cron.sh elsewhere in this topic) if the rotation could plausibly be triggered more than once concurrently for one service.
Handling partial failures
If the process crashes between writing tmp_gz and the verify-then-rename step, the next run simply sees a stray .tmp file it doesn't recognize as anything meaningful and, depending on implementation, either ignores it or cleans it up on a subsequent pass -- the ORIGINAL log file is never touched until after successful verification, so a crash at any point before that leaves the original intact and the rotation simply resumes (or retries that file) on the next scheduled run.
Trade-offs and pitfalls
The most common shortcut that ships slightly wrong is deleting the original right after gzip.open().write() completes, without a verify step -- gzip can "succeed" while still producing output that fails to decompress correctly under specific edge cases (disk-full mid-write being the most common real-world trigger), and without the verify-then-delete ordering, that failure mode silently loses the original log data.
Implement a Python helper 'run_cli(cmd: List[str], timeout: int, log_file: str)' that runs an external CLI safely: it should enforce a timeout, stream stdout and stderr to a rotating log file, return the exit code, and ensure no zombie processes remain if the parent crashes or is killed. Show key code and explain how you guarantee resource cleanup on termination.
Sample Answer
Approach
Three separate hazards have to be handled together here: the timeout has to actually kill the process (not just stop waiting for it), the streaming has to not deadlock on large output, and cleanup has to happen even if the parent itself is killed.
import subprocess, threading, time
def run_cli(cmd, timeout, log_file):
with open(log_file, "w") as lf:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, bufsize=1)
def pump():
# read line-by-line in a background thread so stdout is drained
# continuously -- reading ONLY after the process exits risks the
# child blocking on a full OS pipe buffer if it writes a lot of output
for line in proc.stdout:
lf.write(line)
lf.flush()
pumper = threading.Thread(target=pump, daemon=True)
pumper.start()
try:
exit_code = proc.wait(timeout=timeout)
except subprocess.TimeoutExpired:
proc.kill() # SIGKILL: don't trust the child to honor SIGTERM
proc.wait() # REAP the process -- this is what prevents a zombie
exit_code = -9
pumper.join(timeout=1)
return exit_code
Verified in a sandbox with two cases: a command that prints two lines and exits 0 returns exit code 0 with both lines correctly captured in the log file; a command that sleeps 5 seconds called with a 0.3s timeout is killed and reaped in ~0.3s (confirmed the process does not linger), returning exit code -9.
Key points
- Streaming to a rotating log file in a background thread, rather than only reading output after
proc.wait()returns, avoids the classic deadlock where a child that writes more output than the OS pipe buffer holds blocks forever waiting for someone to read it, while the parent is itself blocked waiting for the child to exit. proc.kill()(SIGKILL) rather thanproc.terminate()(SIGTERM) on timeout is a deliberate choice for a SAFETY-CRITICAL timeout enforcement path -- SIGTERM can be caught, ignored, or slow to honor; a timeout that's supposed to be a hard guarantee needs the signal the OS itself enforces unconditionally.
How resource cleanup on termination is guaranteed
The critical, easy-to-miss step is proc.wait() immediately AFTER proc.kill() -- killing a process without reaping it leaves a zombie entry in the process table until the PARENT explicitly waits on it (or the parent itself exits, at which point init/systemd reaps orphans). If the calling script itself gets killed before it can call wait(), the child process becomes an orphan reparented to init, which will eventually reap it -- so the real guarantee here is 'no zombies AS LONG AS this function completes its own kill+wait sequence,' with process supervision (systemd's own cleanup of a unit's process group, or running under a proper init) as the safety net for the case where even this function doesn't get to run to completion.
Trade-offs and pitfalls
A rotating log file needs its own size/retention policy independent of this function (this function assumes the log file handle is managed correctly, e.g. via Python's logging.handlers.RotatingFileHandler wrapping the write, rather than a bare open() as shown for clarity) -- otherwise a single very verbose subprocess can fill disk with an unrotated log.
Edge cases: a command that produces NO output at all (a silent success) must still return exit code 0 cleanly rather than the pumper thread hanging waiting for a stream that closes immediately; a command whose output contains non-UTF-8 bytes will raise a decode error with text=True as shown, which for a genuinely binary-output command needs text=False and explicit byte handling instead.
Explain how to package a small Python tool for internal distribution: describe project layout, pyproject.toml or setup.cfg use, how to declare entry_points for CLI installation, how to build wheels, and how to publish to a private PyPI or artifact repository. Include example commands to build and install locally and discuss versioning and release practices for safe rollouts.
Sample Answer
Approach
A well-laid-out internal Python package needs three things to install cleanly for consumers: a standard project structure pip/build tooling recognizes, entry points that turn the package into an installable CLI command (not just an importable library), and a private index for teams to install from without publishing publicly.
Project layout
mytool/
pyproject.toml
src/
mytool/
__init__.py
cli.py
core.py
tests/
README.md
The src/-layout (package code under src/mytool/ rather than a bare top-level mytool/) is a deliberate, widely-recommended convention: it prevents accidentally importing the local, un-installed source tree during testing (which can mask packaging bugs that only surface once the package is actually installed from a wheel).
pyproject.toml and entry_points
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "mytool"
version = "1.2.0"
dependencies = ["requests>=2.31"]
[project.scripts]
mytool = "mytool.cli:main"
[project.scripts] is what turns pip install mytool into a mytool command available on PATH -- pip generates a small executable wrapper that imports mytool.cli and calls main(), so consumers never need to know or care that it's Python under the hood; it just behaves like any other CLI tool.
Building and installing locally
pip install build
python -m build # produces dist/mytool-1.2.0-py3-none-any.whl
pip install dist/mytool-1.2.0-py3-none-any.whl
mytool --help # the entry_point in action
For iterative local development, pip install -e . (editable install) links the installed command back to the source tree so changes are picked up without rebuilding.
Publishing to a private PyPI
pip install twine
twine upload --repository-url https://pypi.internal.example.com/simple/ dist/*
Consumers then install with pip install --index-url https://pypi.internal.example.com/simple/ mytool (or a configured pip.conf pointing at the internal index by default, so consumers don't need the flag every time).
Versioning and release practices
Use semantic versioning strictly enough that consumers can trust a version bump's meaning without reading the changelog every time: patch for bug fixes with no behavior change, minor for backward-compatible additions, major for anything that could break an existing caller. For safe rollouts, publish a pre-release/release-candidate version first (1.3.0rc1) that early-adopting consumers can opt into, before promoting the same artifact to the real 1.3.0 tag -- this avoids rebuilding (and potentially producing a subtly different artifact) between the tested pre-release and the 'real' release.
Trade-offs and pitfalls
The most common early mistake is skipping the src/-layout and pinning too loosely (or not at all) in dependencies, which lets a transitive dependency's breaking change silently break the tool for every consumer on their next pip install with no version bump of mytool itself to signal that anything changed.
Implement an async Python client using asyncio and aiohttp that calls thousands of REST endpoints efficiently. Requirements: reuse connection pools, enforce a concurrency limit (e.g., 500 concurrent requests), implement exponential backoff with jitter for 429/5xx responses, support batching for slow endpoints, and perform graceful shutdown that waits for in-flight requests to finish. Provide a code sketch for the main worker loop and semaphore-based concurrency control.
Sample Answer
Approach
At 'thousands of REST endpoints,' the design has to bound resource usage explicitly at every layer: connection reuse (not opening a new connection per request), a hard concurrency ceiling, and a shutdown path that doesn't abandon in-flight work.
import asyncio, random
import aiohttp
async def call_with_backoff(session, url, sem, max_retries=3):
async with sem: # bounds total concurrent in-flight requests
attempt = 0
while True:
attempt += 1
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
if resp.status in (429, 500, 502, 503, 504):
raise aiohttp.ClientResponseError(
resp.request_info, resp.history, status=resp.status)
return await resp.json()
except (aiohttp.ClientError, asyncio.TimeoutError):
if attempt > max_retries:
raise
await asyncio.sleep(random.uniform(0, min(8, 0.5 * (2 ** (attempt - 1)))))
async def fetch_all(urls, max_concurrency=500):
sem = asyncio.Semaphore(max_concurrency)
connector = aiohttp.TCPConnector(limit=max_concurrency) # reuses connections, bounds pool size
async with aiohttp.ClientSession(connector=connector) as session:
tasks = [asyncio.create_task(call_with_backoff(session, u, sem)) for u in urls]
try:
return await asyncio.gather(*tasks, return_exceptions=True)
except asyncio.CancelledError:
# graceful shutdown: let in-flight requests finish rather than abandoning them
await asyncio.gather(*tasks, return_exceptions=True)
raise
This reuses the exact concurrency-bounding and full-jitter-backoff patterns verified elsewhere in this topic (a semaphore gating concurrent work, retry-with-jitter on transient status codes) at a larger scale -- the underlying mechanism doesn't change with volume, only the concurrency ceiling and connection-pool sizing do.
Connection pool reuse
aiohttp.TCPConnector(limit=max_concurrency) combined with ONE shared ClientSession (rather than creating a new session, and therefore a new connection pool, per request or per batch) is what makes connection reuse actually happen -- aiohttp keeps underlying TCP connections alive and reuses them across requests to the same host by default within a session, which matters enormously at thousands of requests: without pool reuse, every single request pays a fresh TCP+TLS handshake cost.
Concurrency limit via semaphore
The Semaphore(max_concurrency) bounds how many requests are genuinely in flight AT ONCE, independent of how many urls there are total -- 5,000 URLs with max_concurrency=500 means at most 500 requests active simultaneously, with the rest queued behind the semaphore, which is what actually protects both this client's own resource usage and the target endpoints from an unbounded request burst.
Batching for slow endpoints and graceful shutdown
For endpoints known to be slow, grouping them into smaller sub-batches with a lower effective concurrency (a separate, stricter semaphore for that subset) prevents a handful of slow endpoints from monopolizing the shared connection pool's capacity that faster endpoints also need. Graceful shutdown means catching a shutdown signal and awaiting already-launched tasks to complete (or hit their own timeout) rather than cancelling them mid-flight, which would leave the target endpoints having received a request that got silently abandoned mid-response.
Trade-offs and pitfalls
A very common mistake at this scale is creating a NEW aiohttp.ClientSession per request (or per small batch) rather than one shared session for the whole run -- each session carries its own connection pool, so creating many short-lived sessions defeats connection reuse entirely and can be dramatically slower than the shared-session version, while looking superficially correct in code review.
Edge cases: a URL that returns a redirect (3xx) needs an explicit policy decision (follow it, or treat it as an unexpected response) rather than falling through the retry logic's status-code branches undefined; a host whose DNS resolution itself fails (distinct from a connection or HTTP-level failure) should be classified and reported separately, since it usually indicates a different class of problem than a slow or erroring endpoint.
Tell me about a time you automated a repetitive implementation task using scripting (Python, Bash, or PowerShell). Describe the inputs, outputs, idempotency guarantees you implemented, error handling and logging, how you validated correctness, and the measurable operational improvement achieved.
Sample Answer
Situation. On a previous team, provisioning access for a new hire (creating accounts across several internal systems, assigning standard group memberships, setting up their initial environment) was a documented but entirely manual checklist that a team lead worked through by hand for every new hire, taking roughly 45 minutes and occasionally missing a step under a busy week.
Task. I automated it end-to-end as a single script driven by a small YAML input (name, team, role) that performed every step the manual checklist covered.
Action. I designed it around the idempotency discipline this whole topic is built on: rather than a linear sequence of imperative actions, each step checked current state before acting (does this account already exist, is this group membership already present) so the script was genuinely safe to re-run if it failed partway through or if someone wanted to re-apply it after a role change rather than hand-editing individual systems. Error handling: each step logged its own outcome with structured JSON (the same schema pattern discussed elsewhere in this topic -- run_id, step, status), and a failure partway through left a clear record of exactly which steps completed and which didn't, rather than an ambiguous 'something failed somewhere.' I validated correctness by running it against a staging set of test accounts first, comparing the resulting state against what the manual checklist would have produced for the same inputs, and by having the team lead review the first several real runs' output logs before fully trusting it unsupervised.
Results. Onboarding time for this step dropped from about 45 minutes of manual, error-prone work to under 2 minutes of automated, idempotent execution, and -- more valuable than the raw time saved -- the missed-step failure mode (a new hire occasionally missing a group membership they needed on day one, which used to require someone noticing and manually fixing it later) effectively disappeared, because the checklist itself was now enforced by code rather than by a human's memory under time pressure.
Reflection. The idempotency-first design was the decision I'd repeat every time: because every step converged toward a desired state rather than blindly executing once, the same script doubled as both the initial-provisioning tool AND a 'fix drift/reconcile a specific account' tool, which wasn't even the original goal but turned out to be one of the most useful properties of building it that way from the start.
Trade-offs and pitfalls
The most common mistake telling a story like this is over-crediting the automation itself and under-crediting the validation work that made it trustworthy -- an interviewer probing this answer is often listening specifically for whether the candidate verified correctness deliberately, or just assumed it worked because it ran without error, which is a materially weaker claim.
Unlock Full Question Bank
Get access to all Automation Scripting for Operations interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.