Automation and Scripting Questions

Covers practical and architectural skills for writing production safe automation and operational scripts as well as building reusable automation systems. Topics include designing idempotent automation, safe retries, robust error handling, structured logging and observability, argument parsing and command line interface design, configuration management, and secure credential handling. Emphasis on testing and validation of scripts and automation code, packaging, documentation, deployment, and maintainability so automation can be operated by other team members. Includes integration with schedulers such as cron and systemd timers, continuous integration and continuous delivery pipelines, orchestration and configuration management systems, and common operational patterns such as log processing, backups, polling, multi step orchestration, provisioning, configuration changes, and routine maintenance. Also assesses language selection and trade offs among Python, Go, Bash and other tooling, concurrency and performance considerations, and at senior levels the design and architecture of reusable automation frameworks and strategies for scaling automation to reduce toil.

MediumTechnical

74 practiced

Implement or outline a reusable retry decorator in Python that supports exponential backoff with jitter, a configurable max attempts, and a predicate callback to classify retryable exceptions. The decorator should be usable on synchronous functions and support logging each attempt. Explain how idempotency assumptions affect your wrapper and where idempotency tokens should be applied when calling external APIs.

Sample Answer

**Approach**- Build a reusable decorator parameterized for max_attempts, base_delay, max_delay, jitter, and predicate to decide retryable exceptions. Log each attempt and backoff. Keep synchronous-only for simplicity.

python

import time, random, functools, logging
logger = logging.getLogger(__name__)

def retryable(max_attempts=5, base_delay=0.5, max_delay=30.0, jitter=0.1, predicate=lambda e: True):
    """
    Retry decorator with exponential backoff + jitter.
    predicate(exception) -> True to retry
    """
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            attempt = 1
            while True:
                try:
                    logger.debug("Attempt %d calling %s", attempt, fn.__name__)
                    return fn(*args, **kwargs)
                except Exception as e:
                    should = predicate(e)
                    logger.warning("Attempt %d failed with %s (retryable=%s)", attempt, e, should)
                    if not should or attempt >= max_attempts:
                        logger.error("Giving up after %d attempts", attempt)
                        raise
                    # exponential backoff with full jitter
                    exp = min(max_delay, base_delay * (2 ** (attempt - 1)))
                    sleep = random.uniform(0, exp) + (random.uniform(-jitter, jitter) * exp)
                    sleep = max(0, min(max_delay, sleep))
                    logger.info("Sleeping %.2fs before retry %d", sleep, attempt + 1)
                    time.sleep(sleep)
                    attempt += 1
        return wrapper
    return decorator

**Key points / reasoning**- Exponential backoff reduces thundering herd; jitter prevents synchronized retries.- Predicate lets caller distinguish transient (HTTP 502/503, network errors) vs permanent (400) failures.- Logging each attempt gives observability in pipelines and alerts.

**Idempotency**- Retries are safe only for idempotent operations. For non-idempotent actions (e.g., charge a card, create unique resource), use idempotency tokens.- Apply idempotency tokens at the API boundary: include a client-generated UUID in request headers/body (e.g., Idempotency-Key) so server deduplicates retries.- In CI/CD/automation, generate tokens per operation (job id + attempt) and persist until final success.

**Usage example**- Decorate deployment-step functions that call cloud APIs; set predicate to retry on network/timeouts and 429/5xx responses, and ensure create calls carry Idempotency-Key.

EasyTechnical

82 practiced

Compare cron with systemd timers for scheduling operational jobs. For a maintenance job that must run hourly and should not overlap if a previous run is still executing, which would you choose and why? Provide an example of a systemd unit/timer configuration (describe the key directives) and explain how logging and output should be handled to integrate with the journal.

Sample Answer

**Compare cron vs systemd timers (short)**

- cron: simple, ubiquitous, fine for many jobs, but limited control (no dependency graph, coarse logging, no native service lifecycle).- systemd timers: tighter integration with unit management, dependencies, resource limits, accurate calendar events, StartLimit, and better observability via journal. For a maintenance job that must not overlap, systemd timers + service unit gives clearer control.

Choice: systemd timers. They integrate with units and journal; use a lock (flock) or systemd-run scope to prevent overlap.

Example unit + timer (key directives explained):

ini

# /etc/systemd/system/my-maintenance.service
[Unit]
Description=Hourly maintenance job

[Service]
Type=oneshot
User=maintenance
WorkingDirectory=/srv/app
ExecStart=/usr/local/bin/my-maintenance.sh
# prevent overlap by using flock inside the script or wrap with:
# ExecStart=/usr/bin/flock -n /var/lock/my-maintenance.lock /usr/local/bin/my-maintenance.sh
StandardOutput=journal
StandardError=journal

ini

# /etc/systemd/system/my-maintenance.timer
[Unit]
Description=Run maintenance hourly

[Timer]
OnCalendar=hourly            # run on the top of every hour
Persistent=true              # catch up missed runs after downtime
AccuracySec=1min             # acceptable timer jitter
Unit=my-maintenance.service

[Install]
WantedBy=timers.target

Why this prevents overlap- Type=oneshot runs and exits; to avoid overlapping runs, use an OS lock (flock) in ExecStart or make the script acquire a lock. Alternatively, use a separate systemd service with `RefuseManualStart` is not for this; flock is simple and reliable.

Logging and output integration- systemd/journald captures stdout/stderr by default; set StandardOutput=journal and StandardError=journal (or forward to syslog) so logs appear in journalctl -u my-maintenance.service.- Emit structured logs (journalctl supports JSON) or use logger(1) if needed.- Include exit codes: systemd records non-zero exit as failure; use Restart=on-failure if you want retries.

Operational notes- Test with systemctl start/stop and journalctl -u my-maintenance.service -f.- Use systemd timers when you need observability, ordering, failure handling, and tight integration with system services.

MediumTechnical

132 practiced

A nightly cleanup automation started failing intermittently. Describe a structured troubleshooting approach to find root cause: what logs and metrics to collect, how to reproduce the issue safely, how to form and test hypotheses, and how to implement and roll out a fix with minimal user impact. Include communication and rollback plans.

Sample Answer

**Situation & goal**I’m a DevOps engineer diagnosing an intermittent nightly cleanup job that started failing; goal is find root cause, fix with minimal user impact, and verify.

**1) Collect logs & metrics**- Job logs (stdout/stderr), scheduler logs (cron/k8s CronJob), app/service logs for resources touched.- System metrics around failures: CPU, memory, disk I/O, disk usage, network, file descriptors, process counts (Prometheus/Grafana).- Orchestration events: Kubernetes events, pod restart counts, node conditions.- Storage/DB metrics: latency, connection errors, locking, txn failures.- Timestamps and correlation IDs; gather several failure and success runs for comparison.

**2) Reproduce safely**- Re-run job in a staging environment with identical inputs, schedule, and resource limits.- Use traffic-shaping or snapshot of production data (anonymized) to avoid affecting users.- Increase logging (debug) and add tracing/span IDs for one controlled run.- If intermittent, run many iterations or inject fault conditions (chaos testing) to amplify.

**3) Form & test hypotheses**- Common hypotheses: resource exhaustion, race condition, transient network/storage errors, timing/dependency change, permission/rotation issues.- For each: design tests (e.g., reduce concurrency, increase resources, run with mock DB vs real DB, simulate network latency).- Use A/B runs: baseline vs change. Capture metrics and logs; confirm hypothesis when failures disappear or specific error appears.

**4) Implement fix & rollout**- Prefer configuration or code fix validated in staging (increase retries, backoff, idempotency, resource requests, fix timeouts).- Use feature flags or canary rollback-capable rollout: deploy to one node/cronJob instance, monitor 2–3 runs.- Gradual roll-out: 10% → 50% → 100% with health checks and SLO/alert watching.

**5) Communication & rollback**- Notify stakeholders (on-call, SRE, product owners) with summary, ETA, and impact.- Update incident channel/status page during rollout.- Rollback plan: revert config or deployment, or pause cronJob; test rollback in staging first.- Post-mortem: document root cause, fix, monitoring added, and preventive actions (alerts, SLA changes, automated remediation).

This approach prioritizes data-driven diagnosis, safe reproduction, hypothesis testing, staged rollout, clear communication, and a tested rollback.

HardSystem Design

81 practiced

Design a distributed job orchestrator capable of running approximately 100k scheduled automation tasks per day across multiple data centers with at-most-once execution semantics, retries, SLA targets, and cross-region failover. Discuss schedule storage, leader election model, deduplication, idempotency strategies, scaling approaches (sharding, worker pools), monitoring, and disaster recovery.

Sample Answer

**Clarify requirements & constraints**- ~100k tasks/day (~1.2/s average, peaks higher), at-most-once delivery, retries with SLA targets, cross‑region failover, multi-DC.

**High-level architecture**- Central scheduler service (control plane) + many workers (data plane) in each region.- Persistent schedule store (primary DB) + distributed task queue (per-shard) + leader election via Raft (etcd/Consul).

**Schedule storage**- Use a strongly consistent metadata DB (e.g., CockroachDB or Cloud Spanner) for canonical schedules, next-run times, owner, idempotency key, retry policy, SLA.- Materialize ready tasks into durable per-shard queues (Kafka/SQS/RabbitMQ) for delivery.

**Leader election & control plane**- Lightweight leaders per logical shard using etcd/consensus. Leaders are responsible for moving due jobs from DB to queue and owning shard leases.- Lease TTLs (short, e.g., 10s) ensure quick failover; leaders renew leases via etcd; if fail, another node acquires.

**Deduplication & at-most-once**- Assign each scheduled run a globally unique run_id and store run state (PENDING, DISPATCHED, COMPLETED, FAILED) in DB.- Use conditional writes (compare-and-set) so only one controller can mark PENDING→DISPATCHED and push to queue.- Workers acknowledge completions with run_id; controller transitions to COMPLETED. If no ack before SLA/retry window, controller may schedule retry per policy.- Use queue semantics: prefer work queues with “exactly-once-ish” or at-least-once but rely on run_id + controller CAS to enforce at-most-once.

**Idempotency strategies**- Encourage idempotent task handlers by requiring idempotency keys and operation tokens; provide helper libs that check and store last-effect timestamp in a durable idempotency store (Redis with persistence or DB table).- For non-idempotent operations, use distributed locks or commit markers (transactional outbox pattern).

**Scaling approaches**- Sharding: partition schedules by hash(schedule_id % N) where N scales; each shard has its leader.- Worker pools: autoscaling groups per region, worker types by concurrency or task type, backed by Kubernetes HPA using queue length/latency metrics.- Horizontal scale: add shards (resharding via rebalancing utility) and increase worker pool size.

**Retries, SLA & backoff**- Store retry policy per job (max_attempts, backoff_exp, deadlines). Controller enforces exponential backoff and total SLA deadline.- SLA monitoring: track start-to-complete latency percentiles; preemptively escalate tasks near SLA to higher-priority worker queue.

**Monitoring & observability**- Metrics: runs/sec, success/fail rates, latency percentiles, in-flight per-shard, lease renewals, queue lag.- Logs: structured logs with run_id, shard_id, region.- Traces: distributed tracing for task lifecycle.- Alerts: SLA breaches, leader flapping, queue backlog growth, consumer lag.- Dashboards per region and global.

**Disaster recovery & cross-region failover**- Active-active where schedule DB is globally consistent (Cloud Spanner/Cockroach) and per-region controllers operate; leaders acquire shard leases per-region but only one leader holds lease globally.- If region fails: lease TTL expiry allows other region to acquire and resume dispatching. Queues are replicated (Kafka mirroring or SQS global queues) or controller can rehydrate queue from DB.- Periodic DR drills and automated failover playbooks; warm standby controllers in secondary regions.

**Trade-offs & considerations**- Strong consistency simplifies at-most-once but increases latency/cost.- If using eventually-consistent DB, implement stronger dedupe logic and longer lease checks.- Complexity vs reliability: use managed consensus/DB services to reduce operational burden.

This design balances at-most-once semantics (via run_id + CAS + leader leases), scalability (shards + autoscaling workers), observability, and cross-region resilience suitable for DevOps-run deployments.

HardSystem Design

78 practiced

Design an orchestration engine for multi-service operations using the Saga (compensating transactions) pattern. Requirements: define step idempotency, compensation actions, durable transaction log, classification of errors (retryable vs irrecoverable), visibility into partial success, and operator controls (pause/resume/compensate manually). Explain how you would test compensation logic reliably.

Sample Answer

**Overview / Goals**Design an orchestration engine that runs multi-service Sagas: durable orchestration, per-step idempotency, explicit compensation, error classification, operator controls, and observability. I'll describe components, data model, flows, and testing.

**High-level architecture**- Orchestrator service (stateless workers) + durable Transaction Log (append-only, e.g., DynamoDB/CockroachDB) + Step Executors (call services via HTTP/gRPC) + Operator API/UI + Event bus (Kafka) + Metrics/Tracing.

**Core data model**- Saga instance: id, status (IN_PROGRESS/PAUSED/COMPENSATING/COMPLETED/FAILED), created_at.- Step record: step_id (unique per saga), sequence, input, output, status, retry_count, compensation_ref, idempotency_key, timestamp.- Audit log for state transitions.

**Idempotency**- Each step executes with an idempotency_key (saga_id + step_id + attempt). Executors must support idempotent semantics (dedupe via key). Orchestrator checks Step record: if status SUCCESS, skip re-execution and replay output.

**Compensation actions**- Every forward step registers a compensation_ref (service endpoint + idempotency semantics). Compensation runs in reverse order for all SUCCESS steps. Compensations are also idempotent and logged as steps with a compensation flag.

**Durable transaction log & replay**- All commands/events persisted before dispatch (write-ahead). On restart, orchestrator replays IN_PROGRESS/PAUSED sagas to resume deterministically.

**Error classification & retries**- Executors return structured error codes: RETRYABLE (transient network/5xx), IRRECOVERABLE (validation/409), MANUAL (business conflict).- Orchestrator policy: exponential backoff with max attempts for RETRYABLE; on IRRECOVERABLE -> transition to COMPENSATING or PAUSED (based on config); MANUAL -> PAUSED and notify operators.

**Visibility & operator controls**- UI/API shows step-level status, logs, traces, and ability to: PAUSE/RESUME, TRIGGER_COMPENSATION (full or partial up to N steps), SKIP step, or REPLAY step. Actions append commands to transaction log and require RBAC + audit.

**Testing compensation reliably**- Use deterministic integration tests with service test doubles that can be scripted to: - Simulate success, transient failures, irrecoverable failures. - Assert log entries, step order, reverse compensation ordering, and idempotency by replaying the same saga multiple times.- Chaos tests in staging: network partitions, delayed responses, duplicate events, process restarts; verify no double-commit and proper compensation.- Property-based tests: for random step graphs, assert invariants (either all forward succeeded or compensations applied for each succeeded forward step).- Contract tests: require each service to expose a test endpoint to verify idempotency and safe replays.- Observability-driven assertions: traces and metrics confirm exactly-once semantics per step idempotency_key and that compensation runs only for previously successful steps.

**Trade-offs**- Centralized orchestrator simplifies logic but is a single control-plane (use HA via leader election). Choreography reduces central state but complicates compensation ordering. Durable log increases latency but guarantees recoverability.

This design treats compensations as first-class, durable, idempotent operations with strong observability and operator controls—suitable for production DevOps automation.

Unlock Full Question Bank

Get access to hundreds of Automation and Scripting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.