Senior Systems Administrator Interview Preparation Guide - Doordash

Runbooks rot because nothing forces them to change when the system they describe changes. Keeping them accurate means giving every runbook a named owner, tying updates to the events that actually invalidate a runbook (a relevant code or infra change, or an incident where it was used), and having a lightweight, recurring check that catches staleness before an incident does, rather than relying on someone remembering to update it.

Ownership

Every runbook has one named primary owner, the team or person who owns the service it covers, recorded in the runbook itself, not a separate spreadsheet that gets forgotten. Ownership isn't honorary: the owner is the one who signs off that the runbook is still accurate at each review point, and the one paged if a stale runbook causes a bad outcome during an incident.

What triggers a review, not just a calendar date

Calendar-only review cadences ("review every quarter") catch some staleness but miss the more common case: a runbook goes stale the day the system changes underneath it, not on a schedule. Two triggers matter more than the calendar:

• Change-linked: any deploy that touches the commands, infra, or thresholds a runbook references should require a runbook update as part of that same change, enforced by a checklist item on the pull request, not a follow-up ticket that competes with the next sprint.
• Incident-linked: every time a runbook is actually used during an incident, the post-incident review includes a specific question, did the runbook match reality, and any gap becomes a tracked follow-up before the incident is closed.

Lifecycle stages

Catching a stale runbook before someone relies on it

A stale runbook is genuinely dangerous exactly because it looks trustworthy right up until the moment it's wrong. The cheapest catch is a light automated check, does every command in the runbook reference a tool, dashboard, or endpoint that still exists, run periodically and flagging anything broken for owner review. The more valuable catch is a periodic game-day: pick a runbook, have someone unfamiliar with the system try to follow it against a staging environment, and see where it breaks. Anything that trips someone up in a drill would have tripped up the on-call engineer at 3am.

Worked example: what enforcement actually looks like

A pull request changes the retry/backoff config for a service. The PR template includes a checkbox, "does this change any runbook referenced by this service's on-call docs?" Because the change alters a value a runbook's remediation step depends on, the author checks yes and links the runbook update in the same PR. Two months later, that runbook gets used during an incident; the post-incident review confirms the values matched, so the runbook stays Active with its last-reviewed date updated, no separate ticket needed because the change-linked trigger already did the work.

Trade-offs and pitfalls

Tying every infra change to a mandatory runbook update adds friction to routine PRs, so the check needs to be scoped narrowly (does this specific change affect a specific runbook) rather than a blanket "update all docs" gate that people learn to click through without reading. The most common failure mode is having a lifecycle model on paper but no one actually enforcing the Needs review to Active transition, so runbooks accumulate silently in Needs review and the label stops meaning anything; the fix is making that queue visible (a dashboard, not a buried label) and reviewing it in the same recurring meeting as on-call handoffs.

Direct answer
Code review becomes a coaching tool the moment you separate what has to change before this merges from what's worth teaching, and handle each differently, since blocking mixes poorly with explaining. What counts as the important risk to teach toward also shifts by what's being reviewed: correctness and style for typical application code, reproducibility and data leakage for ML work, and blast radius for infrastructure changes.

Separate blocking feedback from teaching feedback

• Mark comments explicitly as blocking versus non-blocking (or use a similar convention), so the author isn't left guessing what actually has to change before merge. Teaching comments that aren't required for merge belong in the non-blocking bucket, otherwise you either water down real teaching moments to keep the change unblocked, or block a mergeable change to make a point.
• Ask before you tell: a comment phrased as a question ("what happens if this list is empty?") invites the author to find the issue themselves, which teaches the underlying reasoning; a comment phrased as an instruction just transmits the fix.

What "the important risk" means shifts by artifact type

• Typical application code: the coaching focus is usually correctness, readability, and test coverage; the failure mode being taught against is a defect shipping or the next person not being able to follow the change.
• ML notebooks and experiment configs: the review risk is different in kind, not just degree. The critical things to check and teach toward are reproducibility (is the seed pinned, is the environment specified, can someone else get the same result) and data leakage (does the training data have any path back to the evaluation set, directly or through a shared preprocessing step). A notebook can be clean, readable code and still be dangerously wrong for reasons that have nothing to do with code style.
• Terraform and other infrastructure-as-code changes: the review risk is blast radius, not defects in the traditional sense. A small, correct-looking diff can still be catastrophic if it touches a shared resource or removes a safeguard. Coaching here means teaching someone to ask what does this affect beyond what's in the diff before asking is this line correct.

Making it a genuine teaching moment, not just a gate

• When there's something worth teaching, don't just fix it in the comment; explain the why, and where useful, point to a real example elsewhere in the codebase rather than a generic principle.
• For anything too deep to unpack asynchronously in a comment thread, offer a short pairing session instead of a long comment chain; some things teach faster live than in writing.
• Close the loop: after a pattern comes up more than once for the same person, raise it directly in a 1:1 rather than only ever surfacing it inside individual review threads, so it becomes a recognized growth area instead of a recurring surprise.

Worked example
Reviewing a teammate's change that added a new model training script, the code itself was clean and well-tested in the conventional sense. The actual coaching moment was elsewhere: the evaluation split was built after a preprocessing step that had already seen the full dataset, which meant the reported accuracy was optimistic in a way unit tests would never catch. Rather than just fixing the split order and moving on, the comment walked through why that ordering matters (what leakage actually does to the reported number) and pointed to another script in the repo where the split happened correctly, before the shared preprocessing step. That change did get blocked, since the leakage was a real correctness issue, but the teaching part was the explanation of why, not the fact that it was blocked.

Trade-offs and pitfalls

• Making every comment a teaching moment, including on merge-blocking issues, slows delivery and can read as review turning into a lecture; save the deeper explanations for the genuinely worthwhile ones and keep routine fixes routine.
• Applying the same review lens (say, defect-finding) to every artifact type misses the risks that matter most for that artifact; a Terraform change reviewed like application code will pass style and correctness checks while missing blast radius entirely.
• If teaching moments only ever show up as isolated review comments and never get named directly to the person as a pattern, growth stays implicit and slower than it needs to be.

Brief thesis
Centralized config management enforces uniform security baselines quickly and audibly; decentralized gives local teams autonomy and resilience. For a Systems Administrator role, weigh governance, failure modes, attack surface, scale, perf, and ops autonomy when choosing or designing a hybrid.

Compare along key dimensions

• Governance: Centralized — single policy source, easier audits and compliance. Decentralized — inconsistent baselines, requires federated policy agreements.
• Single point of failure: Centralized — controller outage impacts many hosts (SCCM/Puppet master risk). Decentralized — failures are localized.
• Attack surface: Centralized — large-value target (API, credential store); easier to harden. Decentralized — many management endpoints increase lateral risk.
• Scalability & performance: Centralized — can bottleneck at push time; needs horizontal controllers, caching CDNs. Decentralized — scales naturally, lower latency for local changes.
• Operational autonomy: Centralized — limited local flexibility; slower change for site-specific needs. Decentralized — fast local fixes, but risk divergence.

Concrete hybrid model (practical for sysadmin)

• Global “guardrail” layer: policy-as-code (e.g., OPA/Rego, CIS profiles) stored in central repo, enforced by automated checks and immutable rules (deny lists, minimal SSH config).
• Local configuration layer: per-site Ansible roles or Puppet environments allowed to extend only whitelisted settings.
• Enforcement architecture: central policy server + distributed agents that cache policies and fall back to last-known-good if central offline. Use signed artifacts, mutual TLS, and short-lived certs.
• Governance & operations: RBAC — central security team controls guardrails; local admins have scoped write access. Continuous compliance via periodic scans, drift detection, and automated remediation tickets.
• Example: central Git repo with protected branches + CI that produces signed policy bundles; edge nodes pull bundles and apply locally.

Criteria to choose

• Highly regulated orgs / small infra → Centralized
• Geographically dispersed, unreliable links, or many autonomous teams → Decentralized or hybrid
• Large scale with need for rapid local changes → Hybrid preferred
• Threat model favors hardening a single surface and strict compliance → Centralized

Final note
As a sysadmin, prioritize signed policy artifacts, agent caching/fallback, RBAC, audit logs, and measurable SLAs for policy distribution to balance security and uptime.

Direct answer
Culture fit asks whether you already share a team's existing norms and behaviors; culture add asks what you would bring that the team does not already have. I would describe myself mostly as a culture add: I share the fundamentals a team needs to trust me (reliability, candor, respect for other people's time), but the useful thing I offer beyond that is a genuinely different working background rather than a mirror of the team that is already there.

Structured elaboration

1. Define both terms precisely before answering for yourself. Culture fit is about alignment on shared behaviors and values: does this person operate the way we already operate. Culture add is about complementary difference: does this person's background, working style, or perspective fill a gap the team doesn't currently have.
2. Explain why the distinction matters, not just define it. A team optimized purely for fit tends toward groupthink: everyone reasons the same way, so blind spots go unchallenged and the same kinds of mistakes recur. A team that only adds without any shared fit becomes uncoordinated: people can't predict each other's reasoning enough to move fast together. The healthy target is fit on a small number of load-bearing behaviors (honesty, follow-through, respect) plus deliberate add on everything else.
3. Give a genuine, specific example of your own add, not a generic trait. Vague claims ("I bring diverse perspectives") are the single most common failure mode here; a strong answer names the concrete gap and the concrete evidence.
4. Anticipate the natural follow-up: how do you know your difference is actually useful, versus just different for its own sake. The answer is to point at a specific decision, disagreement, or piece of feedback that changed because of the difference you brought, not just a credential or background fact.

Worked example
Suppose your last two teams were both product engineering teams building consumer-facing features, and the team you're interviewing for is mostly staffed by engineers with that same background. Your own prior role was on a data-platform team, closer to the systems that feed those consumer features than to the features themselves. A concrete add-story: in a past project, a product team wanted to ship a new recommendation feature quickly; because of your platform background, you asked a question the rest of the team hadn't raised (whether the upstream data pipeline's freshness guarantees actually matched what the feature's UI implied to users), which surfaced a real gap between a 24-hour batch refresh and a UI copy that said "updated just for you." The team fixed the copy and adjusted the refresh cadence before launch rather than after a user complaint. That is a genuine add: a different background produced a question the existing team composition was less likely to ask on its own, and it changed a real outcome.

Trade-offs & pitfalls
The common failure is answering only the definitional half (correctly explaining fit versus add) and then, when asked for a personal example, retreating to generic self-description ("I'm a good communicator", "I care about quality") that any candidate could say and that does not actually demonstrate difference. A second pitfall is overcorrecting into implying you don't fit at all; the strongest answers are explicit that you also share the small set of behaviors every functioning team needs, and that add is about everything on top of that baseline, not a replacement for it.

I would split the solution by responsibility, not by environment. A module should do one job well.

Module layout

• network: VPC, subnets, routes, NAT, and network tags
• compute: app instances, ECS or ASG, load balancer, security groups
• database: managed DB, subnet group, parameter group, DB security group
• root stack: wires the outputs together

Why this works
The network changes slowly, compute changes often, and the database has its own lifecycle and risk. Keeping them separate reduces blast radius and makes reviews easier.

Safe interface
I would expose only what callers need:

• From network: vpc_id, private_subnet_ids, public_subnet_ids
• From compute: alb_dns_name, app_sg_id
• From database: endpoint, port, and maybe a secret reference, not a password

Example
The root module can pass private_subnet_ids = ["subnet-101", "subnet-202"] into compute and database, while dev and prod use different sizes through variables. That keeps composition flexible without letting one team edit module internals.

Approach (summary)
I'll use a mix of target-tracking for CPU and step-scaling for queue_length spikes, with composite rules to avoid underscaling when CPU is low but queue grows.

Policy types

• Target-tracking: keep average CPU ~55% across ASG — handles gradual load changes and daily predictable peaks.
• Step-scaling (queue_length): react to sudden backlog increases using steps (e.g., +1, +3, +5 instances) based on queue size thresholds.

Thresholds & cooldowns

• CPU target: 55% average, cooldown 300s (to allow metrics to stabilize).
• Queue steps (example per-instance capacity = 100 msgs):
  • queue_length > 200 for 60s → +1 instance, cooldown 180s

  • 500 for 30s → +3 instances, cooldown 300s

  • 1000 for 30s → +5 instances, cooldown 600s

• Scale-in guard: require sustained low usage (CPU < 30% AND queue_length < 50) for 10 minutes before scale-in.

Combining metrics

• Use composite alarm / higher-of logic: evaluate both metrics and apply the larger required change. Prefer "max(desired_by_CPU, desired_by_queue)" so a low CPU doesn't prevent scaling when queue backs up.
• Alternatively, weighted metric: composite metric = alphanormalized_queue + betaCPU, tune alpha to prioritize queue.

Evolving policies

• Monitor growth weekly; every 2–4 weeks re-evaluate per-instance throughput and adjust queue thresholds upward as capacity improves.
• Add predictive scaling for predictable daily peaks (scheduled scaling) and increase min-capacity by 10% per sustained weekly growth checkpoint.
• Automate alerts when step actions hit upper bounds to plan capacity changes (bigger instance types, autoscaling limits).
• Run load tests monthly to recalibrate per-instance capacity and cooldowns.

Direct answer

Rank by a deliberate criterion, not gut feel. Name the criterion you're using, which skills unlock the most future scope, which have the highest impact weighed against feasibility, or which close the gap between your current level and the next one, and include both technical and non-technical or soft skills rather than defaulting to an all-technical list, since most next-level gaps involve at least one of each.

Structured elaboration

Pick and state your ranking criterion explicitly before naming the skills, since the same three skills can be justified very differently depending on whether you're optimizing for near-term impact, long-term career growth potential, or impact weighed against feasibility. Naming the criterion is itself part of a strong answer.

Include at least one non-technical or soft skill alongside technical ones. An all-technical list usually signals either an early-career stage where that's genuinely the right focus, or a blind spot at a more senior stage, where communication, prioritization, or influence often matter more than additional technical depth.

Give each skill a brief, honest reason it made the cut over an alternative you considered and rejected. A ranked list without visible trade-offs reads as a wish list, naming what you left off, and why, shows the ranking was real.

Tie each skill back to a concrete situation where its absence cost you something or its presence would have helped, rather than justifying it in the abstract.

Worked example

"Using what most limits my scope right now as my ranking criterion, my list was: first, a specific domain depth I'm missing that currently forces me to hand off certain problems to someone else, second, clearer stakeholder communication, because I've noticed my updates sometimes need a follow-up conversation to clarify what I actually meant, third, prioritization under competing demands, since I've occasionally said yes to too much and delivered several things late rather than a few things well. I considered adding a fourth, negotiation, but ranked it below the other three because I have fewer real situations right now where it's the binding constraint, so investing there first would be lower leverage. If I ranked by long-term career growth potential instead of near-term scope, prioritization and communication would likely move above the technical depth item, since those compound more as scope grows."

Trade-offs & pitfalls

• A list with no explicit ranking criterion invites the interviewer to wonder whether it was thought through or assembled on the spot.
• An all-technical list at a more senior stage often signals a blind spot, since interpersonal and organizational skills tend to become the actual constraint past a certain level.
• An all-soft-skills list with no technical or domain component can read as avoiding the harder, more measurable half of growth.
• Naming skills with no honest reason for the ranking, or no example of where the gap actually showed up, turns a specific answer into a generic one that could apply to almost anyone.

Direct answer

Start with the component diagram. It establishes what pieces exist and who owns each one, before anything about behavior or payloads makes sense; architects can't reason about "what happens when" until they know "what's here."

Structured elaboration

1. Component diagram (what exists). Purpose: boundaries and ownership. Show: application services, cache cluster nodes, the source-of-truth database, an invalidation service, and a message broker. Leave off: exact protocol, message schema, and timing, those belong later.

2. Sequence diagram (what happens, in order). Purpose: the actual interaction for one invalidation event. Show: a write to the database, the database acknowledging it, an event published to the invalidation service, that service publishing an evict message on the broker, the broker fanning out to cache nodes, and one failure path (broker unavailable: what serves stale data, and for how long). Leave off: byte-level payload detail and retention settings, that's the next diagram's job.

3. Data-flow diagram (what exactly, and how stale). Purpose: payloads and guarantees. Show: the invalidation message's schema (key, version, timestamp), time-to-live, message size, and the one metric architects will actually watch, invalidation latency or staleness window. Leave off: anything already covered by the component-level framing.

Why this order helps comprehension: each diagram answers the question the previous one raised. Component diagram: "what is the invalidation service." Sequence diagram: "how does it know to fire." Data-flow diagram: "how stale can a read get before this evicts it." Reversing the order, starting with the sequence diagram, forces you to define every box mid-sentence instead of pointing at one the audience has already seen.

Worked example

The component diagram you'd draw first:

flowchart LR
    App[Application] -->|write| DB[(Database)]
    App -->|read| Cache[(Cache Cluster)]
    DB -->|change event| Invalidator[Invalidation Service]
    Invalidator -->|publish evict msg| Broker[[Message Broker]]
    Broker -->|fan out| Cache
    Cache -->|miss, reload| DB

Narrated: "The application writes to the database. That write triggers a change event to the invalidation service, which publishes an evict message on the broker. The broker fans that message out to every cache node, and the next read that misses reloads from the database."

Translating the core idea for the architects: the jargon term is "cache coherence." Plain version: "keeping the cache from serving an answer that's gone stale since the database changed." Analogy: it's like a library's card catalog. When a book gets re-shelved, someone has to walk over and update the card, or the next person who checks the card gets sent to the wrong shelf. Where the analogy breaks: no single librarian updates every card at once across a building, the fan-out to many cache nodes in parallel, possibly across regions, is exactly what makes this hard in practice, and that's the detail worth naming once the audience has the basic picture.

Trade-offs & pitfalls

The common wrong turn is leading with the sequence diagram because it feels more "technical," which forces you to define the invalidation service, the broker, and the cache cluster mid-sentence instead of pointing at boxes the audience already recognizes. A second pitfall: putting the failure path (broker down) in the component diagram instead of the sequence diagram, error paths are behavior over time and belong where the audience is already reasoning about timing. A third: overloading the data-flow diagram with architectural detail that duplicates the first diagram instead of adding new information (payload size, TTL, staleness), which makes the customer conversation feel repetitive rather than cumulative.

Immediate goals: preserve evidence, gather diagnostic data, minimize downtime (failover/rollback), avoid further writes to damaged system.

Initial triage (both OSes)

• Verify reachability and alerts, note time of crash and affected services.
• Put host into maintenance mode (disable monitoring/automated remediation) and if possible shift traffic to standby.

If system still reachable (Linux)

• Capture uptime/last boot and kernel messages:

who -b
journalctl -k -b -1    # logs from previous boot
dmesg -T | tail -n 200

• Preserve crash dump / vmcore:

# check kdump
systemctl status kdump
ls -l /var/crash /var/lib/kdump
# copy vmcore off-box
scp /var/crash/<vmcore> analyst@collector:/data/

• Trigger safe sysrq (if hung and acceptable):

echo s > /proc/sysrq-trigger   # sync
echo u > /proc/sysrq-trigger   # remount ro
echo b > /proc/sysrq-trigger   # reboot (only if necessary)

• Collect config and state:

tar czf /tmp/sysinfo.tgz /etc /var/log
ss -tunap > /tmp/conns.txt
ps aux > /tmp/ps.txt
scp /tmp/sysinfo.tgz analyst@collector:/data/

• If memory dump missing, use makedumpfile/crash utilities to extract vmcore.

If system still reachable (Windows BSOD remote collection)

• Retrieve minidump and MEMORY.DMP:
  • Copy from \host\C$\Windows\Minidump and C:\Windows\MEMORY.DMP to safe storage.
• Query Windows event logs remotely:

Get-WinEvent -ComputerName host -FilterHashtable @{LogName='System';ID=1001} | Format-List
wevtutil qe System /q:"*[System[(EventID=1001)]]" /f:text > system_bsod.txt

• Check crash settings:

Get-WmiObject -Class Win32_ComputerSystem -ComputerName host

• If machine down, collect SAN snapshots or disk images rather than booting to avoid altering evidence.

Forensics & analysis

• Preserve timestamps and chain-of-custody (who copied which files and when).
• Analyze vmcore/minidump with crash, makedumpfile, WinDbg:
  • Linux: crash /usr/lib/debug/lib/modules/... /path/to/vmcore
  • Windows: WinDbg -z MEMORY.DMP !analyze -v

Restore service safely

• If quick rollback possible, failover to replica or boot last known-good image.
• If repair required, boot into rescue mode, apply kernel rollback or driver updates, test in staging, then bring back gradually.
• Postmortem: root cause, mitigations (kdump enabled, kernel/drivers pinned, monitoring alerts), and update runbooks.

Key priorities: copy crash dumps off-box, avoid writing to evidence disks, restore service via failover, and perform controlled analysis offline.

Direct answer

Regulatory and compliance regimes almost never dictate a specific recovery architecture; they dictate what your continuity program has to contain, document, and be able to prove on demand. In practice that means: a written plan with defined scope, a documented business impact and criticality analysis, defined and justified recovery objectives, evidence that the plan has actually been tested (not just written), and, for most regimes that matter to a mid-size company, explicit coverage of the third parties your critical functions depend on. Different regimes add different specifics on top of that shared skeleton, and the program has to satisfy all of the ones that apply at once, not run a separate plan per regulation.

Structured elaboration

What auditors and examiners actually check. Across the regimes I've worked with, the common ask is: a documented plan naming its scope, a documented impact/criticality analysis, recovery objectives with a rationale for how they were set, dated records of exercises and their findings, evidence that findings got remediated, and a defined review and maintenance cadence. The plan is the artifact; a well-run internal system with no paper trail behind it often fails an audit anyway, because there's nothing to sample.

How named regimes each shape the program:

What actually changes in the program because of these obligations. Scope has to be explicit (what's covered, what's deliberately out, and why), evidence has to be retained (dated exercise records, findings, remediation sign-off), third parties have to be inventoried and included proportional to criticality, and there has to be a named owner and review cadence for the plan itself, not just for the systems it covers.

Worked example

A mid-size regional hospital network under HIPAA needs a written contingency plan covering every system that touches electronic protected health information. Concretely: a data backup plan (required), a disaster recovery plan to restore lost data (required), and an emergency-mode operations plan describing how critical clinical functions, like verifying a patient's active medication list before a new prescription is administered, keep running safely if the electronic health record system is down (required). Testing and a criticality analysis are "addressable": the network can satisfy this by running an annual tabletop exercise and documenting the results, or by documenting a specific reason that approach isn't reasonable for a given system and describing the equivalent measure it uses instead. If the Office for Civil Rights investigates after an incident, what gets requested first is the plan itself, the criticality analysis, and the exercise records, not a description of the network's storage architecture.

A second, shorter example: a mid-size payments company subject to bank-examiner-style BCM oversight maintains a register of its critical ICT third-party providers (its card processor, its cloud host) that states each provider's own tested recovery capability, the contractual audit and notification rights the company holds, and how that provider is included in the company's own exercise program. That register, not a technical diagram, is what an examiner reviews.

Trade-offs and pitfalls

The most common and costly failure is treating compliance as a paperwork exercise: a plan that exists, was reviewed once at signing, and has never been tested satisfies no one, regulator or business, when a real disruption hits. A second failure is misreading HIPAA's "addressable" as "optional": it still requires either the specified control or a documented, reasoned equivalent, and "we didn't get to it" is not that. A third is scoping the plan too narrowly to reduce testing burden, which creates a gap exactly where the regulator (and the business) cares most. Finally, when an organization sits under more than one regime at once (a healthcare company that also processes card payments, for example), the efficient answer is one program whose documentation maps to each regime's specific asks, not parallel plans that drift out of sync with each other.