DoorDash Systems Administrator (Junior Level) - Interview Preparation Guide

Approach

• Create a user "deploy" with sudo and an SSH public key.
• Ensure apt lists are updated and required packages installed.
• Start and enable nginx; install a monitoring agent (example: datadog-agent placeholder).
• Keep the snippet minimal and ready to paste as cloud-init user-data.

cloud-init user-data (YAML)

#cloud-config
users:
  - name: deploy
    gecos: Deploy User
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - "ssh-rsa AAAA...your-public-key... user@example.com"

package_update: true
packages:
  - nginx
  - curl
  - apt-transport-https

runcmd:
  - [ sh, -c, "systemctl enable --now nginx" ]
  - [ sh, -c, "echo 'Installing monitoring agent (example)'; curl -sS https://example.com/install-agent.sh | bash" ]

Notes:

• Replace the SSH key and monitoring-agent install command/URL with your actual public key and vendor instructions (e.g., Datadog, Prometheus node-exporter).
• For production, pin package versions or add apt repositories and GPG keys securely.

Direct answer

Progressive disclosure means showing the minimum someone needs to make their next decision first, then letting them opt into more detail only if they need it, rather than presenting every layer of a decision at once. For technical documentation, that means separating what we decided and why it matters from how it's actually implemented, and only showing the second layer to someone who asks for it.

Structured elaboration

Two concrete ways to build this into documentation:

1. A "decision, then detail" page structure. The top of the page states the decision and its business-relevant effect in one or two sentences. Directly below, an expandable or clearly linked section holds the reasoning (why this option over the alternatives, what constraint drove it), and a separate section holds the implementation (exact commands, config, code). A reader making a go/no-go call never has to scroll past architecture detail to find the decision.
2. Collapsed detail blocks inside a page that stays otherwise readable. Long code blocks, diagrams, or benchmark tables default to collapsed, with a label that tells the reader what's inside before they open it, not just "details." This keeps the page skimmable top to bottom for someone doing a first pass, while an engineer implementing the change can expand everything in order.
   Both work because they let the reader choose their own depth instead of the writer choosing it for them, and because the label on each layer, decision, why, how, tells the reader which layer they're in before they commit to reading it.

Worked example

A raw engineering note might read: "Switched to regional read replicas with async replication and connection pooling via PgBouncer to cut p95 read latency." Applied with progressive disclosure, the page becomes:
Top line (decision layer): "We added copies of the database closer to users in each region so read requests don't have to cross the country, which is what was making some pages feel slow for customers far from our main data center."
Expandable "why" layer: explains the latency problem was concentrated in specific regions and why a cache alone wasn't sufficient, still in plain language.
Expandable "implementation" layer, collapsed by default: regional read replicas (copies of the database kept near each user region), updated by async replication (the copy is written a short delay after the original, not instantly), and connection pooling via PgBouncer (a tool that reuses open database connections instead of opening a new one per request), plus config snippets and the failover procedure.
A reader deciding whether to approve the change never has to parse "PgBouncer" or "async replication" to get the decision; an engineer implementing it clicks straight through to exactly that.

Trade-offs and pitfalls

Progressive disclosure can misfire if the top layer is vague instead of just simple: "we improved performance" tells the reader nothing they can act on, while "reads are faster for users far from our main region" does. It also fails if the label on a collapsed section doesn't say what's inside; readers won't expand something called "details," so label it with what they'll actually get, for example "config and rollback steps." And it isn't free: every layer you maintain is another thing that can drift out of sync with the code, so it's worth it for docs people repeatedly return to, not a one-off internal note nobody will reread.

Immediate goals: preserve evidence, gather diagnostic data, minimize downtime (failover/rollback), avoid further writes to damaged system.

Initial triage (both OSes)

• Verify reachability and alerts, note time of crash and affected services.
• Put host into maintenance mode (disable monitoring/automated remediation) and if possible shift traffic to standby.

If system still reachable (Linux)

• Capture uptime/last boot and kernel messages:

who -b
journalctl -k -b -1    # logs from previous boot
dmesg -T | tail -n 200

• Preserve crash dump / vmcore:

# check kdump
systemctl status kdump
ls -l /var/crash /var/lib/kdump
# copy vmcore off-box
scp /var/crash/<vmcore> analyst@collector:/data/

• Trigger safe sysrq (if hung and acceptable):

echo s > /proc/sysrq-trigger   # sync
echo u > /proc/sysrq-trigger   # remount ro
echo b > /proc/sysrq-trigger   # reboot (only if necessary)

• Collect config and state:

tar czf /tmp/sysinfo.tgz /etc /var/log
ss -tunap > /tmp/conns.txt
ps aux > /tmp/ps.txt
scp /tmp/sysinfo.tgz analyst@collector:/data/

• If memory dump missing, use makedumpfile/crash utilities to extract vmcore.

If system still reachable (Windows BSOD remote collection)

• Retrieve minidump and MEMORY.DMP:
  • Copy from \host\C$\Windows\Minidump and C:\Windows\MEMORY.DMP to safe storage.
• Query Windows event logs remotely:

Get-WinEvent -ComputerName host -FilterHashtable @{LogName='System';ID=1001} | Format-List
wevtutil qe System /q:"*[System[(EventID=1001)]]" /f:text > system_bsod.txt

• Check crash settings:

Get-WmiObject -Class Win32_ComputerSystem -ComputerName host

• If machine down, collect SAN snapshots or disk images rather than booting to avoid altering evidence.

Forensics & analysis

• Preserve timestamps and chain-of-custody (who copied which files and when).
• Analyze vmcore/minidump with crash, makedumpfile, WinDbg:
  • Linux: crash /usr/lib/debug/lib/modules/... /path/to/vmcore
  • Windows: WinDbg -z MEMORY.DMP !analyze -v

Restore service safely

• If quick rollback possible, failover to replica or boot last known-good image.
• If repair required, boot into rescue mode, apply kernel rollback or driver updates, test in staging, then bring back gradually.
• Postmortem: root cause, mitigations (kdump enabled, kernel/drivers pinned, monitoring alerts), and update runbooks.

Key priorities: copy crash dumps off-box, avoid writing to evidence disks, restore service via failover, and perform controlled analysis offline.

Overview / goal
I’d balance immediate reliability, long-term performance, and new features by using a transparent, score-based prioritization that ties technical work to business impact and SLAs.

Prioritization framework

• Score work by: Impact (customer-facing SLA or revenue risk), Urgency (incident vs. planned), Effort (person-days), Risk reduction (likelihood of future incidents).
• Use WSJF-like formula: Priority = (Impact + Urgency + RiskReduction) / Effort.
• Reserve a weekly incident SLA lane (e.g., >30% capacity) for fire-fighting; allocate remaining capacity to a mix: 50% reliability/perf projects, 30% feature ops, 20% innovation/tech debt.
• Example items: patching critical CVE (high urgency, low effort), DB query tuning (high impact, medium effort), instance right-sizing (cost-saving, low risk).

Metrics to demonstrate ROI

• Operational: MTTR, MTBF, incident count by severity, on-call hours.
• Performance: P95 / P99 latency, error rate, throughput.
• Financial: cloud spend before/after (e.g., $/req), avoided outage revenue loss.
• Business: % of SLAs met, customer support tickets related to performance.

Getting leadership buy-in

• Present prioritized roadmap with quantified impact and cost estimates, plus a 90-day pilot showing measurable wins (e.g., reduced P95 by X ms, saved $Y/month).
• Tie improvements to business KPIs (revenue, retention, SLA penalties).
• Provide executive dashboard and monthly playbooks showing trends and risk posture.
• Offer a “stop-the-bleeding” SLA guarantee: incidents addressed within defined window to reassure stakeholders.

Situation & objectives
RTO ≤ 4 hours, RPO ≤ 1 hour. Small app: 3 web servers, Postgres DB, 500 GB NFS user uploads. Minimize cost while meeting SLAs.

Backup strategy (high-level)

• PostgreSQL: continuous WAL shipping (streaming replication to a standby or archive WALs) + nightly base backups (pg_basebackup).
• NFS uploads: hourly incremental snapshots (filesystem-level or block-level) + daily full snapshot.
• Web servers: stateless; keep config as code (Ansible/Git). Back up configs + package lists nightly.

Schedules & RPO/RTO mapping

• WAL streaming enables point-in-time recovery to ≤ 1 hour RPO (WAL upload every 1–5 min).
• Base backup: nightly full (keep 7 days). WAL retention to cover at least 24–48 hours.
• NFS snapshots: hourly retention 24 hours, daily full retention 14 days, weekly archives 90 days (move older to colder storage).

Storage targets & cost

• Primary backups to cheap object storage (S3/Glacier or equivalent). Keep latest 48h snapshots on faster storage (S3 Standard/IA) for quick restore; archive older to Glacier/Coldline.
• Replicate critical DB WALs to a second region/cloud account if budget allows.

Recovery plan & RTO

• DB failover to standby (minutes) or restore from base + WAL replay: estimated < 2 hours.
• NFS restore: restore latest hourly snapshot to a new volume and remount; 500 GB from object-store snapshot to fast block storage (provision parallel workers) — target < 2 hours.
• Web servers redeploy from config repo → < 30 minutes.

Testing & verification

• Quarterly full DR test: simulate db loss and NFS loss; restore into isolated VPC, verify app end-to-end.
• Monthly partial tests: restore one hourly snapshot and apply WALs to validate integrity; verify checksums and file counts.
• Daily automated backup validation: verify snapshot creation, WAL archive success, and run quick checksum.

Retention & cleanup

• Hourly: keep 24
• Daily: keep 14
• Weekly: keep 12 (months)
• Monthly/Archive: keep 1 year on cold storage

Operational notes

• Automate with cron/backup tool (Borg/Restic for files + lifecycle policy to object storage); use monitoring and alerts for failed backups.
• Encrypt backups at rest and in transit; restrict access via IAM.
• Document runbooks for restore steps and maintain RTO timers.

Situation / Goal
I needed Help Desk to reset passwords and unlock accounts only for users in the Employees OU, with full auditability and least-privilege controls.

Delegation steps (practical)

• Open Active Directory Users and Computers, right-click Employees OU → Delegate Control.
• Add the Help Desk group.
• Use the wizard: choose “Create a custom task to delegate” → “Only the following objects in the folder” → select “User objects.”
• Grant permissions: “Reset user passwords and force password change at next logon.”
• If unlock requires explicit ACL change, use Advanced Security on the OU to grant Write lockoutTime (or use granular ACE via “Delegate Control” to add that right).

If using AD ACLs directly

• Open ADUC → View → Advanced Features → OU Properties → Security → Advanced → Add Help Desk group → select specific Extended Rights: “Reset Password” and allow write lockoutTime.

Auditing configuration

• Enable success auditing for “Account Management” in Group Policy (Computer Configuration → Policies → Windows Settings → Security Settings → Advanced Audit Policy Configuration → Account Management → Audit User Account Management: Success).
• Configure SACL on Employees OU (Advanced Security → Auditing) to audit Success for Reset Password and Write lockoutTime.
• Forward logs to SIEM; alert on Event IDs: 4724 (password reset attempt), 4767 (account unlocked), and correlate with ticket IDs.

Procedural controls for least privilege & accountability

• Require ticket with unique ID and approval before Help Desk action; log ticket ID in AD operation notes.
• Enforce MFA for Help Desk consoles and privileged sessions.
• Monthly access review (recertification) of the Help Desk group memberships.
• Implement Just-In-Time elevation (Privileged Access Management) for sensitive cases.
• Retain audit logs 1 year; run weekly automated reports of resets/unlocks and investigate anomalies.

Result: Help Desk can perform necessary support without expanded rights, all actions are auditable and controlled, preserving least privilege and accountability.

Direct answer: Redundancy is having extra, interchangeable components standing by so one can take over if another fails; replication is actively keeping copies of state (data) synchronized across multiple nodes so the state itself survives a failure, not just the compute that serves it. They're often used together, but they solve different problems: redundancy alone is enough for a stateless service, because any interchangeable instance can serve any request; a stateful service needs replication too, because a fresh redundant instance with no data isn't actually a working replacement.

Structured elaboration

Stateless example: a set of identical app server instances behind a load balancer, handling API requests with no local state. If one instance dies, the load balancer routes around it and an autoscaler replaces it; the new instance needs no data transfer because there was never any instance-local state to lose. This is pure redundancy: extra interchangeable copies of the same stateless computation.

Stateful example: a primary-replica database. The primary accepts writes; replicas continuously receive a copy of the write stream (replication). If the primary fails, a replica is promoted to take over. Unlike the stateless case, simply having an extra database instance running (redundancy alone, no replication) would give you an empty database, not a working replacement, because there's no mechanism copying the actual data into it.

A failure mode redundancy alone doesn't protect against, for the stateful case: data loss or corruption on the primary itself. If the primary's disk corrupts a row, or a bad write silently corrupts application-level data, having a redundant (but not yet caught-up, or synchronously replicating that same bad write) standby doesn't help, because either the standby doesn't have the data yet (async lag) or it faithfully replicated the corruption along with everything else (synchronous replication of a logically bad write). Redundancy protects against a node dying; it does not protect against the data itself being wrong, that requires backups (a separate, point-in-time copy decoupled from live replication) and, for silent corruption specifically, checksums or application-level validation.

How this generalizes: a useful mental checklist for fault tolerance covers five distinct techniques, and redundancy and replication are only two of them: retries (recover from a transient failure by trying again), bulkheads (isolate one failure from spreading to unrelated resources), failover (the mechanism that switches traffic to a healthy replacement), redundancy (having that replacement exist at all), and replication (making sure the replacement actually has the state it needs). A strong answer names which of these a given design decision is actually addressing, since "redundancy" gets used loosely to mean all five in casual conversation.

Trade-offs & pitfalls

• Replication has a cost redundancy alone doesn't: network bandwidth, storage for extra copies, and a consistency model to reason about (synchronous replication costs write latency; asynchronous replication risks data loss on failover, the classic RPO trade-off).
• A common wrong turn: assuming "we have 3 replicas" automatically means "we're protected," without checking replication lag. A replica that's minutes behind at failover time silently loses however much data arrived in that window, unless the promotion logic explicitly accounts for lag and refuses to promote a too-far-behind replica.
• Redundancy for stateless services is comparatively cheap and low-risk to over-provision; replication for stateful services is not, since more replicas means more write-path coordination overhead (for synchronous replication) or more divergence risk (for asynchronous/multi-leader), so it isn't a "just add more" lever in the same way.

Direct answer

In the moment, my first job is to actually listen to the criticism rather than start explaining or defending myself before I've fully heard it, even when the instinct to justify is strong. Afterward, rebuilding trust isn't about the conversation where I received the feedback, it's about visibly acting differently going forward in the specific way the feedback pointed at.

Structured elaboration

• Managing the reaction in the moment: the instinct right after an outage, already stressed, is to explain the context and mitigating factors as soon as criticism starts. I've learned to let the person finish first, genuinely hear the specific complaint, and only then respond, since jumping in early to explain often lands as defensiveness even when that isn't the intent.
• Separating the valid signal from the delivery: tough feedback right after an outage often arrives with real frustration attached. The useful move is extracting the actual substance, what specifically should have gone differently, rather than reacting to the tone it arrived in.
• Not over-apologizing either: there's a version of managing the reaction that overcorrects into excessive self-criticism, which doesn't address the substance any better than defensiveness does; the goal is a level, accurate acknowledgment, not performing contrition.
• Rebuilding trust afterward: the actual trust repair happens in what changes afterward, doing the specific thing the feedback pointed at differently next time, not in how gracefully the original conversation went.

Worked example

Right after an outage I'd contributed to, my manager gave me direct, pointed feedback in a one-on-one: that I'd been slow to escalate once it became clear I was stuck, and that the delay had made the outage longer than it needed to be. My first instinct was to explain the reasoning that had made sense to me in the moment, that I'd thought I was close to a fix. I held off on that and let them finish first, and once I actually listened past my own defensiveness, the specific point was fair: I had, in fact, kept trying alone for longer than made sense given how the situation was unfolding.

I acknowledged the specific point directly rather than the vaguer "I hear you, I'll do better," and said what I'd concretely do differently: escalate earlier next time I'm stuck past a set point, rather than continuing to push alone. The actual trust rebuilding happened over the incidents that followed, not in that conversation. In the very next incident where I got stuck, I escalated well before I would have previously, and I made a point of telling my manager afterward that I'd deliberately applied the earlier feedback, which is what actually closed the loop for them, seeing the specific behavior change rather than just hearing that I'd taken the feedback well.

Trade-offs and pitfalls

The common failure mode is treating receiving feedback well as the whole task, being gracious and non-defensive in that one conversation and considering it handled. Without a visible change in behavior afterward, gracious listening reads as agreeable in the moment and forgotten a week later, which damages trust more than a defensive reaction followed by real change would. The other trap is swinging to excessive self-criticism, which can feel like taking it seriously but doesn't actually engage with the specific, actionable substance of the feedback any better than dismissing it does.

Direct answer

That specific message means the SSH daemon's (sshd's) strict permission checks rejected bob's home directory or .ssh directory before it ever evaluated the key itself. By default, sshd refuses to trust a public key if the account's home directory, its .ssh directory, or the key file is writable by anyone other than its owner, because a group- or world-writable path there would let another local user plant their own key and log in as bob. The fix is tightening ownership and permissions on the exact chain of directories sshd checks, then verifying the fix actually took effect.

Structured elaboration

Root causes. Any of these produce the same "bad ownership or modes" message:

1. /home/bob is not owned by bob (or root), often from files restored by a backup tool or copied in by another administrator's account.
2. /home/bob or /home/bob/.ssh has a group- or world-writable bit set (sshd's StrictModes setting, on by default, rejects this).
3. /home/bob/.ssh/authorized_keys itself is writable by group or other, even if the directories above it are fine.

Commands to fix it, applied in the order that matches how sshd walks the path:

# 1. Home directory must be owned by bob, and not writable by group or other.
sudo chown bob:bob /home/bob
sudo chmod go-w /home/bob

# 2. .ssh directory: owner-only access.
sudo mkdir -p /home/bob/.ssh
sudo chown bob:bob /home/bob/.ssh
sudo chmod 700 /home/bob/.ssh

# 3. authorized_keys file: owner-only read/write, no execute needed.
sudo chown bob:bob /home/bob/.ssh/authorized_keys
sudo chmod 600 /home/bob/.ssh/authorized_keys

Verifying the fix. Confirm the actual mode bits, not just that the commands ran without error:

stat -c "%a %U:%G %n" /home/bob /home/bob/.ssh /home/bob/.ssh/authorized_keys

Expect something like 755 bob:bob /home/bob, 700 bob:bob /home/bob/.ssh, and 600 bob:bob /home/bob/.ssh/authorized_keys. Then attempt the actual login with verbose logging so sshd's own reasoning is visible: ssh -vvv bob@host, and separately watch the server side with sudo tail -f /var/log/auth.log (or /var/log/secure on Red Hat-family distributions) for the corresponding "Accepted publickey" line, rather than assuming success from the client side alone.

If permissions look correct and it still fails, check SELinux (Security-Enhanced Linux, a mandatory access control system available on many Linux distributions). Standard Unix permission bits and SELinux are two independent checks; a directory can have perfectly correct ownership and mode and still be denied if its SELinux file context (label) is wrong, most commonly after files were restored from a backup or copied from another location without preserving context:

ls -Z /home/bob/.ssh /home/bob/.ssh/authorized_keys

.ssh and its contents should carry the ssh_home_t context. If it shows something else (commonly a generic context like user_home_t after a restore), restore the correct label rather than trying to set it by hand:

sudo restorecon -Rv /home/bob/.ssh

If that does not resolve it, check for an actual denial being logged rather than guessing further:

sudo ausearch -m avc -ts recent

An access vector cache (AVC) denial entry naming sshd and the .ssh path confirms SELinux, not a stale permission bit, is the actual blocker.

Worked example

A concrete before-and-after for this exact scenario. Suppose /home/bob/.ssh was left group-writable (mode 775) after a provisioning script ran as root and did not lock it down, a common way this exact error gets introduced:

Before: drwxrwxr-x  bob bob  /home/bob/.ssh   (group-writable, sshd rejects this)
        -rw-r--r--  bob bob  /home/bob/.ssh/authorized_keys

After applying the fix commands above:
        drwx------  bob bob  /home/bob/.ssh
        -rw-------  bob bob  /home/bob/.ssh/authorized_keys

Running the fix commands against exactly this before-state (verified directly: chmod go-w on the home directory, chmod 700 on .ssh, chmod 600 on authorized_keys) produces exactly the after-state shown, the group-write bit is gone from .ssh and authorized_keys no longer has any bits set beyond owner read/write. That is the state sshd's strict-mode check requires before it will even look at the key's contents.

Trade-offs and pitfalls

• Fixing only authorized_keys and skipping the parent directories is a common half-fix. sshd checks the whole path, home directory, .ssh, and the key file itself, so a perfectly permissioned key file inside a group-writable .ssh directory still fails.
• Assuming a fix worked because the commands returned no error is a common mistake. Always verify the actual mode bits with stat, and confirm the login itself succeeds with verbose logging, rather than trusting that chmod silently doing nothing wrong means the state is now correct.
• Jumping straight to SELinux before confirming ownership and mode bits wastes time. SELinux denials produce a similar practical symptom (login refused) but a different log signature; check the simpler, more common cause first, and only chase an SELinux label mismatch once the standard permission chain is confirmed correct.
• Manually setting an SELinux context instead of using restorecon risks getting the label subtly wrong. restorecon restores the context the system's own policy defines for that path, which is more reliable than guessing a context string by hand.

Direct answer
maxSurge controls how many EXTRA pods above your target replica count can exist temporarily during a rollout; maxUnavailable controls how many pods can be DOWN (below target) at once. Together they set the trade-off between rollout speed/extra resource usage and how much capacity you're willing to sacrifice mid-rollout.

Structured elaboration
For a 100-pod deployment:

• maxSurge: 25% means up to 25 extra pods can be created (125 total running temporarily) before old ones start being removed.
• maxUnavailable: 25% means up to 25 pods can be unavailable at once, so at minimum 75 pods are always serving traffic.
• Setting both to 0 isn't allowed (Kubernetes needs at least one of them to make forward progress), and setting both to a large value maximizes rollout SPEED at the cost of both extra resource consumption AND reduced available capacity simultaneously.

Latency-sensitive service: prefer maxUnavailable: 0 (never drop below full target capacity, since capacity loss directly hurts p99 latency under load) combined with a modest maxSurge (say 10-25%) so new pods come up gradually without demanding a huge resource spike all at once. This is slower but protects the SLA.

Batch worker: can tolerate maxUnavailable being higher (say 50%), since a temporarily smaller worker pool just means jobs queue a bit longer rather than users seeing degraded latency; this lets the rollout complete faster with less need for surge capacity, since you're willing to sacrifice throughput temporarily instead of paying for extra headroom.

Worked example
For the latency-sensitive service at 100 replicas with maxUnavailable: 0, maxSurge: 25: Kubernetes first creates 25 new pods (125 total), waits for them to become ready, then removes 25 old pods (back to 100, all new), repeats: creates 25 more new (125 again, this time mixed), removes 25 more old, and so on, always keeping at least 100 healthy pods serving traffic throughout. For the batch worker at maxUnavailable: 50, maxSurge: 0: Kubernetes removes up to 50 old pods first (down to 50 capacity), creates 50 new ones to replace them, and repeats, needing no extra capacity headroom at all but running at half capacity partway through.

Trade-offs and pitfalls
Higher surge means faster rollouts but a temporary resource spike your cluster's autoscaler or node capacity needs to be able to absorb; if it can't, new pods stall in Pending and the rollout stalls too. The common mistake is picking these values once and never revisiting them as traffic patterns or cluster capacity change, so a setting that was safe at launch quietly becomes risky (or needlessly slow) as the service scales.