DoorDash Staff Systems Administrator Interview Preparation Guide

Direct answer

Capacity planning for a fixed target load is a four-step exercise: measure how much load one instance can safely handle, add a burst/growth buffer to the raw peak, convert that into an instance count, then validate the fleet still meets the target after you lose a zone. The safety margin exists to absorb burstiness, retries, and the cost of running hot for a bounded period, not to compensate for skipping the measurement step.

Structured elaboration

Data to collect

From data to a per-instance capacity number

Run a load test against a single instance (or a fixed-size shard) and raise load until the p99 latency SLO (service level objective: the latency target you've committed to, e.g. p99 under 300ms) breaches, not until CPU hits 100%. That breach point, not the theoretical ceiling, is the instance's safe capacity.

From per-instance capacity to fleet size

1. Planning target = raw peak times a growth/burst buffer.
2. Divide by (safe per-instance capacity times target operating utilization).
3. Round up to whole instances, then round up again to a multiple of the availability-zone count so load balances evenly.

Validate the zone-loss case

After removing one zone's worth of instances, confirm the remaining fleet still covers the raw peak, not the buffered planning target. Running hot for the duration of a single zone outage is an acceptable, bounded trade; running hot while also absorbing organic growth beyond the raw peak is not.

Worked example

A load test on one 4-vCPU instance shows it sustains 1,000 requests/sec at 60% CPU before p99 latency crosses the SLO. That 60% point, not 100%, is the safe ceiling.

Target steady-state operating point: 50% CPU, leaving headroom for GC pauses, noisy neighbors, and the zone-loss case.

Per-instance safe capacity at the 50% target:

Add a 30% burst/growth buffer to the stated 10,000 req/s peak:

Instances needed for the planning target:

Round up to a multiple of 3 availability zones for even spread: 18 instances, 6 per zone.

Validate the zone-loss case. Losing one zone removes 6 instances, leaving 12:

That equals the raw peak exactly: during a single-zone outage the fleet runs at its safe ceiling with zero spare margin, an acceptable, bounded degradation for a rare event but not a state to run in day to day.

Trade-offs & pitfalls

• Over-provisioning for the zone-loss case permanently (running at reduced utilization at all times to cover a rare event) wastes money; treat zone-loss headroom as a temporary, monitored state, not the steady-state target.
• Linear scaling from a single instance's load test breaks down when a bottleneck is shared across the fleet (database connection pool limits, a shared NAT gateway, a shared cache); load-test a small cluster against the real shared dependency, not one box in isolation.
• Retries during partial degradation can multiply effective load two to three times right when capacity is tightest; a plan that ignores retry amplification underestimates the real peak.
• CPU utilization is a proxy, not the constraint. An I/O-bound service can sit at 20% CPU while thread-pool exhaustion still causes timeouts, so the load test's stopping condition should always be the SLO breach, never a resource metric alone.

Direct answer. Ownership has to be a named role tied to the business function or system the runbook covers, not "whoever wrote it last," with versioning that ties each revision to a specific trigger, not just a date. Catching drift before a real event means testing the runbook during a planned exercise instead of discovering it's wrong while executing it under pressure, paired with a lightweight review trigger whenever something the runbook depends on changes.

1. Ownership

Assign a named owner to every runbook, ideally the person or role accountable for the business function or system it covers, not a shared team inbox and not necessarily the original author, who may move on. Ownership means being responsible for the runbook's accuracy, not personally executing every step during a real event. Make it visible on the document itself (name, role, last-reviewed date) so anyone opening it during an incident can see who to ask if something looks wrong, and so a stale "owner" who's left the organization is an obvious red flag rather than a hidden one. When ownership changes hands, require an explicit handoff with a fresh review as part of the transition, not a silent reassignment.

2. Versioning

Treat runbooks as controlled documents: every change gets a version number, a date, who made it, and why, kept in the document's own change history rather than relying on file metadata or institutional memory. Tie versions to triggers, not just a calendar. A scheduled review is a reasonable baseline, but the more reliable trigger is linking review to the events that actually make a runbook wrong: the system it depends on changes, the org structure around it changes, or a real exercise or incident surfaces a gap. A runbook untouched for eighteen months right after the system it describes was rebuilt is far more suspect than one reviewed on schedule six months ago. Keep prior versions accessible, not just the latest, so a post-event review can tell whether a gap that showed up was already known and fixed but hadn't propagated to whoever was executing, or was genuinely new.

3. Catching drift before it matters

The strongest mechanism is using the runbook for real, on a schedule, inside an exercise (a tabletop discussion or a functional exercise that actually executes some steps), before a live event forces the discovery. A runbook nobody has walked through since it was written is a hypothesis, not a tested procedure. Build a lightweight checklist step into whatever process generates the kind of change that breaks runbooks: when a system change, a supplier change, or an org change is being planned, that process flags which runbooks might be affected, so the review happens close to the change instead of being discovered much later. Track a simple staleness signal across the whole runbook set (time since last review, time since the system or team it covers last changed) and report on it the way you'd report any other risk metric, so an accumulating backlog of unreviewed runbooks is visible to whoever owns the continuity program overall, not discovered runbook by runbook.

Worked example

A payments-processing runbook is owned by the payments platform lead, currently at version 4: v2 added a fallback provider, v3 corrected an escalation contact list after a reorg, v4 followed a functional exercise that found the documented rollback step no longer matched how the system actually rolled back. Six months after v4, the team migrates the payments platform to new infrastructure as part of an unrelated project. Because "infrastructure migration for a system with an owned runbook" sits on that project's change-review checklist, the payments platform lead is flagged automatically and confirms the runbook needs an update before the migration ships, catching the drift as part of the planned change rather than during the next incident. At the next scheduled functional exercise three months later, the team walks through the updated runbook end to end and finds one further step, a monitoring dashboard link, still pointing at the decommissioned infrastructure. That becomes a logged finding and v5.

Trade-offs & pitfalls. A purely calendar-based review cadence catches drift too slowly for a fast-changing system and too often for a stable one; tying review to real change triggers is more work to set up but matches effort to actual risk. Ownership without accountability for staleness (a name on a document nobody ever checks) is barely better than no ownership; the staleness signal has to surface to someone, not just exist. And testing a runbook only during a real event is the worst possible time to discover it's wrong: the exercise programme is what lets a team find the gap on an ordinary afternoon instead of during an actual outage.

Direct answer

Layer LUKS (Linux Unified Key Setup, the standard on-disk format for the kernel's dm-crypt block encryption)
directly on the raw partition or disk, and put LVM (Logical Volume Manager, the layer that lets you resize,
snapshot, and combine block devices without repartitioning) on top of the single decrypted mapping, unless
different logical volumes genuinely need different keys or separate access boundaries, in which case put
LVM below and encrypt each logical volume individually and accept multiple unlock prompts. Resizing,
rekeying, and recovery all follow directly from getting that layering right.

How the layers interact

• LUKS-under-LVM (recommended default): one LUKS container wraps the raw partition; LVM's physical
  volume sits on the single decrypted /dev/mapper/<name> device; one passphrase or key unlocks everything
  underneath. Simpler operations, one thing to unlock at boot.
• LVM-under-LUKS: LVM creates logical volumes directly on the raw partition, and each logical volume is
  individually wrapped in its own LUKS container. More keys to manage and back up, but supports genuinely
  separate access boundaries per volume (for example, different tenants or different sensitivity levels on
  the same physical disk).

Resizing an encrypted LVM volume, bottom to top

The sequence matters strictly bottom to top, each layer only sees the space the layer below reports:

1. Grow the underlying block device itself (a cloud provider's disk-resize API, or a physical disk swap).
2. Grow the partition if one is in play (growpart <device> <partition-number>).
3. cryptsetup resize <mapped-name> to grow the LUKS mapping into the newly available space. Both header
   formats support doing this online, while the mapping stays active and mounted, no luksClose/luksOpen
   cycle is required for either one. The real difference is elsewhere: LUKS1's header never stores a size
   at all, so resizing it is a pure device-mapper table operation that needs no passphrase, while LUKS2
   persists the new size into its own authenticated metadata, so cryptsetup resize on a LUKS2 device
   requires a passphrase to write that update even though the mapping itself is never torn down. (Verified
   directly: growing an active, opened LUKS1 mapping with cryptsetup resize succeeded immediately with
   no passphrase prompt and no close/reopen; the equivalent LUKS2 resize on an active mapping needed a
   passphrase before it would proceed, but neither one required closing the mapping.)
4. pvresize on the mapped device, so LVM's physical volume sees the larger space.
5. lvextend -l +100%FREE <volume-group>/<logical-volume> to grow the logical volume into the newly
   available extents.
6. resize2fs (ext4) or xfs_growfs (XFS) to grow the filesystem to fill the logical volume.
   Doing pvresize before cryptsetup resize is a common, cleanly diagnosable mistake: LVM will report zero
   new physical extents available, because the crypt mapping is still reporting its old size to everything
   above it.

Rekeying with minimal downtime

cryptsetup luksAddKey <device> adds a new passphrase into a free keyslot (LUKS2 supports up to 32
keyslots depending on header parameters, LUKS1 exactly 8) alongside the existing one, without touching the
actual data encryption key at all, only the passphrase-wrapped copy of it changes. Verify the new
passphrase unlocks the device, then cryptsetup luksRemoveKey (or luksKillSlot) the old one. Because the
underlying master key never changes, this completes in effectively constant time regardless of volume size,
with zero downtime since the device stays mounted throughout. Contrast this with cryptsetup reencrypt
(LUKS2's online re-encryption, which genuinely rewrites the master key and re-encrypts every block), a much
heavier operation that LUKS2 supports resuming from a checkpoint if interrupted, precisely because a crash
mid-reencrypt without that resume metadata would be dangerous.

Emergency recovery from a corrupted LUKS header

The header holds every keyslot's metadata; if it's destroyed, the master key may be unreachable through any
passphrase, by design, that unrecoverability is the point of full-disk encryption, not a defect. The only
reliable recovery path is a header backup taken proactively: cryptsetup luksHeaderBackup --header-backup-file <file> <device> immediately after luksFormat and after every luksAddKey/
luksKillSlot change. If a backup exists, cryptsetup luksHeaderRestore --header-backup-file <file> <device> restores it and normal unlock resumes. Without a backup, cryptsetup repair can attempt limited
recovery of a LUKS1 header, and cryptsetup luksDump shows whatever keyslot metadata survives; if even one
valid keyslot remains, the data is recoverable, if none do, it genuinely is not.

Worked example

A database host runs LUKS directly on a partition with LVM on top, needing to grow from 500GiB (gibibyte,
2^30 bytes) to 750GiB after the cloud disk itself was resized: growpart /dev/nvme1n1 1, then
cryptsetup resize luks_data, then pvresize /dev/mapper/luks_data, then lvextend -l +100%FREE /dev/vgdata/lvdata, then resize2fs /dev/vgdata/lvdata. Skipping the cryptsetup resize step and running
pvresize directly after growpart is the concrete, checkable symptom of doing this out of order: pvresize
reports the physical volume is unchanged in size, because the crypt mapping in between still reports the old
500GiB.

Trade-offs and pitfalls

LUKS-under-LVM's single unlock is operationally simpler (one boot-time passphrase, or a network-bound
unlock via a TPM, trusted platform module, backed mechanism) but means anyone holding that one key can see
every logical volume underneath; LVM-under-LUKS supports per-volume keys at the cost of managing and backing
up several header backups instead of one. The most common mistake in "rotation" is running luksAddKey and
stopping there: an old, potentially compromised passphrase left in its keyslot after adding a new one has
not actually been rotated, only supplemented, luksRemoveKey on the old slot is the step that completes it.

Direct answer

On my last project I codified an AWS microservices platform, a set of backend services running on managed servers with their own database, load balancer, and access controls (VPC, EKS, RDS, IAM, ALB, monitoring) in Terraform, using versioned reusable modules composed per environment, remote state (the file Terraform uses to track what it created, kept separate per environment so a mistake in one can't touch another) isolated per environment, secrets pulled from AWS Secrets Manager and Parameter Store rather than stored in code, and a PR-based workflow where a machine-generated plan had to be reviewed and approved before an apply job with a separate, more privileged role could run.

How I structured it

Module layout and versioning

• Modules lived in a private registry: vpc, eks, rds, iam, alb, monitoring, each with a narrow set of inputs/outputs and no hidden side effects.
• Root configurations per environment composed these modules and pinned each one to a semantic version tag (for example vpc ~> 2.3), so a change to a module's source could not silently change an environment that had not explicitly bumped its pin.

Environment separation and state

• Each environment (dev, staging, prod) had its own remote state file in an S3 backend with a DynamoDB lock table, keyed roughly as s3://infra-state/{env}/{component}.tfstate.
• Isolating state per environment (rather than one shared state with workspaces) meant a mistake in dev could not touch prod's state, and the blast radius of a single apply was limited to one component in one environment.

Secrets handling

• Database credentials lived in AWS Secrets Manager, app configuration lived in Parameter Store encrypted with KMS.
• Terraform read them at apply time through data sources (for example aws_secretsmanager_secret_version) rather than having them typed anywhere in .tf files or CI variables.
• The CI runner itself never held a long-lived key: it assumed a role via OIDC scoped to the minimum permissions needed for that environment's apply.

Change approval and safe apply

• Every change went through a PR. CI ran terraform plan and posted the JSON plan (with sensitive values redacted) as an artifact, plus a readable summary on the PR.
• A change needed sign-off from both the infra owner and the owning service team before the apply stage would even unlock.
• Apply ran as a separate pipeline stage using a more privileged, MFA-gated role for prod, and every apply was logged to CloudTrail.
• Destructive changes (anything showing a resource replacement or delete in the plan) required an explicit manual confirmation step, and we took a fresh snapshot or backup first.

Worked example

Concretely: adding a read replica to an existing RDS instance meant one PR that touched only the rds module's call site, a plan that showed one resource create and zero destroys, review from the database owner since it touched a stateful resource, and an apply that ran with the prod-apply role only after both approvals landed. Because the module was version-pinned and state for that component was isolated, the blast radius of that single PR was exactly one RDS resource in one environment: nothing else in the account could be affected by that apply.

Trade-offs and pitfalls

• Splitting state by component reduces blast radius but adds cross-stack coordination cost (remote state lookups or SSM parameters to pass values between components); too many tiny state files becomes its own operational burden.
• A plan-then-approve workflow is only as safe as the reviewers actually reading the plan. Without a policy-as-code gate (something like OPA/Rego or Sentinel evaluating the plan JSON for guardrails such as "no public security groups" or "no unencrypted volumes"), review can degrade into rubber-stamping on a busy day.
• Reading secrets via data sources at apply time keeps them out of source control, but the values still land in the Terraform state file in plaintext, so state encryption and tightly scoped state-read IAM matter just as much as the CI-side handling.

Direct answer

The attack surface of a configuration repository and its promotion pipeline splits into four zones an attacker could target: the REPOSITORY itself (who can commit, whose commits get merged), the CI RUNNERS that execute pipeline steps (what they can read and reach), the SUPPLY CHAIN of dependencies and base images the pipeline pulls in, and the PROMOTION MECHANISM that actually applies changes to production. A prioritized control set addresses the zone with the highest (likelihood times blast radius) first: branch protection and required review (repository), ephemeral, least-privilege runner credentials (runners), pinned and verified dependencies (supply chain), and a promotion gate that cannot be bypassed by a single compromised identity (promotion mechanism).

Structured elaboration

Repository zone risks and controls. Risk: a compromised or malicious contributor merges a change that exfiltrates secrets or grants themselves broader access, or force-pushes to rewrite history and hide evidence. Controls, roughly ordered by effectiveness per unit of operational cost: branch protection requiring review from someone OTHER than the author (cheap, high effect, stops the single most common path); required status checks (policy-as-code, secret-scanning) that must pass before merge (cheap, catches accidental leaks and known-bad patterns); disallowing force-push and requiring signed commits on protected branches (moderate cost, closes the history-tampering path); a CODEOWNERS-style requirement that changes to the pipeline definition ITSELF need a security or platform-team reviewer, not just any team member (moderate cost, closes the meta-attack of modifying the pipeline to weaken its own controls).

CI runner zone risks and controls. Risk: a compromised runner (via a malicious dependency executed during the build, or a vulnerability in the runner's own environment) reads secrets injected into the job, or pivots to reach other systems the runner's network position allows. Controls: short-lived, scoped credentials issued PER JOB (OIDC-based federation to the cloud provider rather than a long-lived static secret stored in the CI system) so a compromised runner's blast radius is bounded to that one job's narrow scope and a short time window; network-isolate runners so they cannot reach anything beyond what THAT specific job legitimately needs; and treat runner logs as potentially containing secrets, redacting known secret patterns automatically rather than trusting every script to avoid printing them.

Supply chain zone risks and controls. Risk: a malicious or compromised third-party action, base image, or dependency executes arbitrary code during the pipeline run. Controls: pin dependencies to exact versions or content hashes rather than floating tags (actions/checkout@<sha> not @v4), scan and sign container images used as pipeline steps, and restrict which third-party actions/images are allowed at all (an explicit allowlist for anything with write access to secrets or production credentials).

Unauthorized promotions zone risks and controls. Risk: a compromised identity, or a legitimate identity acting outside intended process, triggers a production promotion without the required review. Controls: require the promotion trigger (a merge to a protected production-overlay path, or an explicit approval gate in the deployment tool) to be tied to a REVIEWED PR event, not a webhook or manual trigger any single credential can fire; and log every promotion event with enough context (who/what triggered it, which commit, which approvals) to make an unauthorized promotion immediately detectable even if it cannot be prevented outright.

Worked example

Prioritized by (likelihood times blast radius), for a mid-size organization with an existing but not security-hardened pipeline:

The first four sit at the top because they each close a HIGH-BLAST-RADIUS path (a merged malicious change, a compromised runner with broad reach, an unpinned supply-chain dependency, or a promotion nobody reviewed) at comparatively low implementation cost; signed commits and some of the more process-heavy controls are real but address a narrower slice of risk relative to their rollout cost, so they land lower without being unimportant.

Trade-offs and pitfalls

• Common mistake: treating a required status check as equivalent to a required human review. An automated check catches known patterns; it does not catch a change that is subtly malicious but syntactically clean, which is exactly what human review is for. The two are complementary, not substitutes.
• Ephemeral, per-job credentials are the single highest-leverage control on this list and also the one most often skipped, because migrating away from a long-lived static secret already embedded in a working pipeline feels riskier to touch than leaving it; the actual risk asymmetry runs the other way, a long-lived static secret is a standing target for as long as it exists.
• An allowlist for third-party actions/images needs an owner and a review process for ADDING to it, otherwise it either calcifies (blocking legitimate new tooling and creating pressure to bypass it) or erodes (approvals granted too casually under time pressure, defeating its purpose).
• Common mistake: prioritizing controls by ease of implementation rather than by risk reduction per unit of cost. The cheapest controls to implement are not always the highest-leverage ones; a genuinely prioritized list, as above, has to weigh blast radius explicitly, not just roll out whatever is fastest to configure first.

Direct answer

Enforce with token bucket, since bursts are explicitly a requirement, and split enforcement into two tiers: a fast local decision on each edge/LB instance seeded by short-lived leases from a shared store, plus a slower, coarser global check for the hard ceiling. Pure centralized counting gives exact enforcement but adds a network round trip to every request; pure local (fully independent) buckets are fast but overshoot badly once you have many edge instances; leases are the practical middle ground production API gateways actually run.

Token bucket vs. leaky bucket

Both are enforcement algorithms for the same underlying problem, deciding whether to admit or reject a request against a rate limit, but they smooth traffic in opposite ways. Token bucket accumulates capacity (tokens) at a steady refill rate up to a cap, and admits a request as long as a token is available, so unused capacity banks up and can be spent all at once: that banked capacity is exactly what lets a legitimate burst through. Leaky bucket instead models requests (or tokens) as entering a queue and draining out to the backend at a fixed, constant rate, regardless of how bursty the arrivals were, so it smooths traffic into a steady output rate rather than permitting bursts through at all; anything arriving faster than the drain rate either queues, if there's room, or is dropped once the queue is full.

That difference is exactly why token bucket is the right choice here: the question requires allowing legitimate bursts, and leaky bucket has no concept of banked, unused capacity to spend on one, its output is rate-limited to the drain rate no matter how idle the bucket was beforehand. Leaky bucket is the better fit when the goal is protecting a downstream that genuinely cannot tolerate any burst at all, such as a fixed-capacity queue or a downstream with no burst headroom of its own, since it enforces a hard, constant ceiling on the outbound rate. Token bucket is the better fit, and the one used below, whenever some burst tolerance is a feature rather than a bug, which this question states as a requirement.

Design options and their trade-offs

For per-user and global quotas together, use a hierarchy: local, lease-backed enforcement for the fast per-user path, and a separate, coarser global counter that only needs reconciling periodically, not per request, since the global ceiling has more slack per individual request.

Worked example

Assume a per-user quota of $100$ requests/minute and a global quota of $50{,}000$ RPS spread evenly across 3 regions absent per-region traffic data:

Per-user leases: the nominal per-user rate is

Choose a lease refresh interval of $5\text{s}$ (a design choice: short enough to bound overshoot, long enough to keep the allocator's own request rate manageable). Lease size, rounded up so a well-behaved user is never starved mid-window:

If a user's requests can land on up to $N = 3$ edge instances within one lease interval (each holding its own local bucket topped up independently), the extra tokens beyond what a single-instance view would allow is bounded by:

That's at most an extra $18/5 = 3.6$ req/s of burst headroom above the nominal $1.667$ req/s during any single 5-second window, a known and tunable bound (shrink the interval or cap $N$ to tighten it), converging back toward the true 100/min rate as leases keep renewing at the true rate over longer horizons.

For the "approximate counting" framing: a sliding-window counter (used instead of a sliding log at this scale) estimates the count in the trailing window as

where $N_{curr}$ is the count so far in the current fixed sub-window, $N_{prev}$ is the previous sub-window's count, and $f$ is the fraction already elapsed into the current sub-window; this blends two cheap counters into an approximation of the true sliding count without storing individual timestamps.

Architecture

flowchart LR
  Client --> Edge1[Edge LB 1: local bucket]
  Client --> Edge2[Edge LB 2: local bucket]
  Client --> Edge3[Edge LB 3: local bucket]
  Edge1 -->|periodic lease request| Store[Shared lease allocator]
  Edge2 -->|periodic lease request| Store
  Edge3 -->|periodic lease request| Store
  Store -->|reconciles| Global[Global quota counter]

Trade-offs and pitfalls

• Fail-open vs fail-closed when the central store is unreachable is a business decision, not just an engineering default: fail-closed protects backends but turns a rate-limiter outage into a full outage; fail-open protects availability but removes protection exactly when a traffic spike might also be stressing the store. A common middle ground is falling back to the last-known lease/local bucket state for a bounded grace period, then failing open.
• Identity matters before any of this works: decide up front how you key a client (authenticated user ID, API key, or IP for unauthenticated traffic), since IP-keyed limits are coarser and easier to evade behind shared NAT.
• Leases trade a small, bounded overshoot for removing the store from the hot path; if the requirement is "never exceed N, ever" (e.g. a paid third-party API billed per call), leases are the wrong tool and the centralized counter's exactness is worth the latency cost.
• Global quota reconciliation across regions needs its own cadence: checking it per request defeats the purpose of regional autonomy, but checking it too infrequently lets regions collectively exceed the ceiling before anyone notices.
• Approximate sketches suit extreme cardinality (e.g. anti-abuse detection across millions of anonymous IPs) but are a poor fit for quotas you have to justify to a paying customer, since "why was I throttled" needs a precise answer.

Approach overview
I’d balance uptime and safety with a mixed strategy: prefer live patching for urgent security fixes, use rolling reboots for full-kernel changes, rely on HA/failover for planned reboots, and use boot-segmentation (A/B) for fast rollback of system-wide upgrades.

Strategies evaluated

• Live patching (kpatch, ksplice, Ubuntu Livepatch)
  • Pros: near-zero downtime for many CVEs, can be scripted and deployed quickly.
  • Cons: only modifies existing functions/patch points; cannot change data structures, syscall ABI, core boot code, initramfs, major driver logic, scheduler/core layout, or kernel version.
• Rolling reboots
  • Pros: deterministic; required for kernel-version bumps and incompatible fixes.
  • Cons: short planned outages per host; needs careful orchestration with HA.
• HA & failover
  • Use load balancers, cluster fencing, quorum, maintenance-mode drains to shift traffic before reboots.
• Boot segmentation (dual rootfs / A-B)
  • Enables atomic boots and fast rollback by switching boot entry.

Testing requirements

• Unit/kernel module testbeds + CI build of livepatch modules
• Staging cluster that mirrors prod kernel/config
• Canary waves (1–5 hosts) with automated smoke checks, performance and workload tests
• Long-running soak tests for stateful services

Limitations of live patching

• Cannot add/remove syscalls, change on-disk metadata formats, alter kernel boot path, or update initramfs/firmware-dependent drivers.
• Complex fixes that touch global structs, locking semantics, or scheduling policies often require reboot.

Fallback / rollback plans

• Prefer unload of livepatch (if supported) and revert to prior package; if unload unsafe, failover traffic off host and reboot to known-good kernel.
• A/B boot: set previous slot and reboot.
• Automated snapshots (LVM/ZFS VM snapshots or image-based rollback) and playbooks to reprovision.
• Post-incident: collect crash dumps, enable kdump, run root-cause tests, and delay further rollouts until fixes validated.

This combination keeps uptime high while ensuring safe application of both emergency and planned kernel changes.

Direct answer
An incident is unplanned degradation you did not choose the timing of, and it demands an immediate, ad hoc response. A change is planned work you scheduled, reviewed, and can roll back on your own terms. The core difference is control: with a change you set the clock and the safety net in advance; with an incident, both are decided under pressure, in real time.

Structured elaboration

• Response process. A change follows a pre-agreed plan (a rollout schedule, a rollback procedure written before you started). An incident has no such plan available; the responder is improvising against a runbook at best, from first principles at worst.
• Communication expectations. A change is usually announced in advance ('deploying at 2pm, expect brief latency') and confirmed complete afterward. An incident communication starts reactively ('we're aware of X, investigating') and needs a cadence of updates because nobody agreed to this timing.
• Approvals. A change typically needs sign-off before it happens (a peer review, a change-advisory step for higher-risk changes). An incident response needs no advance approval to act, since delay itself has a cost, though bigger interventions (a full rollback, a customer-facing statement) may still need someone with authority to say go.
• Post-activity documentation. A completed change gets a short record that it happened and worked as intended. An incident gets a fuller review, because the whole point is extracting a lasting lesson from something nobody planned for.

Worked example
A team schedules a canary rollout of a new caching layer for 2pm, reviewed and approved the day before, with an automatic rollback if error rate crosses a threshold. That's a change: planned, approved, monitored against a pre-set safety trigger. At 2:15pm the canary's error rate stays low but an unrelated dependency the caching layer talks to starts timing out, and the on-call engineer gets paged for a 5xx spike unrelated to the rollout. That's now an incident: nobody scheduled it, there's no pre-agreed plan for this specific failure, and the engineer has to improvise triage in real time, even though it happened to start during a change window.

Trade-offs and pitfalls
A common failure mode is applying incident-level ceremony to every change, which breeds change fatigue and makes people route around the process. The opposite failure is worse: not declaring an incident when a change goes wrong, often because the team feels responsible ('we did this to ourselves, let's just quietly fix it') and skips the visibility, communication, and review that an incident would otherwise get. A bad outcome caused by your own planned work is still an incident and deserves the same rigor as one caused by anything else.

Direct answer

Choose dedicated interconnect (a private, physical circuit connecting your site directly into the cloud provider's network, bypassing the public internet), such as AWS Direct Connect, Azure ExpressRoute, or GCP Dedicated Interconnect, for predictable, high-volume, latency-sensitive traffic, and internet-based site-to-site VPN for low-volume or ad-hoc traffic where the fixed cost of a dedicated circuit is not justified. The crossover point is a straightforward cost calculation once both options' actual pricing is known, not a rule of thumb, because dedicated interconnect trades a fixed monthly cost for a lower per-gigabyte rate, and at low enough volume that trade loses.

Structured elaboration

Comparison table

Cost-factor checklist for dedicated interconnect

When pricing an interconnect option, name every line item rather than only the headline port fee: the port allocation cost for reserving the physical port capacity itself, the monthly port charge billed regardless of how much of it is used, the data-transfer or egress rate for traffic actually sent over the circuit, usually discounted relative to standard internet egress but not free, and cross-connect fees charged by the colocation facility for the physical cable connecting your equipment to the cloud provider's, separate from anything the cloud provider itself bills. Missing any one of these line items when estimating cost is the most common reason a dedicated interconnect ends up more expensive than projected.

Worked cost comparison and the crossover point

Using representative, illustrative rates, confirm current list prices with the specific provider before budgeting: an internet VPN's data transfer runs at roughly $0.08 per GB in this example, while a dedicated interconnect carries a $300 monthly port fee plus a discounted $0.02 per GB transfer rate. Setting the two monthly costs equal, 0.08 times GB equals 300 plus 0.02 times GB, gives 0.06 times GB equals 300, so GB equals 5,000, or 5 TB per month. Below 5 TB per month, the VPN is cheaper because the interconnect's fixed port fee has not been earned back yet; above 5 TB per month sustained, the interconnect is cheaper, and the gap widens linearly with volume. The actual decision rule is to compute this crossover with real quoted numbers for the specific volume in question, rather than defaulting to either option by habit.

Three concrete usage scenarios

1. Ad-hoc, low-volume: a quarterly batch export of a few hundred gigabytes to a partner's cloud environment. Internet VPN is the right call, since standing up a dedicated circuit for traffic that runs a few days a quarter would sit almost entirely idle, paying the fixed port fee for capacity that is rarely used.
2. Predictable, high-volume: continuous database replication moving several terabytes per day between an on-prem data center and a cloud region. Dedicated interconnect is the right call, since this is exactly the sustained-volume, latency-sensitive case where the fixed port fee is earned back quickly, well above the 5 TB per month crossover in the example above, and the consistent latency matters for keeping replication lag bounded.
3. Latency-sensitive but moderate-volume: a hybrid application where a subset of transactions must complete within a tight latency budget, but total data volume is moderate. Here the decision is not purely cost-driven; even below the cost crossover point, the SLA and latency predictability of a dedicated interconnect may be worth paying for if a VPN's variable latency risks violating the application's own latency requirement, a case where the guaranteed-performance argument overrides the raw cost comparison.

Worked example

A company evaluates options for a workload projected at 8 TB per month of sustained replication traffic. Over the internet VPN at $0.08 per GB, that is 8,000 times $0.08, or $640 per month. Over the dedicated interconnect, $300 plus 8,000 times $0.02, or $300 plus $160, equals $460 per month, cheaper by $180 per month and widening every month volume grows, on top of the latency and SLA benefits, making the interconnect the clear recommendation for this specific volume.

Trade-offs and pitfalls

The most common mistake is comparing only the headline per-gigabyte rate or only the fixed port fee in isolation, instead of the total monthly cost at the workload's actual projected volume, which the crossover calculation above is built specifically to avoid. A second is choosing internet VPN for a genuinely latency-sensitive workload purely because it is cheaper at the current volume, without weighing scenario 3's point that SLA and predictability sometimes justify the interconnect below the cost crossover. A third is forgetting the cross-connect and port-allocation fees when budgeting an interconnect, only to find the actual monthly bill higher than the headline port-fee-plus-data-rate estimate suggested.

Type 1 vs Type 2 hypervisors — definition & examples

• Type 1 (bare-metal): Runs directly on host hardware, provides better performance and isolation for production workloads. Examples: VMware ESXi, Microsoft Hyper-V (Windows Server Hyper-V).
• Type 2 (hosted): Runs on top of a general-purpose OS; suited for desktop/dev/testing. Examples: Oracle VirtualBox, VMware Workstation.

Three on‑prem operational tasks that differ from managing cloud VMs

1. Host hardware & firmware management

   • I perform BIOS/UEFI updates, vendor firmware/driver and RAID controller tuning, and coordinate maintenance windows for physical servers — tasks not needed for cloud instances.

2. Hypervisor patching, clustering and live-migration

   • I patch ESXi/Hyper‑V hosts, manage maintenance mode, handle vMotion/Live Migration, and maintain HA/DRS clusters to avoid downtime.

3. Storage/network fabric and capacity planning

   • I manage SAN/NFS zoning, LUN mapping, VMFS/datastore growth, NIC teaming and VLANs, and plan physical capacity (CPU/memory/storage) — unlike cloud where many of these are abstracted.

Each task requires hardware-level troubleshooting, vendor coordination, and careful maintenance scheduling to preserve uptime.