DoorDash Staff Systems Administrator Interview Preparation Guide

Direct answer. Ownership has to be a named role tied to the business function or system the runbook covers, not "whoever wrote it last," with versioning that ties each revision to a specific trigger, not just a date. Catching drift before a real event means testing the runbook during a planned exercise instead of discovering it's wrong while executing it under pressure, paired with a lightweight review trigger whenever something the runbook depends on changes.

1. Ownership

Assign a named owner to every runbook, ideally the person or role accountable for the business function or system it covers, not a shared team inbox and not necessarily the original author, who may move on. Ownership means being responsible for the runbook's accuracy, not personally executing every step during a real event. Make it visible on the document itself (name, role, last-reviewed date) so anyone opening it during an incident can see who to ask if something looks wrong, and so a stale "owner" who's left the organization is an obvious red flag rather than a hidden one. When ownership changes hands, require an explicit handoff with a fresh review as part of the transition, not a silent reassignment.

2. Versioning

Treat runbooks as controlled documents: every change gets a version number, a date, who made it, and why, kept in the document's own change history rather than relying on file metadata or institutional memory. Tie versions to triggers, not just a calendar. A scheduled review is a reasonable baseline, but the more reliable trigger is linking review to the events that actually make a runbook wrong: the system it depends on changes, the org structure around it changes, or a real exercise or incident surfaces a gap. A runbook untouched for eighteen months right after the system it describes was rebuilt is far more suspect than one reviewed on schedule six months ago. Keep prior versions accessible, not just the latest, so a post-event review can tell whether a gap that showed up was already known and fixed but hadn't propagated to whoever was executing, or was genuinely new.

3. Catching drift before it matters

The strongest mechanism is using the runbook for real, on a schedule, inside an exercise (a tabletop discussion or a functional exercise that actually executes some steps), before a live event forces the discovery. A runbook nobody has walked through since it was written is a hypothesis, not a tested procedure. Build a lightweight checklist step into whatever process generates the kind of change that breaks runbooks: when a system change, a supplier change, or an org change is being planned, that process flags which runbooks might be affected, so the review happens close to the change instead of being discovered much later. Track a simple staleness signal across the whole runbook set (time since last review, time since the system or team it covers last changed) and report on it the way you'd report any other risk metric, so an accumulating backlog of unreviewed runbooks is visible to whoever owns the continuity program overall, not discovered runbook by runbook.

Worked example

A payments-processing runbook is owned by the payments platform lead, currently at version 4: v2 added a fallback provider, v3 corrected an escalation contact list after a reorg, v4 followed a functional exercise that found the documented rollback step no longer matched how the system actually rolled back. Six months after v4, the team migrates the payments platform to new infrastructure as part of an unrelated project. Because "infrastructure migration for a system with an owned runbook" sits on that project's change-review checklist, the payments platform lead is flagged automatically and confirms the runbook needs an update before the migration ships, catching the drift as part of the planned change rather than during the next incident. At the next scheduled functional exercise three months later, the team walks through the updated runbook end to end and finds one further step, a monitoring dashboard link, still pointing at the decommissioned infrastructure. That becomes a logged finding and v5.

Trade-offs & pitfalls. A purely calendar-based review cadence catches drift too slowly for a fast-changing system and too often for a stable one; tying review to real change triggers is more work to set up but matches effort to actual risk. Ownership without accountability for staleness (a name on a document nobody ever checks) is barely better than no ownership; the staleness signal has to surface to someone, not just exist. And testing a runbook only during a real event is the worst possible time to discover it's wrong: the exercise programme is what lets a team find the gap on an ordinary afternoon instead of during an actual outage.

Direct answer

The attack surface of a configuration repository and its promotion pipeline splits into four zones an attacker could target: the REPOSITORY itself (who can commit, whose commits get merged), the CI RUNNERS that execute pipeline steps (what they can read and reach), the SUPPLY CHAIN of dependencies and base images the pipeline pulls in, and the PROMOTION MECHANISM that actually applies changes to production. A prioritized control set addresses the zone with the highest (likelihood times blast radius) first: branch protection and required review (repository), ephemeral, least-privilege runner credentials (runners), pinned and verified dependencies (supply chain), and a promotion gate that cannot be bypassed by a single compromised identity (promotion mechanism).

Structured elaboration

Repository zone risks and controls. Risk: a compromised or malicious contributor merges a change that exfiltrates secrets or grants themselves broader access, or force-pushes to rewrite history and hide evidence. Controls, roughly ordered by effectiveness per unit of operational cost: branch protection requiring review from someone OTHER than the author (cheap, high effect, stops the single most common path); required status checks (policy-as-code, secret-scanning) that must pass before merge (cheap, catches accidental leaks and known-bad patterns); disallowing force-push and requiring signed commits on protected branches (moderate cost, closes the history-tampering path); a CODEOWNERS-style requirement that changes to the pipeline definition ITSELF need a security or platform-team reviewer, not just any team member (moderate cost, closes the meta-attack of modifying the pipeline to weaken its own controls).

CI runner zone risks and controls. Risk: a compromised runner (via a malicious dependency executed during the build, or a vulnerability in the runner's own environment) reads secrets injected into the job, or pivots to reach other systems the runner's network position allows. Controls: short-lived, scoped credentials issued PER JOB (OIDC-based federation to the cloud provider rather than a long-lived static secret stored in the CI system) so a compromised runner's blast radius is bounded to that one job's narrow scope and a short time window; network-isolate runners so they cannot reach anything beyond what THAT specific job legitimately needs; and treat runner logs as potentially containing secrets, redacting known secret patterns automatically rather than trusting every script to avoid printing them.

Supply chain zone risks and controls. Risk: a malicious or compromised third-party action, base image, or dependency executes arbitrary code during the pipeline run. Controls: pin dependencies to exact versions or content hashes rather than floating tags (actions/checkout@<sha> not @v4), scan and sign container images used as pipeline steps, and restrict which third-party actions/images are allowed at all (an explicit allowlist for anything with write access to secrets or production credentials).

Unauthorized promotions zone risks and controls. Risk: a compromised identity, or a legitimate identity acting outside intended process, triggers a production promotion without the required review. Controls: require the promotion trigger (a merge to a protected production-overlay path, or an explicit approval gate in the deployment tool) to be tied to a REVIEWED PR event, not a webhook or manual trigger any single credential can fire; and log every promotion event with enough context (who/what triggered it, which commit, which approvals) to make an unauthorized promotion immediately detectable even if it cannot be prevented outright.

Worked example

Prioritized by (likelihood times blast radius), for a mid-size organization with an existing but not security-hardened pipeline:

The first four sit at the top because they each close a HIGH-BLAST-RADIUS path (a merged malicious change, a compromised runner with broad reach, an unpinned supply-chain dependency, or a promotion nobody reviewed) at comparatively low implementation cost; signed commits and some of the more process-heavy controls are real but address a narrower slice of risk relative to their rollout cost, so they land lower without being unimportant.

Trade-offs and pitfalls

• Common mistake: treating a required status check as equivalent to a required human review. An automated check catches known patterns; it does not catch a change that is subtly malicious but syntactically clean, which is exactly what human review is for. The two are complementary, not substitutes.
• Ephemeral, per-job credentials are the single highest-leverage control on this list and also the one most often skipped, because migrating away from a long-lived static secret already embedded in a working pipeline feels riskier to touch than leaving it; the actual risk asymmetry runs the other way, a long-lived static secret is a standing target for as long as it exists.
• An allowlist for third-party actions/images needs an owner and a review process for ADDING to it, otherwise it either calcifies (blocking legitimate new tooling and creating pressure to bypass it) or erodes (approvals granted too casually under time pressure, defeating its purpose).
• Common mistake: prioritizing controls by ease of implementation rather than by risk reduction per unit of cost. The cheapest controls to implement are not always the highest-leverage ones; a genuinely prioritized list, as above, has to weigh blast radius explicitly, not just roll out whatever is fastest to configure first.

Direct answer
An incident is unplanned degradation you did not choose the timing of, and it demands an immediate, ad hoc response. A change is planned work you scheduled, reviewed, and can roll back on your own terms. The core difference is control: with a change you set the clock and the safety net in advance; with an incident, both are decided under pressure, in real time.

Structured elaboration

• Response process. A change follows a pre-agreed plan (a rollout schedule, a rollback procedure written before you started). An incident has no such plan available; the responder is improvising against a runbook at best, from first principles at worst.
• Communication expectations. A change is usually announced in advance ('deploying at 2pm, expect brief latency') and confirmed complete afterward. An incident communication starts reactively ('we're aware of X, investigating') and needs a cadence of updates because nobody agreed to this timing.
• Approvals. A change typically needs sign-off before it happens (a peer review, a change-advisory step for higher-risk changes). An incident response needs no advance approval to act, since delay itself has a cost, though bigger interventions (a full rollback, a customer-facing statement) may still need someone with authority to say go.
• Post-activity documentation. A completed change gets a short record that it happened and worked as intended. An incident gets a fuller review, because the whole point is extracting a lasting lesson from something nobody planned for.

Worked example
A team schedules a canary rollout of a new caching layer for 2pm, reviewed and approved the day before, with an automatic rollback if error rate crosses a threshold. That's a change: planned, approved, monitored against a pre-set safety trigger. At 2:15pm the canary's error rate stays low but an unrelated dependency the caching layer talks to starts timing out, and the on-call engineer gets paged for a 5xx spike unrelated to the rollout. That's now an incident: nobody scheduled it, there's no pre-agreed plan for this specific failure, and the engineer has to improvise triage in real time, even though it happened to start during a change window.

Trade-offs and pitfalls
A common failure mode is applying incident-level ceremony to every change, which breeds change fatigue and makes people route around the process. The opposite failure is worse: not declaring an incident when a change goes wrong, often because the team feels responsible ('we did this to ourselves, let's just quietly fix it') and skips the visibility, communication, and review that an incident would otherwise get. A bad outcome caused by your own planned work is still an incident and deserves the same rigor as one caused by anything else.

Direct answer

Capacity planning for a fixed target load is a four-step exercise: measure how much load one instance can safely handle, add a burst/growth buffer to the raw peak, convert that into an instance count, then validate the fleet still meets the target after you lose a zone. The safety margin exists to absorb burstiness, retries, and the cost of running hot for a bounded period, not to compensate for skipping the measurement step.

Structured elaboration

Data to collect

From data to a per-instance capacity number

Run a load test against a single instance (or a fixed-size shard) and raise load until the p99 latency SLO (service level objective: the latency target you've committed to, e.g. p99 under 300ms) breaches, not until CPU hits 100%. That breach point, not the theoretical ceiling, is the instance's safe capacity.

From per-instance capacity to fleet size

1. Planning target = raw peak times a growth/burst buffer.
2. Divide by (safe per-instance capacity times target operating utilization).
3. Round up to whole instances, then round up again to a multiple of the availability-zone count so load balances evenly.

Validate the zone-loss case

After removing one zone's worth of instances, confirm the remaining fleet still covers the raw peak, not the buffered planning target. Running hot for the duration of a single zone outage is an acceptable, bounded trade; running hot while also absorbing organic growth beyond the raw peak is not.

Worked example

A load test on one 4-vCPU instance shows it sustains 1,000 requests/sec at 60% CPU before p99 latency crosses the SLO. That 60% point, not 100%, is the safe ceiling.

Target steady-state operating point: 50% CPU, leaving headroom for GC pauses, noisy neighbors, and the zone-loss case.

Per-instance safe capacity at the 50% target:

Add a 30% burst/growth buffer to the stated 10,000 req/s peak:

Instances needed for the planning target:

Round up to a multiple of 3 availability zones for even spread: 18 instances, 6 per zone.

Validate the zone-loss case. Losing one zone removes 6 instances, leaving 12:

That equals the raw peak exactly: during a single-zone outage the fleet runs at its safe ceiling with zero spare margin, an acceptable, bounded degradation for a rare event but not a state to run in day to day.

Trade-offs & pitfalls

• Over-provisioning for the zone-loss case permanently (running at reduced utilization at all times to cover a rare event) wastes money; treat zone-loss headroom as a temporary, monitored state, not the steady-state target.
• Linear scaling from a single instance's load test breaks down when a bottleneck is shared across the fleet (database connection pool limits, a shared NAT gateway, a shared cache); load-test a small cluster against the real shared dependency, not one box in isolation.
• Retries during partial degradation can multiply effective load two to three times right when capacity is tightest; a plan that ignores retry amplification underestimates the real peak.
• CPU utilization is a proxy, not the constraint. An I/O-bound service can sit at 20% CPU while thread-pool exhaustion still causes timeouts, so the load test's stopping condition should always be the SLO breach, never a resource metric alone.

Overview / Goal
Design a non‑disruptive online schema migration for multi‑TB tables that avoids downtime and minimizes write latency. Focus: change propagation, dual‑write/shadow patterns, backfill, safe cutover, monitoring, rollback.

1) Prep & validation

• Add new nullable columns or use shadow table to avoid blocking DDL.
• Validate schema compatibility, indexes, and query plans in staging using representative data.
• Estimate backfill work (rows/sec), IO, and storage overhead.

2) Change propagation

• Use online DDL tools (example: pt‑online‑schema‑change, gh‑ost) or DB native online alter to apply nonblocking changes.
• If DB lacks safe online DDL, create shadow table (same schema + new columns/indexes).

3) Dual‑write / shadow strategy

• Implement dual‑write at application or middleware layer:
  • Synchronous for a short guarded window only if latency impact is acceptable.
  • Prefer asynchronous fan‑out: write to primary and enqueue change to a worker that writes to shadow table (e.g., Kafka -> consumer).
• Use feature flags to enable/disable dual‑write per service.

4) Backfill mechanism

• Backfill from primary -> shadow using multi‑threaded batch jobs with idempotent upserts.
• Throttle ingestion to keep write latency within SLA; use windowed, chunked queries (pk ranges, timestamp ranges).
• Use consistent snapshot reads to avoid long locks.

5) Safe cutover

• Run parallel reads: route a percentage of reads to shadow/new schema via canary tests.
• Verify data parity with checksums (row counts, sample hashes).
• Gradually increase traffic to new schema; when parity and performance are good, flip feature flag to route writes to new schema only.
• Finalize by removing dual‑write and cleaning up triggers/queues.

6) Monitoring & observability

• Track: replication/backfill progress (rows processed, rows remaining), replication lag, queue depth, write latency (p95/p99), DB CPU/IOPS, error rates.
• Automated alerts for lag > threshold, backfill stall, or error spikes.
• Continuous checksum jobs detect data drift.

7) Rollback considerations

• Keep dual‑write enabled until cutover fully validated to allow instant fallback.
• On failure: disable writes to new schema (flip flag), drain queues, resume primary-only writes.
• Maintain snapshot/backups before major steps; keep shadow data for verification and possible recovery.
• Document and rehearse rollback runbooks and time budgets.

Tradeoffs / Best practices

• Avoid synchronous dual‑write for high‑QPS systems to prevent latency spikes.
• Prefer eventual consistency with strong monitoring and deterministic backfills.
• Test the full flow in staging with production-sized data and runbooks before production execution.

Direct answer

Enforce with token bucket, since bursts are explicitly a requirement, and split enforcement into two tiers: a fast local decision on each edge/LB instance seeded by short-lived leases from a shared store, plus a slower, coarser global check for the hard ceiling. Pure centralized counting gives exact enforcement but adds a network round trip to every request; pure local (fully independent) buckets are fast but overshoot badly once you have many edge instances; leases are the practical middle ground production API gateways actually run.

Token bucket vs. leaky bucket

Both are enforcement algorithms for the same underlying problem, deciding whether to admit or reject a request against a rate limit, but they smooth traffic in opposite ways. Token bucket accumulates capacity (tokens) at a steady refill rate up to a cap, and admits a request as long as a token is available, so unused capacity banks up and can be spent all at once: that banked capacity is exactly what lets a legitimate burst through. Leaky bucket instead models requests (or tokens) as entering a queue and draining out to the backend at a fixed, constant rate, regardless of how bursty the arrivals were, so it smooths traffic into a steady output rate rather than permitting bursts through at all; anything arriving faster than the drain rate either queues, if there's room, or is dropped once the queue is full.

That difference is exactly why token bucket is the right choice here: the question requires allowing legitimate bursts, and leaky bucket has no concept of banked, unused capacity to spend on one, its output is rate-limited to the drain rate no matter how idle the bucket was beforehand. Leaky bucket is the better fit when the goal is protecting a downstream that genuinely cannot tolerate any burst at all, such as a fixed-capacity queue or a downstream with no burst headroom of its own, since it enforces a hard, constant ceiling on the outbound rate. Token bucket is the better fit, and the one used below, whenever some burst tolerance is a feature rather than a bug, which this question states as a requirement.

Design options and their trade-offs

For per-user and global quotas together, use a hierarchy: local, lease-backed enforcement for the fast per-user path, and a separate, coarser global counter that only needs reconciling periodically, not per request, since the global ceiling has more slack per individual request.

Worked example

Assume a per-user quota of $100$ requests/minute and a global quota of $50{,}000$ RPS spread evenly across 3 regions absent per-region traffic data:

Per-user leases: the nominal per-user rate is

Choose a lease refresh interval of $5\text{s}$ (a design choice: short enough to bound overshoot, long enough to keep the allocator's own request rate manageable). Lease size, rounded up so a well-behaved user is never starved mid-window:

If a user's requests can land on up to $N = 3$ edge instances within one lease interval (each holding its own local bucket topped up independently), the extra tokens beyond what a single-instance view would allow is bounded by:

That's at most an extra $18/5 = 3.6$ req/s of burst headroom above the nominal $1.667$ req/s during any single 5-second window, a known and tunable bound (shrink the interval or cap $N$ to tighten it), converging back toward the true 100/min rate as leases keep renewing at the true rate over longer horizons.

For the "approximate counting" framing: a sliding-window counter (used instead of a sliding log at this scale) estimates the count in the trailing window as

where $N_{curr}$ is the count so far in the current fixed sub-window, $N_{prev}$ is the previous sub-window's count, and $f$ is the fraction already elapsed into the current sub-window; this blends two cheap counters into an approximation of the true sliding count without storing individual timestamps.

Architecture

flowchart LR
  Client --> Edge1[Edge LB 1: local bucket]
  Client --> Edge2[Edge LB 2: local bucket]
  Client --> Edge3[Edge LB 3: local bucket]
  Edge1 -->|periodic lease request| Store[Shared lease allocator]
  Edge2 -->|periodic lease request| Store
  Edge3 -->|periodic lease request| Store
  Store -->|reconciles| Global[Global quota counter]

Trade-offs and pitfalls

• Fail-open vs fail-closed when the central store is unreachable is a business decision, not just an engineering default: fail-closed protects backends but turns a rate-limiter outage into a full outage; fail-open protects availability but removes protection exactly when a traffic spike might also be stressing the store. A common middle ground is falling back to the last-known lease/local bucket state for a bounded grace period, then failing open.
• Identity matters before any of this works: decide up front how you key a client (authenticated user ID, API key, or IP for unauthenticated traffic), since IP-keyed limits are coarser and easier to evade behind shared NAT.
• Leases trade a small, bounded overshoot for removing the store from the hot path; if the requirement is "never exceed N, ever" (e.g. a paid third-party API billed per call), leases are the wrong tool and the centralized counter's exactness is worth the latency cost.
• Global quota reconciliation across regions needs its own cadence: checking it per request defeats the purpose of regional autonomy, but checking it too infrequently lets regions collectively exceed the ceiling before anyone notices.
• Approximate sketches suit extreme cardinality (e.g. anti-abuse detection across millions of anonymous IPs) but are a poor fit for quotas you have to justify to a paying customer, since "why was I throttled" needs a precise answer.

Direct answer
A decision genuinely reflects a stated value, rather than merely being compatible with it, when the value actually changed what you chose to do, not just how you described it afterward. The strongest answers make that causal link explicit: what you would have done differently if the value hadn't been a factor.

Structured elaboration

• Situation and task: the decision point, described briefly.
• The counterfactual test: name what the default, easier choice would have been, and what specifically made you choose differently.
• Action: what you actually did, including who you had to convince or coordinate with.
• Result: the outcome, and ideally a signal that the choice was validated rather than merely feeling principled at the time.

Worked example
Faced with a choice between shipping a quick, directionally useful analysis in time for a decision meeting, or spending an additional two weeks on a more rigorous version, the default and professionally "safer" choice would have been to wait for rigor. Choosing to ship the quicker, clearly caveated version instead, because the business decision had a hard deadline and a rigorous-but-late analysis would have been useless, shows a genuine trade-off rather than a reflexive one. The decision was validated when the more rigorous follow-up analysis, completed afterward, confirmed the same direction, meaning the faster call hadn't cost the business a wrong decision.

Trade-offs and pitfalls
A story where the value and the easy choice happen to be the same thing doesn't actually demonstrate anything, since no real trade-off was made; choose a story with genuine tension in it. Naming the value first and building a story to fit it, rather than the reverse, tends to produce something that sounds rationalized rather than genuine; a genuinely reflective answer usually names the counterfactual without being asked. A result stated only as "and it felt right" is weaker than any concrete validation signal, even an imperfect one.

Direct answer
My mentoring approach starts from diagnosing where someone actually is, not applying one fixed style, and it balances technical guidance with career development by treating them as two separate but connected tracks: technical guidance closes the gap between where they are and what the work in front of them needs right now, while career conversations look further out at where they're trying to go. The mix between the two shifts substantially depending on how experienced the person already is.

Structured elaboration

Diagnosing before applying a style

The first move with any new mentee is figuring out their actual starting point and goals, not assuming based on title or tenure. Two people at the same level can need very different things: one might need technical unblocking, another might already be technically strong but stuck on visibility or scope.

Balancing technical guidance and career development

• Technical guidance tends to dominate early in a relationship or when someone's working in genuinely new territory; it's concrete, has fast feedback loops, and builds the trust that makes career conversations land later.
• Career development becomes a larger share of the time as technical competence stabilizes; someone who's already reliable on the day-to-day work benefits more from conversations about scope, visibility, and where they're headed than from more line-by-line guidance.
• The two aren't fully separable in practice: a well-run technical conversation often surfaces the real career question underneath it (they're not struggling with the code, they're struggling with whether this kind of work is even what they want to be doing).

How the approach changes: newer teammate vs. experienced one

• A newer teammate typically needs a tighter structure: explicit expectations, closer review, and a higher ratio of technical to career conversation, because there usually isn't yet a track record to have a grounded career conversation about.
• A more experienced teammate usually needs the opposite ratio: less hands-on technical guidance (often none at all on execution, more on judgment calls and trade-offs), and more time spent on career and scope, sometimes including the expectation that they take on some mentoring of their own, since that's often the actual next step in their growth.

Worked example

Applying the philosophy

With a newer teammate, most of an early 1:1 might genuinely be spent walking through a specific technical decision they made, only pivoting to career topics once they'd built enough of a track record to have something concrete to talk about. With a more experienced teammate on the same team, the same 1:1 slot might be spent almost entirely on a scope or visibility question, with technical guidance limited to a quick sanity check on a hard trade-off they'd already mostly worked out themselves.

Signal of it working

The clearest sign the ratio was right in either case wasn't a specific number, it was whether the conversation actually used the full time productively: a newer teammate's 1:1 running long on technical questions because they had real ones was a good sign; the same happening with an experienced teammate, repeatedly, usually meant something else was being avoided, often a harder career conversation neither of us had opened yet.

Trade-offs & pitfalls

• Applying the same ratio to everyone regardless of experience. A fixed philosophy that doesn't flex by seniority isn't really a philosophy, it's a script, and it under-serves experienced mentees while potentially overwhelming newer ones.
• Letting technical conversations become a permanent default because they're easier. Technical questions have clear right answers and fast feedback; career conversations are ambiguous and can feel uncomfortable. A senior mentor notices when technical talk has become an avoidance pattern rather than what's actually needed.
• Treating career conversations as an occasional add-on rather than a real track. If career development only comes up during formal review cycles, it usually means the day-to-day mentoring relationship isn't actually addressing it.

Approach summary

• Use PowerShell remoting (Invoke-Command / New-PSSession) and Get-Counter on remote hosts
• Throttle concurrency with Invoke-Command -ThrottleLimit or with runspaces for higher scale
• Implement retry logic with try/catch + backoff
• Aggregate results to objects and Export-Csv
• Secure creds via PSCredential from Windows Credential Manager or a secrets store (eg. Azure Key Vault); avoid plain text

Modules / cmdlets

• Microsoft.PowerShell.Core (Invoke-Command, New-PSSession)
• CimCmdlets (Get-CimInstance if needed)
• Diagnostics (Get-Counter)
• CredentialManager or Az.KeyVault for secrets

Pseudo-code / sample

# get credential securely (example: stored credential)
$cred = Get-StoredCredential -Target 'MonitoringAcct'   # from CredentialManager module

$servers = Get-Content servers.txt
$results = @()

foreach ($batch in $servers -split 20) {               # simple batching for throttle
  $jobs = foreach ($s in $batch) {
    Start-Job -ArgumentList $s,$cred -ScriptBlock {
      param($server,$cred)
      $maxRetries = 3; $delay=5
      for ($i=1; $i -le $maxRetries; $i++) {
        try {
          $counters = Get-Counter -ComputerName $server -Counter '\Processor(_Total)\% Processor Time',
                                                           '\Memory\Available MBytes',
                                                           '\PhysicalDisk(_Total)\Avg. Disk Queue Length',
                                                           '\Network Interface(*)\Bytes Total/sec' -ErrorAction Stop
          return [pscustomobject]@{
            Server = $server
            CPU    = ($counters.CounterSamples | where Path -like '*Processor(_Total)*').CookedValue
            MemMB  = ($counters.CounterSamples | where Path -like '*Available MBytes*').CookedValue
            DiskQ  = ($counters.CounterSamples | where Path -like '*Disk Queue Length*').CookedValue
            NetBps = ($counters.CounterSamples | where Path -like '*Bytes Total/sec*').CookedValue
            Timestamp = Get-Date
          }
        } catch {
          if ($i -eq $maxRetries) { return @{Server=$server; Error=$_.Exception.Message} }
          Start-Sleep -Seconds ($delay * $i)
        }
      }
    }
  }
  $jobs | Wait-Job | Receive-Job | ForEach-Object { $results += $_ }
  $jobs | Remove-Job
}

$results | Export-Csv -Path .\perf-aggregate.csv -NoTypeInformation

Security & best practices

• Store service account in Windows Credential Manager, or use Azure Key Vault/HashiCorp Vault and fetch at runtime.
• Limit account privileges to read performance counters only.
• Use HTTPS/WinRM over TLS or JEA constrained endpoints for remote execution.
• Monitor and log failures; alert on repeated retry exhaustion.

Complexity / notes

• Invoke-Command with -ThrottleLimit is simpler for small fleets; runspaces scale better.
• Get-Counter may require firewall/Perf counters enabled; fallback to Get-CimInstance Win32_PerfFormattedData_* if blocked.

Approach — goal & constraints

• Goal: find root cause of intermittent high latency between on‑prem app and cloud service. Work from measurement → isolation → hypothesis testing → remediation.
• Start non‑disruptively, gather distributed telemetry, then perform packet captures and targeted tests during or reproducing incidents.

What to collect (metrics & logs)

• Latency/time-series: application logs, server tcp_connect times, CloudWatch/Azure Monitor latency, host-level iostat/cpu/memory.
• Network metrics: interface errors, drops, bandwidth, queue lengths (ifcfg/ip -s, ethtool), switch counters.
• Flow logs: VPC Flow Logs / NSG Flow Logs, firewall logs (timestamps, 5‑tuple, bytes, actions).
• Routing/BGP: BGP state, route changes, AS path, route origin changes from routers and cloud edge (collect MRT/bgpmon alerts).
• Packet captures: tcpdump on on‑prem server and cloud VM/edge; record during incident windows.
• ICMP/trace tools: traceroute/mtr, path MTU probes.
• Timestamps & correlation: enable NTP/chrony across hosts and record timezone/offset.

Tools & commands (examples)

• latency + path:

mtr -rwzbc 100 <cloud-ip>
traceroute -I <cloud-ip>

• packet capture:

sudo tcpdump -i eth0 host <cloud-ip> and \(tcp or icmp\) -w capture.pcap

• MTU/MSS checks:

ping -M do -s 1472 <cloud-ip>   # IPv4: 1500 - 28 = 1472
# or use tracepath/tracepath6
tracepath <cloud-ip>

• check BGP:

show ip bgp summary
show ip bgp <prefix>

Correlation strategy

• Normalize all telemetry to UTC with NTP-accurate timestamps.
• Use 5‑tuple and TCP sequence/ack numbers to match packets across on‑prem and cloud captures; compare observed RTTs in tcpdump via SYN→SYN/ACK timestamps.
• Map high‑latency windows from app logs to VPC flow logs and router syslogs/BGP updates to spot route flaps or ACL rate‑limiting coincident with spikes.
• Use flow logs to see if packets are egressing through different cloud edge IPs (asymmetric routing).

Hypothesis tests

• MTU/fragmentation:
  • Use ping -M do to find largest non‑fragmenting packet; check for ICMP "Fragmentation required" in captures.
  • In tcpdump look for DF bit set and ICMP unreachable(need frag).
• Asymmetric routing:
  • Capture both ends simultaneously; if forward path latency differs from return, compare traceroute from on‑prem and from cloud VM to on‑prem.
  • Use Paris traceroute to avoid per‑flow load‑balancing artifacts.
• Middlebox/queuing:
  • Run iperf3 TCP and UDP tests across windows to see throughput vs latency patterns.
  • Check interface counters and QoS/queueing config on routers/firewalls.
• BGP/routing instability:
  • Correlate BGP updates and AS path changes with latency spikes; use BGP collector or RPKI/BGPmon feeds.

Decision & remediation

• If MTU: adjust MTU/MPLS settings, enable TCP MSS clamping on edge firewall.
• If asymmetry/load‑balancer issue: pin flows (source port) or adjust load‑balancer settings; work with ISP/cloud to fix transit issue.
• If queuing/congestion: apply QoS, increase capacity, or reroute.

Reporting

• Summarize findings with timeline: timestamps, screenshots of tcpdump excerpts (SYN/SYN-ACK RTTs), traceroutes, BGP update excerpts, and recommended fixes.