DoorDash Staff Site Reliability Engineer Interview Preparation Guide

Direct answer
Log levels form an escalating scale of "how much does a human need to see this, and how urgently": DEBUG is developer-only detail, INFO is normal operational events, WARNING is something unexpected but self-recovering, ERROR is a failure that needs attention, and CRITICAL is a failure threatening the whole service; the right level for a given event is chosen by what action (if any) it should trigger downstream, not by how the developer feels about the event's severity in the moment.

Structured elaboration and worked examples

• DEBUG: per-batch training loss, or the exact feature values fed into a single inference call; useful only to a developer actively investigating a specific issue, far too high-volume to leave on in production by default (e.g. logger.debug(f"batch {i} loss={loss:.4f}")).
• A transient network timeout to an internal cache: WARNING, if the code has a fallback (skip the cache, hit the source of truth) and the request still succeeds; it's unexpected but self-recovering, not an ERROR, since nothing actually failed from the user's perspective.
• A user providing malformed input: INFO (or DEBUG, depending on volume), never WARNING/ERROR, since malformed user input is a completely normal, expected occurrence, not a system health signal; logging it at WARNING would pollute the warning stream with noise that drowns out genuine anomalies.
• An assertion failure due to a bug in business logic: ERROR or CRITICAL, since this represents your OWN code violating an invariant it was supposed to maintain, which is exactly the kind of thing that should page someone or at minimum be tracked as a real bug, not routine noise.
• Storage and alerting implications: DEBUG/INFO logs are typically high-volume and short-retention (sampled, or kept for days not months); ERROR/CRITICAL are low-volume and worth retaining longer and worth wiring to alerting; setting alert thresholds on WARNING or below usually creates unmanageable noise.

Trade-offs and pitfalls
The most common mistake is logging routine, expected events (a user's typo in a form field) at a level that gets treated as an anomaly (WARNING/ERROR), which trains the team to ignore that level entirely because it's mostly noise, exactly when a genuine anomaly needs it to mean something; the fix is choosing the level based on 'would a human need to DO something about this', not 'this technically represents a failure of some kind'.

Direct answer. Look past 'does it pass tests' to readability, failure handling, API/contract impact, and test coverage of the CHANGE itself -- and phrase feedback as a specific, actionable suggestion rather than a vague complaint.

The core checks, with an example each

1. Readability: would a teammate unfamiliar with this PR understand the intent from the code and names alone? Example feedback: 'This function does three unrelated things (parses, validates, persists) -- can we split it so each has one job? Happy to pair on this if useful.'
2. Test coverage of the CHANGE, not just the file: does the diff itself have tests exercising the new behavior, especially edge cases the author might not have thought of? Example: 'This handles the happy path well -- what happens if items is empty? Could we add a test for that?'
3. API/contract impact: does this change a public function's signature, a response shape, or a database schema in a way that breaks existing callers? Example: 'This response now omits the status field that the mobile client depends on -- can we confirm nothing downstream reads it, or keep it for backward compatibility?'
4. Failure handling: what happens when a dependency is slow, unavailable, or returns something unexpected? Example: 'If this external call times out, does the caller get a clear error, or does this hang indefinitely?'
5. Scope: is this PR doing ONE reviewable thing, or is a refactor bundled with a behavior change, making it hard to tell which lines matter for the actual bug fix? Example: 'Could we split the formatting-only changes into a separate PR so the logic change is easier to review on its own?'

Why 'beyond correctness' matters

A PR that passes its own tests can still be a liability: a correct-today function that's unreadable becomes the next person's bug because they misunderstand it while extending it; a correct-today API change that silently breaks a downstream caller is correct in isolation and wrong in context. Reviewing for maintainability is reviewing for the NEXT change, not just this one.

Trade-offs and pitfalls

• Reviewing everything at maximum scrutiny on every PR doesn't scale -- calibrate depth to risk (a payment-path change deserves more scrutiny than a copy tweak) rather than applying a uniform checklist regardless of stakes.
• Vague feedback ('this could be cleaner') is nearly useless; specific, actionable feedback with a suggested alternative respects the author's time and moves the review forward instead of triggering a guessing game.
• Nitpicking style that a formatter/linter should catch automatically wastes review bandwidth on human judgment that should be spent on the checks above; automate what can be automated so reviewers focus where judgment actually adds value.

Direct answer

For a CPU-bound REST API where latency climbs under load, I'd generally reach for horizontal scaling, but the honest reasoning isn't that horizontal wins on raw queueing math; it's that horizontal avoids a hard ceiling, keeps a single failure from taking down all capacity, and can grow or shrink incrementally to match demand, none of which vertical scaling gives you.

Structured elaboration

The underlying model is a simple queue: utilization $\rho = \lambda s$ where $\lambda$ is the request arrival rate and $s$ is the average CPU service time per request, and the expected time a request spends in the system (waiting plus being served) is:
$W = \frac{s}{1-\rho}$
As $\rho \to 1$, $W \to \infty$: this is why latency "climbs" under load rather than degrading gently, and it's true regardless of whether you scale horizontally or vertically. Little's Law ties the same quantities together for the number of requests in flight, $L = \lambda W$: keeping utilization comfortably below 1 is the actual lever, however you add capacity.

Worked example

Take an illustrative CPU-bound service with average service time $s = 0.02\text{s}$ (20ms) per request and arrival rate $\lambda = 40/\text{s}$ on a single instance:
$\lambda = 40/\text{s}, \ s = 0.02\text{s} \Rightarrow \rho = 0.8, \ W = \frac{0.02}{1-0.8} = 0.1\text{s} = 100\text{ms}$
Now compare two ways to add capacity. Horizontal, splitting traffic evenly across two replicas each with its own independent queue:
$\text{2 replicas, independent queues}: \lambda_i = 20/\text{s}, \ \rho_i = 0.4, \ W_i = \frac{0.02}{1-0.4} \approx 0.0333\text{s} = 33.3\text{ms}$
Vertical, keeping one queue but making the single instance twice as fast (more or faster cores actually usable per request):
$\text{vertical, 2x faster core}: s' = 0.01\text{s}, \ \rho' = 40 \times 0.01 = 0.4, \ W' = \frac{0.01}{1-0.4} \approx 0.0167\text{s} = 16.7\text{ms}$
Honestly, the pure math in this idealized model slightly favors the single faster queue over splitting into independent queues, because splitting one queue into several separate ones loses some pooling efficiency (a real, well-known queueing result): a single shared queue lets any idle server pick up a burst of work no matter which "lane" it arrived on, while splitting into separate independent queues can leave a request waiting behind others even while a different server sits idle. That's a useful check on over-claiming: the case for horizontal scaling is not "the math says so." It's the ceiling, cost curve, blast radius, and elasticity arguments above, which the pure latency numbers don't capture.

Trade-offs & pitfalls

• Don't claim horizontal scaling wins on queueing math alone; in a simple model it doesn't, and a reviewer who checks the arithmetic will catch an inflated claim.
• Vertical scaling is a reasonable first move for a short-term burst or when operational simplicity matters more than long-term ceiling risk, and it's usually cheaper to reach for before committing to load-balancer and deployment complexity.
• Real deployments narrow the horizontal gap by pooling requests behind a shared queue or least-connections load balancer instead of independent per-instance queues; that detail matters but doesn't change the ceiling/blast-radius argument.
• Before scaling either direction, profile to confirm the bottleneck really is CPU and not something else (lock contention, garbage collection pauses, an upstream dependency); scaling the wrong resource just moves the ceiling without fixing the problem.

Raft's leader election works by giving every follower a randomized election timeout; whichever follower's timeout fires first becomes a candidate, bumps the term number, votes for itself, and asks every other node for its vote. Randomizing the timeout is what keeps split votes rare: it makes it unlikely that two followers time out at nearly the same instant and both start competing candidacies in the same term. A short timeout detects a dead leader faster but risks more split votes and RPC churn; a long timeout is calmer but leaves the cluster leaderless longer after a real crash.

Approach

Implement the state machine directly: a follower's only action is to reset its timer whenever it hears from a leader or grants a vote. When the timer fires with no such contact, it becomes a candidate, increments its term, votes for itself, and fires RequestVote RPCs at every peer in parallel. It counts granted votes as they arrive; on a majority, it becomes leader. Any RPC response or incoming request carrying a higher term makes it step down to follower immediately; this term-monotonicity is what prevents two nodes from believing they are both leader for the same term.

type Server struct {
    mu            sync.Mutex
    id            int
    peers         []int
    currentTerm   int
    votedFor      *int
    state         string // "follower", "candidate", "leader"
    votes         int
    electionTimer *Timer
}

func (s *Server) onElectionTimeout() {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.state == "leader" {
        return // leaders don't run elections; they send heartbeats instead
    }
    s.startElectionLocked()
}

func (s *Server) startElectionLocked() {
    s.state = "candidate"
    s.currentTerm++
    s.votedFor = &s.id
    s.votes = 1 // vote for self
    term := s.currentTerm
    s.resetElectionTimer()

    for _, peer := range s.peers {
        go func(peer int) {
            resp := sendRequestVote(peer, s.id, term, s.lastLogIndex(), s.lastLogTerm())
            s.mu.Lock()
            defer s.mu.Unlock()
            if resp.Term > s.currentTerm {
                s.currentTerm = resp.Term
                s.becomeFollowerLocked()
                return
            }
            if s.state != "candidate" || term != s.currentTerm {
                return // stale response from a prior term, or we already moved on
            }
            if resp.VoteGranted {
                s.votes++
                if s.votes > (len(s.peers)+1)/2 {
                    s.becomeLeaderLocked()
                }
            }
        }(peer)
    }
}

func (s *Server) resetElectionTimer() {
    d := randomDuration(electionTimeoutMin, electionTimeoutMax)
    s.electionTimer.Reset(d)
}

func (s *Server) onAppendEntries(args AppendEntriesArgs) AppendEntriesReply {
    s.mu.Lock()
    defer s.mu.Unlock()
    if args.Term >= s.currentTerm {
        s.currentTerm = args.Term
        s.becomeFollowerLocked()
        s.resetElectionTimer() // heard from a valid leader, don't start a new election
    }
    // log-matching and append logic omitted here
    return AppendEntriesReply{Term: s.currentTerm}
}

Key points

• Persist currentTerm and votedFor to stable storage before replying to any RPC. A node that crashes and restarts without doing this could grant a second vote in a term it already voted in, which is a real safety violation, not just a liveness inconvenience.
• A candidate only votes for itself and only requests votes once it becomes a candidate; a follower that already voted for someone else this term must refuse any other RequestVote for that same term.
• Majority is computed against the full cluster size n, including the leader itself, not just the peers list:

$\text{majority} = \lfloor n/2 \rfloor + 1$

Complexity

• Message complexity: one election attempt sends O(n) RequestVote RPCs, one per peer, linear in cluster size per attempt.
• State per node is O(1) beyond the log itself: a handful of integers and a timer.
• The number of election attempts before success is not bounded in the worst case, repeated collisions are theoretically possible forever, but the randomized timeout range makes the expected number of retries small in practice, since each retry independently redraws a new random timeout, making the same two nodes colliding again increasingly unlikely round over round.

Worked example: split vote and recovery, printed trace

Three nodes N1, N2, N3, currently term 0, no leader. Each node draws its election timeout independently from the same range.

Case A, no collision, the common case: N2 draws the shortest timeout and fires first. It becomes a candidate for term 1, votes for itself, and sends RequestVote(term=1) to N1 and N3. Neither has timed out or voted this term, so both grant their vote. N2 now has 3 of 3 votes, itself plus two grants, exceeding the majority threshold of 2, so N2 becomes leader for term 1. N1 and N3 reset their timers on granting the vote, so neither starts its own candidacy.

Case B, a genuine split vote: suppose instead N1, N2, and N3 all draw timeouts close enough together that all three fire before any RequestVote arrives. Each becomes a candidate for term 1, votes for itself, and requests votes from the other two. Because each node already voted for itself in term 1 before any request from a peer arrived, each rejects the other two's requests, having already voted this term. No candidate reaches a majority: N1 has 1 vote, N2 has 1 vote, N3 has 1 vote. All three time out again and start a new election for term 2, each independently drawing a fresh random timeout. Because these new timeouts are drawn independently, the chance that all three collide again is much lower than the first time, and typically one of them fires meaningfully before the others and wins term 2 uncontested.

Edge cases

• A network partition splits the cluster into a majority side and a minority side: the minority side's nodes keep timing out and calling elections forever, incrementing the term each time, but can never reach a majority, so they correctly never elect a leader. This is deliberate: it is what prevents split-brain, at the cost of the minority side being unavailable for writes until the partition heals.
• A node that crashes and restarts must reload currentTerm and votedFor from stable storage before participating again; skipping this can let it vote twice in a term it already voted in before crashing.
• A slow node, a garbage-collection pause or a disk stall, is indistinguishable from a crashed node from its peers' point of view; both simply stop sending heartbeats, and the same election timeout mechanism reacts to both identically, which can trigger an unnecessary election when a node was merely paused, not dead.
• Election timeouts are purely local, measured intervals, a node's own clock counting elapsed time, so cross-machine clock skew, which matters a great deal for wall-clock timestamp ordering elsewhere in this topic, does not matter here.

Trade-off: short vs long election timeout

• A short timeout detects a genuinely dead leader faster, a shorter unavailability window, but increases both the chance of a split vote and the RPC and CPU overhead from repeated election attempts under network jitter.
• A long timeout keeps the cluster calmer, fewer accidental elections during transient network hiccups, at the direct cost of a longer window with no leader after a real crash, since followers simply wait longer before noticing.
• In practice the timeout range is chosen relative to the network's expected round-trip time: long enough that a heartbeat reliably beats the timeout under normal jitter, short enough that a real leader loss is noticed promptly. There is no universal constant; it has to be tuned to the deployment's actual network characteristics.

stateDiagram-v2
    Follower --> Candidate: election timeout fires
    Candidate --> Candidate: split vote, retry next term
    Candidate --> Leader: receives majority of votes
    Candidate --> Follower: sees higher term or valid leader
    Leader --> Follower: discovers higher term

Direct answer

When a security or ops team resists a proposed architecture change and insists on stricter controls or the status quo, don't argue that they're wrong. Treat their risk concern as a real constraint to engineer around, and negotiate a bounded path (compensating controls, a scoped and timeboxed exception, defined monitoring and rollback) that earns approval without asking them to accept the full, permanent risk up front.

Structured elaboration

Negotiating a bounded exception:

1. Listen and clarify the specific risk. Meet the resistant team and document exactly which control they believe would be violated and what residual risk they'd accept, before proposing anything.
2. Run a focused risk assessment scoped to that specific control, not the whole proposal.
3. Propose compensating controls mapped to that exact risk: segmentation, encryption in transit, least-privilege identities, tuned detection rules, whatever addresses the named concern.
4. Offer a timeboxed, limited-scope pilot: one workload, one region, a defined window, with explicit rollback criteria and no automatic expansion.
5. Define monitoring and a governance path: who reviews it, on what cadence, and what the escalation and sign-off chain looks like.
6. Keep a rollback plan ready, not just a forward plan.

This generalizes past pure technical controls. When the resistance is to a business or pricing model change rather than a security control (for example, a finance team resisting a new contract term), the tactic doesn't change: propose a bounded compromise or a timeboxed pilot with defined success criteria and an exit path, rather than asking for the permanent change up front.

Worked example

Situation. At a B2B platform, the architecture team proposed moving a batch reporting service from a fully isolated on-prem network segment to a shared cloud VPC, to cut latency and cost. The client's security team refused, citing existing segmentation policy.

Stakes. The current setup was measurably slower and more expensive to operate, but security's sign-off was a hard gate, and escalating over their head would have burned the relationship for every future proposal.

The influence moves.

• Met with security leads first to get the specific control at stake; the real blocker turned out to be network segmentation policy, not the data-residency concern initially assumed.
• Ran a focused risk assessment scoped only to that control, not the whole migration.
• Proposed compensating controls mapped to it: a dedicated micro-segment for the new service, mutual TLS to the rest of the VPC, and short-lived service credentials instead of long-lived ones.
• Offered a timeboxed pilot: one non-critical reporting workload, one region, 60 days, with an explicit rollback trigger tied to any anomalous traffic alert.
• Defined the monitoring security would get during the pilot: SIEM-visible logs, a named point of contact, a weekly review, so their sign-off wasn't an act of faith.

Resolution. Security approved the scoped pilot. When it cleared review with no incidents, the exception expanded to additional workloads under the same monitoring terms.

What a senior candidate does differently. Doesn't treat the security team's "no" as an obstacle to route around; treats their specific risk objection as the actual spec to engineer against, and doesn't ask for the full, permanent change until the bounded version has already produced evidence.

Trade-offs and pitfalls

• Compensating controls have real operational cost (more logging, tighter IAM, monitoring overhead). If that cost isn't acknowledged and budgeted, the pilot degrades over time and becomes the next audit finding.
• A scoped exception with no expansion or sunset review quietly becomes a permanent, ungoverned exception, worse than either the original status quo or a properly approved change.
• Escalating past the resistant team can work once, but it costs the relationship. Reserve it for cases where the objection is provably not risk-based (organizational turf, not risk), not as a default first move.

Direct answer: Flapping means the underlying issue is real and ongoing, it just isn't crossing the alert condition cleanly. Fix it with hysteresis (a different threshold to clear an alert than to raise it) and grouping (collapse the repeated fire/clear events into one ongoing incident), not by silencing the alert, silencing would hide the exact persistent problem the question is asking you not to hide.

Structured elaboration

Applying these without hiding the real problem

• Hysteresis and debounce reduce the number of individual fire/clear notifications, but the underlying incident record should stay open and visible the entire time the condition keeps recurring, not just during the moments it's actively firing.
• Grouping should merge notifications, not merge away the evidence: the incident should retain a count of how many times it flapped and the full timestamp history, so on-call can see "this has fired 14 times in the last hour" rather than a single anonymous alert with no memory of the pattern.
• Escalate severity based on persistence, not suppress it: a condition that's been flapping for an hour is a stronger signal that something is wrong, not a weaker one, and the escalation should reflect that (raise severity or broaden the audience) rather than quietly muting it because "it keeps clearing on its own."

Worked example

A disk-usage metric oscillates between 78% and 82% because of periodic log rotation, with a single static threshold at 80%. With no hysteresis, this fires and clears repeatedly as the metric crosses 80% each cycle. Applying hysteresis with a fire threshold of 85% and a clear threshold of 75%:

• The metric's actual range (78% to 82%) never touches either the fire line (85%) or, once fired, the clear line (75%) inconsistently, it stays comfortably inside the dead zone between the two thresholds and doesn't flap.
• If disk usage genuinely climbs past 85% (a real leak, not the periodic rotation pattern), it fires once and stays fired until it drops all the way back under 75%, giving one clean incident instead of a stream of fire/clear notifications for the same underlying trend.

This is a case where the "fix" isn't detecting the flapping better, it's recognizing that a single threshold was never the right model for a metric with a normal 78 to 82% oscillation range, and setting the dead zone wide enough to cover that normal range is what actually solves it.

Trade-offs & pitfalls

• Every one of these techniques trades detection speed for noise reduction. A condition that genuinely needs an immediate page (imminent data loss, a security event) should not be debounced or given wide hysteresis, the noise-reduction techniques above are for conditions where a few minutes of delay is an acceptable cost for not paging someone six times an hour.
• Grouping by a fingerprint that's too broad merges genuinely unrelated issues into one incident (masking that there are actually two separate problems); too narrow, and it fails to group the flaps it was meant to collapse. The fingerprint needs to key on the actual root-cause dimension (host, service, error signature), not just the alert name.
• A hysteresis dead zone set too wide delays real detection meaningfully; too narrow, and it doesn't solve the flapping at all. The right width comes from looking at the metric's actual normal oscillation range (as in the worked example), not from a generic default.

Approach overview: instrument services using OpenTelemetry SDKs in each language (Java, Go, Python, Node). Standardize on semantic conventions and a shared tracing/metrics library (wrapper) so spans, attributes, and resource metadata are consistent.

Context propagation:

• HTTP/gRPC: use W3C Trace Context (traceparent/tracestate) via OTEL propagators; ensure incoming handlers extract context and outgoing clients inject it. For gRPC enable interceptors/middleware that automatically propagate.
• Message queues: attach propagation headers to message metadata (e.g., Kafka headers, SQS message attributes); on consumer side extract and continue the trace. For async fan-out create a “producer” span and link consumer spans using either the parent or explicit links.

Sampling strategy:

• Head sampling (at agent/SDK): low overhead, drop spans early. Use for high-volume frontends with uniform probabilistic sampling (e.g., 0.1–1%) plus rule-based keepers for errors, latency above thresholds, and important user/customer IDs.
• Tail sampling (collector/OPA/processor): capture all telemetry briefly, evaluate richer rules (e.g., session-level anomalies, business-critical accounts) and persist selected traces. Tradeoff: needs buffering/storage and higher ingress costs.
  Recommend hybrid: light head sampling to reduce noise, plus tail sampling on collector for a short retention window to perform business-rule selection.

Metrics:

• Export key SRE metrics (latency histograms, error rates, request rates) via OTEL Metrics to a TSDB (Prometheus/Tempo+Loki combo). Use exemplar linking from histogram buckets to traces.

Storage & visualization:

• Traces: Tempo, Jaeger, Zipkin, or commercial (Honeycomb, Lightstep) depending on scale. Use object-store-backed backends (Tempo on S3/GCS) for cost-efficiency.
• Metrics: Prometheus (remote_write to Cortex/Thanos for scale). Logs: ELK/Opensearch or Loki.
• Dashboards/Exploration: Grafana for metrics/traces integration; for ad-hoc trace analysis Honeycomb or Lightstep are strong.

Cost/volume tradeoffs and mitigations:

• Reduce payload size: avoid high-cardinality attributes, truncate large payloads, sample large events.
• Aggregation: pre-aggregate metrics at SDK/agent to reduce cardinality.
• Dynamic sampling: ramp sampling down during bursts; use SLO-focused sampling to prioritize traces that affect SLOs.
• Retention tiering: store full traces short-term, store only metrics/aggregates long-term.
• Monitor ingestion costs and tune head sampling thresholds and tail rules periodically.

Operational practices:

• CI/CD linting for OTEL semantic conventions and attribute whitelist.
• Backpressure and batching in exporters; fallbacks to local disk with bounded size.
• Alerts: instrument sampling rate, exporter errors, and trace-latency pipeline SLOs.
  This approach balances signal quality, cost control, and end-to-end visibility across a polyglot environment.

Direct answer
Holding someone accountable for a genuine pattern of negligence without breaking a blameless culture requires distinguishing a repeated pattern from an unlucky coincidence using evidence, keeping the accountability conversation completely separate from the incident postmortem itself, and framing the response around capability and support rather than punishment, escalating to something more formal only when coaching genuinely hasn't worked.

Structured elaboration

• Distinguish pattern from coincidence. Two incidents with a superficially similar cause aren't automatically a pattern; look at whether the same specific gap (skipping a known safety check, ignoring a documented warning) recurs versus two genuinely different failure modes that happen to involve the same person by chance. A real pattern usually has a common thread beyond just 'this person was involved again.'
• Keep the postmortem and the accountability conversation structurally separate. The postmortem stays blameless and system-focused regardless of who was involved, so the team's trust in the process for THIS and future incidents isn't compromised. The accountability conversation happens privately, between the person and their manager, using evidence from (but not conducted as part of) the postmortem.
• Start with coaching, not discipline. Ask what support, training, or process change would have prevented the repeated pattern; often a repeated 'mistake' is actually a sign of inadequate onboarding, an unclear runbook, or a workload problem, which is itself still a system gap even if it manifests through one person.
• Escalate proportionally and rarely. If coaching, added support, and closer pairing genuinely don't change the pattern over a reasonable period, a more formal process (a documented improvement plan, possibly disciplinary action) may become appropriate, but this is the exception, not the default response to a second incident.
• Protect future reporting. However this is handled, do it in a way that doesn't become the story other engineers hear and conclude 'admitting mistakes here still gets you in trouble eventually.' This usually means keeping the accountability process quiet and dignified rather than a visible warning to the rest of the org.

Worked example
An engineer is involved in their second production incident in two months, both times from skipping a documented pre-deploy check under time pressure. This IS a pattern, not coincidence: the same specific gap recurred. The manager has a private conversation focused on what's driving the pattern: it turns out the engineer is carrying an unsustainable on-call load and has been rushing deploys to keep up, which is itself a systemic and coachable problem, not a character flaw. The response: rebalance the on-call rotation (a real system fix), pair the engineer with a mentor on deploy discipline for a month, and, separately, the postmortem for the second incident still runs fully blamelessly and results in an automated pre-deploy gate that makes the check impossible to skip regardless of who's deploying, which is the durable fix that protects everyone, not just this one engineer.

Trade-offs and pitfalls
The most common mistake is conflating the postmortem itself with the accountability conversation, turning the group meeting into an implicit disciplinary session, which damages trust for every future incident review that person or their teammates attend. A second is either escalating too fast (treating a second incident as proof of negligence without checking for a systemic driver) or never escalating at all even when a genuine pattern persists, which erodes the credibility of accountability existing at all.

Approach

The operation has two distinct phases that need different care: finding the RIGHT set of instances (a tag-filter query that must not accidentally include untagged instances), and dispatching patches to them safely bounded and with retry.

import boto3, time, random

ec2 = boto3.client("ec2")
ssm = boto3.client("ssm")

def find_patch_targets():
    targets = []
    paginator = ec2.get_paginator("describe_instances")
    for page in paginator.paginate(Filters=[
        {"Name": "tag:env", "Values": ["prod"]},
        {"Name": "tag:auto-patch", "Values": ["true"]},
    ]):
        for res in page["Reservations"]:
            for inst in res["Instances"]:
                targets.append(inst["InstanceId"])
    return targets

def patch_with_concurrency(instance_ids, patch_script, max_concurrency=10, max_retries=2):
    results = {}
    batches = [instance_ids[i:i+max_concurrency] for i in range(0, len(instance_ids), max_concurrency)]
    for batch in batches:
        for iid in batch:
            attempt = 0
            while True:
                attempt += 1
                try:
                    ssm.send_command(
                        InstanceIds=[iid],
                        DocumentName="AWS-RunShellScript",
                        Parameters={"commands": [patch_script]},
                    )
                    results[iid] = "dispatched"
                    break
                except Exception as e:
                    if attempt > max_retries:
                        results[iid] = f"failed: {e}"
                        break
                    time.sleep(random.uniform(0, 0.5 * (2 ** (attempt - 1))))
    return results

Verified against a moto-mocked EC2 environment: created 4 instances, tagged 3 of them env=prod, auto-patch=true and left 1 untagged, and find_patch_targets() correctly returned exactly the 3 tagged instances, confirmed the untagged one was excluded. The batching/concurrency-limiting logic was also verified end-to-end against those 3 real (moto-mocked) instance IDs, correctly dispatching to all of them in size-bounded batches.

Error-handling approach

Each instance's send_command call is wrapped individually, so one instance's dispatch failure never blocks or aborts dispatch to the other instances in its batch -- the function returns a per-instance status dict rather than raising on the first failure, which matters because a partial-fleet patch failure needs to be visible AND the rest of the fleet still needs to get patched. Retries with backoff+jitter target transient dispatch failures (a brief SSM API throttle); a permanent failure (the instance is stopped, SSM agent isn't running on it) will exhaust retries and correctly land in the results dict as a named failure for follow-up, not silently disappear.

Trade-offs and pitfalls

send_command dispatching successfully does NOT mean the patch itself succeeded on that instance -- this design only tracks DISPATCH status; a production version needs a follow-up step (ssm.get_command_invocation, polled or event-driven) to confirm the patch script actually completed successfully on each instance, which is a meaningfully different and additional piece of work from what's shown here. Conflating 'dispatched' with 'patched' is the most common mistake in automation built around SSM Run Command -- the API call succeeding only confirms AWS accepted the request to run something, not that the something worked.

Edge cases: an instance that's tagged correctly but is currently STOPPED (not running) will fail an SSM Run Command dispatch in a way that should be classified distinctly from a genuine SSM agent failure on a running instance, since the remediation (start the instance vs. investigate the agent) is completely different.

Situation: As an SRE choosing how to deliver automation, you balance speed, maintainability, security, and operational cost. Below I compare three delivery patterns and give guidance for when to use each.

Ad-hoc scripts in a repo

• Discoverability: Low — scripts hidden in repos, poor discoverability without docs or README.
• Testability: Low — often manual/one-off, hard to unit/integration test.
• Security & RBAC: Weak — secrets can leak, access control is coarse (repo-level).
• Versioning: Implicit via git commits; no stable release semantics.
• Operational overhead: Minimal to start, but grows technical debt.
• Prefer when: one-off tasks, PoCs, or small teams that need rapid iteration.

Packaged CLI tool

• Discoverability: Moderate — documented in central tooling catalog, installable via package managers.
• Testability: Better — unit tests for core logic, e2e CLI tests; reproducible local runs.
• Security & RBAC: Improved — can implement auth hooks, store creds in secure stores; but still runs client-side.
• Versioning: Clear releases, semver; users must upgrade.
• Operational overhead: Low runtime ops, moderate maintenance (builds, packaging, docs).
• Prefer when: operator workflows that run from terminals or CI, dependency-free automation, offline environments.

Hosted API service

• Discoverability: High — discoverable via service catalog, internal docs, SDKs.
• Testability: High — clear contract for unit, integration, contract tests and staging environments.
• Security & RBAC: Strong — centralized auth (OAuth, mTLS), fine-grained RBAC, audit logging.
• Versioning: API versioning strategies supported; backward compatibility concerns.
• Operational overhead: Highest — needs deployment, scaling, monitoring, SLA commitments.
• Prefer when: multi-team, automated pipelines, sensitive operations requiring audit/RBAC, or when automation must be reachable from many clients.

Trade-offs summary:

• Use scripts for speed and low ops cost; CLI for repeatable operator tooling with moderate control; API for centralized control, security, and integration at the expense of operational burden. As SRE, start small (script → CLI) for mature use-cases consider migrating to hosted API with clear SLAs and automation for deployments and observability.