FAANG-Standard Interview Preparation Guide: Mid-Level DevOps Engineer

Direct answer
A progressive-delivery ramp for a payment service needs the automation to actively gate each stage's advance on real metric checks, not just wait out a timer, and the partial-rollback plan for the 50% stage needs to distinguish cleanly between requests that already went through the new code (which may have real side effects, like a payment already processed) and requests still ahead of the rollback taking effect.

Structured elaboration

• 1% canary: the smallest, most cautious stage, watched closely with a shorter observation window since the blast radius is tiny; metric checks focus on error rate and latency deltas against the stable baseline, plus a payment-specific correctness signal (successful-transaction rate, any reconciliation mismatch) since a payment service's most dangerous bugs may not show up as a raw HTTP error at all.
• Ramp to 50% over two hours if clean: this isn't a single jump, it's itself a staged ramp (say 1% -> 10% -> 25% -> 50%, each requiring its own clean metric window before advancing), automated so a human doesn't have to manually approve every micro-step, but with metric checks gating EVERY step, not just the final 50% checkpoint.
• 100% after 24 hours: a long hold at 50% specifically to accumulate enough transaction volume and TIME (payment issues can be slow-building, like a subtle reconciliation drift that only shows up after a batch settlement process runs) before committing to full exposure.
• Partial rollback if problems appear at 50%: reduce the new version's traffic share back down (not necessarily to zero immediately, potentially stepping back to a smaller, still-nonzero percentage to keep gathering diagnostic data on a contained population while you investigate), while the ALREADY-PROCESSED transactions on the new code path need their own review: were any payments processed incorrectly, and do they need a compensating action (a reversal, a manual reconciliation) distinct from the traffic-routing rollback itself?

Worked example
At the 50% stage, an automated check flags a reconciliation discrepancy in a batch of transactions processed by the new code. The traffic-routing rollback (scaling the new version's share back to 5%, not necessarily zero, to preserve some live diagnostic signal) happens within minutes via the automated pipeline. Separately and on a different timeline, a manual reconciliation process reviews every transaction that went through the new code path during its exposure window to determine whether any need a compensating correction, since simply routing future traffic away doesn't undo whatever the already-processed transactions did.

Trade-offs and pitfalls
Payment services are the canonical example of where "roll back the traffic" and "the problem is fixed" are NOT the same thing, since money may have already moved; the automation needs to be scoped clearly to what it CAN fix (stop MORE transactions from hitting the bad path) while explicitly flagging what it can't (undo transactions that already happened), which needs a human-driven reconciliation process rather than being folded into the automated rollback itself.

Framing

This is a classic space-versus-time trade-off: precompute-and-cache pays a fixed memory cost to hold results ready, so every read is cheap, while compute-on-demand pays a repeated CPU cost every single time a result is needed but uses no extra memory. Which one is cheaper in dollars depends on how many times a cached value gets reused before it needs recomputing, weighed against cloud pricing for memory (billed per GB, whether or not it is being read right now) against vCPU time (billed roughly per unit of work actually done).

Worked example (one basis: illustrative cloud rates)

Say each ranking feature vector is 1 KB, and there are 10 million items, so the full cached set is about 10 GB. At an illustrative memory-optimized rate of $0.01 per GB-hour, holding that cache resident costs about $0.10/hour, regardless of how many times it gets read.

Computing a feature on demand instead takes, say, 2 milliseconds of CPU time. At an illustrative $0.04 per vCPU-hour, a single computation costs about $0.0000000222 (2ms is 2/3,600,000 of an hour, times $0.04). That looks tiny, but it is paid on every read, while the cache cost is paid once regardless of read volume.

Where the crossover is

Setting the two costs equal and solving for the read rate (queries per second, QPS) where they break even: at this compute cost per item, on-demand computation only matches the flat $0.10/hour cache cost at around 1,250 QPS sustained:

QPS x 3,600 sec/hour x $0.0000000222/computation = $0.10/hour
QPS = $0.10 / (3,600 x $0.0000000222) ~= 1,250
``` Below that read rate, compute-on-demand is cheaper, since you are not paying to hold memory mostly idle. Above it, caching is cheaper, since you are avoiding repeated compute on the same value over and over.

**How I'd measure and decide**

- Measure the real request rate against this feature, and measure the real per-item compute cost by timing the actual feature computation, not an estimate.
- Account for staleness: caching only works if the feature does not need to be freshly recomputed on every read, so factor in how often the underlying data changes and how expensive it is to refresh the cache versus how expensive it is to just always compute fresh.
- Recompute the crossover QPS whenever cloud pricing or the feature's compute cost changes, since both are exactly the two numbers the decision hinges on, and a hard-coded decision quietly goes stale.

Direct answer

Every row in a shared-responsibility matrix is answering the same question at a different layer: who has to act if something at that layer breaks, needs patching, or gets misconfigured? As you move from IaaS to PaaS to SaaS, the line of provider ownership climbs higher, and the one row that never fully crosses over is data: you always decide what your data means and who can see it, even when the provider secures every technical layer underneath it.

The matrix

Networking and virtualization are provider-owned at every tier: you never touch the physical switches, cables, or the hypervisor that slices one machine into many virtual ones. OS, runtime and middleware are where the boundary actually moves, crossing over entirely between IaaS and PaaS. Application crosses over last, between PaaS and SaaS. Data is the one row that stays with the customer across all three models, because ownership of meaning and access is not a technical layer the provider can absorb.

A genuine gray area: virtual networking configuration

"Networking" reads as cleanly provider-owned in the table above, and for the physical network fabric it is. But the moment you get to the virtual networking artifacts sitting on top of that fabric, such as subnets, route tables, security groups and network access control lists (NACLs), responsibility becomes shared in practice even under IaaS: the provider builds the tooling and guarantees isolation between tenants, but you configure which ports are open and which resources can reach the internet. This is exactly the boundary cloud providers describe as "security of the cloud" (the provider's job) versus "security in the cloud" (yours). It is a genuine gray area because a customer who assumes "networking is the provider's row" and never reviews their own security group rules is the single most common real-world path to an accidental public data exposure, even though the table technically marks networking as provider-owned.

A second, smaller gray area worth naming: under PaaS, the provider's automatic OS and runtime upgrades can silently change application behavior (a language runtime minor version bump, for instance). The table says "provider owns runtime," but if that upgrade breaks your application, you are the one who has to notice and fix it, which is a shared failure mode the matrix alone doesn't capture.

Trade-offs and pitfalls

The matrix is a useful starting point, but it is easy to over-trust as a complete answer. Identity and access management (IAM), the system that controls who can do what, cuts across every layer and every model: a customer's own IAM misconfiguration (an over-privileged access key, for example) can undermine security regardless of which service model is in play, and no single row in the table captures that cross-cutting risk. Treat the matrix as the starting framework for a conversation about responsibility, not as a checklist that, once satisfied, guarantees safety.

kube-scheduler places a pod in two passes: filtering, which throws out every node that cannot possibly run the pod, and scoring, which ranks the survivors and picks the best one. Since Kubernetes 1.19 this two-pass model is implemented as the scheduling framework, a set of extension points (PreFilter, Filter, PreScore, Score, Reserve, Permit, Bind, and others) that built-in and custom plugins hook into; the older "predicates and priorities" terminology describes the same two passes from before the framework existed.

What the filter and score passes check

• Filter (hard constraints): available CPU/memory versus the pod's resource requests, node taints against the pod's tolerations, nodeSelector and nodeAffinity (required rules), volume topology and attach limits, and any PodAntiAffinity or topology spread constraint marked as required.
• Score (soft preferences): bin-packing versus spreading, preferred node affinity weights, image locality (a node that already has the image scores higher), and inter-pod affinity/anti-affinity preferences.

Taints, tolerations, and the two affinity mechanisms

A taint on a node (kubectl taint nodes node1 key=value:NoSchedule) repels pods unless they carry a matching toleration; this is how you reserve nodes (for example GPU nodes, or nodes mid-drain, which Kubernetes taints with node.kubernetes.io/unschedulable) for only the workloads that opt in. nodeSelector is a flat, exact-match label requirement: a pod either has a node with all the listed labels or it doesn't. nodeAffinity is strictly more expressive: it supports operators beyond equality (In, NotIn, Exists, Gt, Lt), and it splits into requiredDuringSchedulingIgnoredDuringExecution (a hard filter, same as nodeSelector but richer) and preferredDuringSchedulingIgnoredDuringExecution (a soft, weighted scoring input). Use nodeSelector for a simple single-label requirement and nodeAffinity once you need "any of these labels" logic or a graceful preference rather than a hard rule.

Preemption

If no node passes the filter pass for a pod, and that pod has a higher PriorityClass than pods already running, the scheduler can evict (preempt) lower-priority pods on a candidate node to make room, then schedule the higher-priority pod there. Preemption is deliberately a last resort: it only runs after normal scheduling fails, and it respects each evicted pod's PodDisruptionBudget (PDB, an object that caps how many replicas of a workload can be voluntarily disrupted at once) where possible, though a sufficiently high-priority pod can still violate a PDB if there is no other way to place it.

Debugging checklist for a pod stuck Pending

Start from the pod's own events, which is where the scheduler reports why it failed, then work outward:

$ kubectl describe pod checkout-7d9f6c-abcde -n prod
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  85s   default-scheduler   0/12 nodes are available: 4 Insufficient cpu,
                                                        3 node(s) didn't match Pod's node affinity/selector,
                                                        5 node(s) had untolerated taint {dedicated: gpu}.

That single event line already narrows the search: this pod failed on three distinct filters across the fleet. The checklist:

1. kubectl describe pod and read the FailedScheduling event message: it names the exact filters that eliminated nodes, and how many nodes each filter eliminated.
2. Compare requested CPU/memory against actual free capacity: kubectl describe nodes | grep -A5 Allocated or kubectl top nodes, and check whether requests (not limits) are what's exhausting the cluster.
3. Check the pod's nodeSelector / nodeAffinity labels against kubectl get nodes --show-labels for a typo or a label that no longer exists on any node.
4. Check node taints (kubectl describe node <n> under Taints:) against the pod's tolerations.
5. Check PersistentVolumeClaim (PVC) binding status if the pod mounts a volume (kubectl get pvc -n <ns>); an unbound claim or a StorageClass with no available capacity in the pod's zone blocks scheduling too.
6. Check PodAntiAffinity and topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule for an over-constrained placement rule (common after adding a third replica to a two-zone cluster).
7. Check whether the pod's PriorityClass is high enough that you'd expect preemption, and if so, why it didn't fire (often: the only feasible node candidates were also excluded by a hard filter, so there is nothing to preempt into).
8. If everything above looks clean, raise kube-scheduler's log verbosity (-v=10 on the scheduler component, or the equivalent in your managed offering) to see the plugin-by-plugin filter trace for that pod.
9. In a multi-scheduler setup, confirm the pod's spec.schedulerName actually matches the scheduler you're inspecting.

At-scale scheduler tuning

On very large clusters, kube-scheduler does not necessarily score every feasible node before choosing one; percentageOfNodesToScore in the scheduler configuration caps how many nodes it evaluates once a cluster passes roughly 100 nodes, trading a small chance of a marginally less optimal placement for scheduling throughput. If pods are scheduling slowly at scale (not stuck Pending, just slow to place), this setting, along with the number of scheduler profiles and any scheduler extenders in the pipeline, is where to look before assuming it's a resource-shortage problem.

Cold-start latency

A pod that schedules quickly can still take a long time to reach Running if the node has to pull a large image cold. Pre-pulling common images onto nodes (via a DaemonSet or node image baked into the AMI), keeping images small, and using an appropriate startupProbe so a slow-starting container isn't killed by liveness checks before it finishes initializing, are the standard levers; that's a related but distinct problem from scheduling itself, since a pod experiencing this has already left Pending.

Trade-offs and pitfalls

• Reading only the first FailedScheduling message and assuming a single cause is the most common debugging mistake; the message lists every filter that eliminated nodes, and a fix that addresses only one of them can still leave the pod Pending.
• Loosening nodeAffinity from required to preferred unblocks scheduling but silently removes a placement guarantee (for example zone isolation); confirm that's actually acceptable before doing it under incident pressure.
• Preemption is a blunt instrument: overusing high PriorityClass values on routine workloads causes churn as pods repeatedly evict each other rather than the cluster simply scaling out.

Direct answer

I took on a third concurrent project on top of two I was already stretched across, and within a few weeks I noticed my own review standards slipping, catching fewer edge cases in my own work before sending it out, before anyone else raised it. Once I noticed, I renegotiated specific commitments rather than trying to quietly power through, and what keeps it from happening again is a concrete capacity check I now run before agreeing to new work, not just a general intention to say no more.

How I noticed

The signal wasn't a single dramatic mistake, it was a pattern I caught in my own behavior: I found myself skipping a self-review step I normally did before sending work out, telling myself it was fine this once, three separate times in the same week. Individually each of those felt like a reasonable shortcut under pressure; noticing the pattern, not just the individual instances, is what told me something was actually slipping rather than me just having a busy week.

What I did once I noticed

I went to my manager before it became visible as an external problem, with a specific account of what I'd taken on and where I felt the quality risk actually was, rather than a vague "I'm busy." We renegotiated one of the three commitments, pushing a deliverable's timeline by two weeks, which meant having an uncomfortable conversation with that stakeholder myself rather than letting my manager absorb that cost. I also went back through my recent work from the previous two weeks specifically looking for the kind of mistake my slipping review process would have missed, and found one, a data validation step I'd skipped, that I corrected before it caused a downstream problem.

What keeps it from happening again

The general resolution to "manage my time better" hadn't worked for me in the past, so instead I built a specific check: before I say yes to new work, I look at what's already committed and ask whether taking this on would mean dropping a specific quality step somewhere, not just whether I have hours free on a calendar. That reframes the question from "do I have time" to "what exactly would I stop doing to make time," which is a much harder question to wave away.

Trade-offs and pitfalls

The pitfall is treating "I'm managing" as proof that standards haven't slipped, when the slip is often invisible from the inside until you look for the specific behavior, like a skipped review step, rather than trusting how in-control you feel. The trade-off in raising it before anyone else notices is that it feels like admitting a weakness proactively, but it's far cheaper than the alternative of someone else catching the actual mistake downstream.

Direct answer

A cloud-native web application protection layer against Layer 3 through Layer 7 (L3-L7) Distributed Denial of Service (DDoS) and application-layer attacks stacks its defenses by network depth, volumetric attacks absorbed at the network edge, protocol-level attacks handled by the load-balancing and content delivery layer, and application-layer attacks inspected by a web application firewall (WAF), because each layer's detection mechanism is fundamentally different and none of them can substitute for another; the hardest part is not any single layer, it is detecting an application-layer attack specifically engineered to look like legitimate traffic to a signature-based WAF.

Structured elaboration

Traffic routing and edge presence. Route all traffic through a global content delivery network (CDN) edge (CloudFront or Azure Front Door) before it reaches origin infrastructure at all; this gives the provider's own massive edge capacity as the first absorption layer for volumetric (L3/L4) attacks, since the attack traffic dissipates across a distributed edge network rather than concentrating on the application's own, comparatively much smaller, origin capacity.

Edge caching. Cache cacheable content (static assets, and, where the application's data freshness requirements allow, some API responses) at the edge, so a volumetric attack targeting those cacheable paths is served from cache rather than reaching origin at all; this both reduces the attack's actual impact on origin and reduces legitimate traffic's baseline load, giving more effective headroom during a genuine attack.

Rate-based and behavioral rules. Rate-based WAF rules block a single source exceeding a request-rate threshold, catching straightforward volumetric application-layer abuse; behavioral rules (anomaly detection comparing current traffic patterns against an established baseline) catch a more sophisticated attack distributed across many sources, each individually staying under the rate threshold, precisely the pattern a pure rate-limit rule cannot see.

Automated mitigation actions. Configure the DDoS protection service (AWS Shield Advanced, or the equivalent) to trigger automated mitigation (traffic scrubbing, automatic rule tightening) without waiting for human intervention during a detected attack, since a volumetric or protocol-level attack can escalate faster than a human response time allows; automated actions should have a defined, reviewed scope (specifically what they are permitted to change automatically) so an automated response cannot itself cause an unintended availability impact on legitimate traffic.

Detecting application attacks that evade signature-based WAF rules. Signature-based rules match known attack patterns; an attacker who varies payload encoding, splits an attack across multiple requests, or crafts a request that is individually benign-looking but adversarial in aggregate (a slow, low-rate credential-stuffing attempt distributed across many source IPs and many hours, for instance) evades signature matching entirely. Detecting this requires behavioral and statistical analysis layered on top of signature matching: per-endpoint request-pattern baselining (this specific API endpoint typically receives requests with this shape, from this rough geographic distribution, at this rate), flagging a deviation even when no individual request matches a known-bad signature; and correlating across a wider time window than a single request or a single source IP, since the evasive pattern's defining property is often precisely that no single request or source looks anomalous in isolation.

Worked example

An attacker mounts a credential-stuffing campaign against the application's login endpoint, using a large, distributed set of source IPs, each making requests at a rate well under any reasonable per-IP rate limit, and each individual request looking like a normal login attempt with no signature-matchable malicious payload. A pure signature-based WAF rule set does not flag any individual request. Behavioral baselining, tracking the login endpoint's aggregate failed-login rate and the diversity of source IPs attempting logins against a specific set of usernames, flags the pattern as anomalous once the aggregate failed-login volume and the unusual username-targeting pattern (many different source IPs, but converging on a narrow, specific set of usernames, rather than the broad, random distribution a genuine traffic surge would show) cross an established threshold; this triggers a step-up authentication challenge (a CAPTCHA or equivalent) specifically for the login endpoint, rather than a blanket rate-limit or IP block that would also affect legitimate users, since the actual anomaly here is the aggregate pattern, not any single source's behavior.

Trade-offs and pitfalls

• Signature-based WAF rules and behavioral detection catch fundamentally different attack shapes, and a design that relies on signature rules alone will structurally miss the worked example's credential-stuffing pattern regardless of how current the signature rule set is kept, since no individual request in that campaign matches a known-bad signature to begin with; behavioral detection is not a redundant enhancement, it is the layer that closes a gap signature matching cannot close by design.
• Automated mitigation actions carry a real risk of blocking legitimate traffic during a genuine attack, particularly for a behavioral rule tuned too aggressively; the response in the worked example (a step-up challenge specifically for the affected endpoint, rather than a blanket block) reflects a deliberate choice to minimize legitimate-user impact while still disrupting the attack, a more surgical response than an aggressive automated block would have been.
• Edge caching's DDoS-mitigation benefit only applies to genuinely cacheable content; an attack specifically targeting an uncacheable, dynamic endpoint (the login endpoint in the worked example, which cannot be cached) gets none of that benefit, which is exactly why the worked example needed a behavioral, endpoint-specific response rather than relying on the edge-caching layer to absorb it.
• Per-endpoint behavioral baselining requires enough historical traffic to establish a meaningful baseline in the first place, a real limitation for a newly-launched endpoint or a rarely-used one, where "normal" traffic volume and pattern are not yet well-established; a new endpoint needs either a more conservative default sensitivity or a longer observation period before its behavioral detection can be trusted at the same confidence level as an established one.

Direct answer

Structure it as an evidence led case, not a request for a favor. Open by naming the specific ask, promotion, a stretch assignment, or dedicated time and budget, back it with three or four concrete instances of impact and readiness, and pre-empt the most likely objection with a fallback. The conversation should feel like two people already broadly aligned on the goal, working out timeline and specifics, not a persuasion contest.

Structured elaboration

Open with the ask itself. Name what you want as your first sentence, not your last. Ambiguity in the open lets the conversation get steered before you've made your case.

Bring evidence, not adjectives. Two to four concrete instances where you already operated at the level you're asking for, a project led beyond formal scope, a decision others now rely on, a skill built and applied. Evidence should be specific enough that your manager could describe it to their manager without you in the room.

Anticipate the likely objections. There's no open role at that level, the timing is wrong for budget, you need more evidence in one area. A prepared response isn't a rebuttal, it's a next step, what would close the gap and by when.

Bring a fallback. If the primary ask can't be granted in full, have a smaller alternative ready, an interim scope change, a defined stretch project with a review date, or a partial commitment such as title now and a compensation review next quarter. Arriving with only one possible outcome makes it binary and easy to defer.

Close with a mechanism. Propose a specific follow up date and what would need to be true by then for the answer to change.

Worked example

"I asked for time on my manager's calendar and opened directly, saying I wanted to talk about taking the stretch assignment leading the migration project and what that meant for my scope going forward. I brought three examples where I'd already operated at that level informally, a cross team escalation I'd resolved without waiting for my manager, a proposal the team had adopted, and feedback from a peer who said they now came to me first on a certain class of problem. My manager's first response was that the team couldn't spare me from current work. I'd anticipated that and offered a fallback, take the assignment for the first phase only with a defined handoff point, so my current responsibilities weren't left uncovered. We agreed to that scope, with a check in scheduled for the midpoint to decide whether to extend it."

Trade-offs & pitfalls

• Leading with feelings instead of evidence invites the manager to respond to the emotion rather than the case.
• Bringing only one possible outcome, with no fallback, turns the conversation into a yes or no vote you can lose outright.
• Overloading the evidence list dilutes it. Two or three strong, specific instances beat six vague ones.
• Skipping the close is the most common gap. A conversation that ends without an agreed next step tends to quietly disappear from both people's priorities.

Direct answer

Pick a project that genuinely needed more than one function, and be specific about two things: what YOU owned (not what 'the team' did), and the one concrete mechanism that determined whether the collaboration worked, such as a shared definition of done, a clear handoff point, or clarity on who decided what when opinions differed. Vague answers ('we communicated well') sound rehearsed; specific answers sound lived-in.

What the story needs to show

Your specific contribution. Interviewers are listening for what you personally decided or built, distinct from what your collaborators did. If every sentence is 'we', the interviewer cannot tell what you'd do differently on the next team.

A mechanism-level explanation. Organize the story around one of three lenses:

• Shared goal: did every function agree on what 'done' looked like and how success would be measured, or was each function quietly optimizing for its own definition?
• Interface or handoff: was there a clear point where work crossed from one function to another, and was that point actually defined, or did people guess?
• Decision rights: when functions disagreed, was it clear whose call it was, or did disagreement just stall until someone got tired of arguing?

Honesty if it's a struggle story. The question explicitly allows 'succeed or struggle'. A good struggle story ends on what you changed about the collaboration, not on who was at fault.

Worked example

Situation: [your team] needed to deliver [a feature or initiative] that required real work from [Team A, for example a design or research function] and [Team B, for example a data or infra function], against a fixed external date.
Task: your role was the one connecting the three groups, for example owning the shape of the interface between design and engineering, or owning how data requirements got translated into a schema.
Action: early on, each function had a different idea of what 'done' meant for their piece, which caused rework when the pieces met. You wrote a short one-page agreement naming the shared definition of done and who would sign off on each handoff, and used it to resolve the next two disagreements without a meeting.
Result: the project shipped on the revised date, and the agreement itself became something the group reused on the next cross-functional piece of work, which is the real marker of a story about redesigning the collaboration rather than just pushing through it.

To make that skeleton concrete rather than a fill-in-the-blank: picture a checkout redesign that needed real work from the design function and the payments engineering function, against a fixed external date tied to a promotional campaign launch. The specific disagreement was about what 'done' meant for the new payment-method selector: design considered the screen done once every state (loading, error, empty) matched the approved mockups pixel-for-pixel, while payments engineering considered it done once the integration correctly handled every payment-provider response code, even ones with no mockup drawn yet. That mismatch caused two rounds of rework when a payment-provider error state shipped without a design pass. The one-page agreement that resolved it included this line: 'A screen is done when it matches an approved mockup for every state the payments API can return, and any new state discovered after mockups are drawn triggers a joint 15-minute review before either side builds it.' That single sentence is what let the two functions stop re-litigating 'done' every time a new edge case appeared, and both sides signed off on it before the next round of work began.

Trade-offs and pitfalls

• A generic 'we all communicated well' answer with no mechanism is the single most common weak version of this story, avoid it.
• Over-crediting the team at the expense of your own specific contribution leaves the interviewer unable to evaluate you.
• If you pick a struggle story, resist framing it as the other function's fault. The senior version of this answer explains what you changed about how the groups worked together, not who dropped the ball.
• The strongest answers show you redesigning a structure (a handoff, a shared definition, a decision rule), not just working harder inside a broken one.

Direct answer

Frame the cut through the 99.99% error budget, not a blanket percentage. 99.99% allows about 52.6 minutes of downtime a year; anything that doesn't touch the paths that consume that budget is safe to cut aggressively, and anything that does must be modeled against it explicitly before you touch it. In practice that means chasing waste and inefficiency hard (usually most of the 30%), and treating redundancy, failover paths, and DR test cadence as protected unless you can show the specific budget impact is acceptable.

A three-bucket framework for the savings

That number is the currency every cut gets priced in.

Chase bucket 1 and 2 first and aggressively (they rarely conflict with availability); only reach into bucket 3 if 1 and 2 don't get you to 30%, and price every bucket-3 cut against the budget above.

Worked example

Suppose an audit of monthly compute spend of $300,000 finds 15% idle or overprovisioned capacity, plus another 10% recoverable through rightsizing and reserved commitments on stateless, replaceable capacity:

That's 25 of the 30 points from buckets 1 and 2 alone, with essentially zero availability risk. The remaining 5 points has to come from bucket 3, for example dropping a redundant standby from 3 independent paths to 2 on some service. Whether that's safe depends entirely on whether that service is in the 99.99% critical scope. Reusing the same series/parallel redundancy model above (an N-way redundant path's unavailability is the product of each independent unit's own unavailability, since all N units have to fail at the same time for the whole path to be down: $U_N = (1-a)^N$), for a building block at $a=0.99$:

Dropping that one path from N=3 to N=2 raises its expected downtime contribution from about half a minute a year (0.5256 min/yr) to 52.56 minutes a year, a 100x jump that would consume the entire annual budget for the 99.99% target on that one path alone. That's the concrete argument for why redundancy counts on in-scope critical services are protected regardless of the cost target: the math shows the cut doesn't save what it looks like it saves once you price the risk.

Trade-offs and pitfalls

The classic failure is applying a flat 30% cut across the board instead of segmenting critical from non-critical, which either misses easy wins in non-critical systems or, worse, quietly erodes redundancy on a critical path because nobody explicitly modeled the budget impact. Watch for Goodhart's-law style gaming too: cutting observability or alerting spend looks free on the invoice but raises mean time to detect, which inflates the effective downtime against the same budget without showing up as an "availability" line item until an incident hits. What to protect no matter what: replica or quorum counts below the tested minimum on in-scope services, cross-region failover paths, backup and DR test cadence, and on-call staffing, because all four either directly hold the redundancy math above or determine how fast you can react when it fails.

Direct answer
The real question isn't which option is faster, it's which one is cheaper to be wrong about given what you currently know. I'd gather a quick read on how quantifiable and recoverable the rollback's data loss actually is, and how bounded or open-ended the patch's propagation risk is, before deciding, and I'd document the decision and get a second person's sign-off given the stakes.

Structured elaboration

• Quantify the rollback's cost. Is the lost day of data truly unrecoverable, or can some of it be reconstructed from logs, upstream systems, or replay? A rollback that loses genuinely unrecoverable customer data is a much bigger deal than one that loses data you can mostly reconstruct.
• Bound the patch's risk. Is 'risking propagation' a vague fear or a specific, boundable failure mode? If you can identify exactly what could go wrong and put a guard around it (a feature flag, a canary rollout of the patch itself), the risk becomes much more manageable than an open-ended 'we're not sure how bad this could get.'
• Consider reversibility of each option, not just its immediate outcome. A rollback is usually fast to reverse if it turns out to be wrong (roll forward again); a patch that goes wrong under time pressure is often harder to cleanly undo, especially if it's already started propagating.
• Decide who needs to sign off. For a decision with real data loss or real risk of making things worse, this shouldn't be a unilateral call under pressure; loop in whoever owns the data (for the rollback) or whoever understands the propagation risk best (for the patch), even if briefly.
• Document the decision and the reasoning, not just the action taken, so it can be explained afterward and revisited in the postmortem.
• Explain the decision honestly to the people who bear its cost. If you accept the rollback's data loss, tell the specific team or customers whose day of writes is gone what was lost and why, rather than a vague status update; if you accept the patch's propagation risk, tell whoever owns the systems it could spread to what you're watching for. People affected by a real cost should hear the reasoning, not just the outcome.
• Watch for a subtlety: your 'safe' default might not be safe. Sometimes the rollback mechanism itself is the risky part (a rollback tool that has its own failure modes, or that could trigger a different kind of cascading failure), so the instinct to always default to rollback as 'the safe choice' needs the same scrutiny as the riskier-looking option.

Worked example
A bad deployment has two disagreeing camps: roll back (losing a day of writes) or patch forward (risking the underlying bug spreading to more data). First, the team checks whether the day of writes can be reconstructed from an upstream event log; it turns out about 80% of it can be replayed after rollback, meaningfully reducing the real cost of that option. Second, they check whether the patch's propagation risk can be bounded; the bug only affects a specific, identifiable code path, so a scoped patch with a feature flag around just that path is possible rather than a full risky redeploy. Given both, the team chooses the scoped patch behind a flag, since the true data loss from rollback (even reduced by replay) is now higher-cost than a well-bounded patch, and they get a second engineer to review the patch's blast radius before shipping it, given the stakes.

Trade-offs and pitfalls
A common mistake is defaulting to rollback simply because it feels psychologically safer ('undo the thing we just did'), without actually quantifying whether the data loss is worse than the alternative risk; rollback is not automatically the conservative choice. Another mistake, at the opposite extreme, is defaulting to a quick patch because it avoids acknowledging any data loss, without honestly bounding how far the underlying problem could actually spread. A related edge case: sometimes the rollback path itself carries risk (a partial rollback failing midway, or a rollback tool triggering a separate cascading failure under load), so 'roll back, it's the safe option' deserves the same scrutiny as the alternative, not an automatic pass.