Senior Backend Developer Interview Preparation Guide - FAANG Standards

Direct answer

A disaster recovery runbook is only useful if a stressed engineer can execute it top to bottom without needing to look anything else up. That means naming an owner, stating the RTO and RPO it is designed to meet, listing ordered and specific actions (exact commands or console steps, not descriptions), and ending with concrete verification steps that prove the system is actually back, not just that the steps were followed.

Structured elaboration

Versioning and access. Store the runbook in source control with required review on changes, so every edit has an author and a diff. Test it in a real drill, not a tabletop discussion only, on a cadence tied to how critical the service is, quarterly for anything customer-facing. Keep it reachable when the primary systems it recovers are down: a runbook that lives only on an internal wiki hosted in the region that just failed is not a disaster recovery runbook.

Worked example

For a service with RTO = 30 minutes, a well-built runbook's step timings should sum to that budget, and the sum should be checked, not assumed:

That sum matching the stated RTO exactly is what makes the RTO a testable claim rather than a number pasted at the top of the document. If a quarterly drill shows execution consistently takes 20 minutes instead of 15, the runbook's RTO is wrong and needs to be corrected, not explained away.

Trade-offs & pitfalls

• A runbook with no owner drifts out of date the first time the architecture changes; ownership is not optional metadata.
• Testing via tabletop discussion only, never an actual drill, hides the gap between the steps sounding right and the steps working; most drift is caught only by execution.
• Over-specifying every command for a fast-moving system creates a maintenance burden that causes the runbook to be abandoned; balance specificity against how often the underlying commands change.
• Storing the only copy behind the same authentication system that depends on the region that just failed is a common, self-defeating mistake.

Put a durable, buffered transport (Kafka or similar) between the per-host log shippers and the indexing layer, so a traffic spike or a downstream slowdown turns into a growing queue instead of dropped logs or a crashed indexer. Feed a fast, short-retention hot index for recent search, and drain the same durable stream into cheap columnar cold storage for the longer archive. Backpressure matters most at exactly that shipper-to-transport boundary and at the transport-to-indexer boundary, not at the shippers themselves, which turn out to be the least-loaded part of the system at this scale.

Framework

Components:

• Per-host/per-pod shipper (Vector or Fluent Bit): tails logs, does minimal parsing, buffers to local disk, forwards to the transport layer. Disk buffering here protects against short transport outages.
• Durable transport (Kafka): decouples producers (shippers) from consumers (indexers), absorbs spikes as queue depth rather than dropped data, and gives you replay if a downstream consumer falls behind or needs to be rebuilt.
• Stream processor: parses, validates schema, and enriches (adds service metadata, drops known-noisy fields) before writing onward.
• Hot index (OpenSearch/ClickHouse, days-to-weeks retention): fast, full-text/aggregation queries for recent incidents.
• Cold archive (object storage, columnar format): cheap, long-retention, queried via a batch engine rather than an interactive index.

flowchart LR
  A[Services, ~few hundred] --> B[Local shipper\ndisk buffer]
  B --> C[Kafka\ndurable buffer]
  C --> D[Stream processor\nparse + enrich]
  D --> E[Hot index\nrecent, searchable]
  D --> F[Cold archive\nS3, columnar]

Where backpressure matters most. Two places, and they behave very differently:

1. Shipper to Kafka. If Kafka is briefly unreachable, the shipper's local disk buffer absorbs the gap. This is cheap to size generously because, per host, the actual data volume is small (see the worked example below).
2. Kafka to indexer. If the hot index can't keep up with a traffic spike, Kafka's own retention (not a local disk buffer) is what protects you: consumer lag grows, but nothing is dropped, and the indexer catches up once the spike passes. This is the boundary to actually provision and monitor for, since it's shared across the whole fleet rather than per-host.

Retention doesn't have to be uniform across signal types. Not every log line is equally valuable for the same length of time. A reasonable policy differentiates: DEBUG/INFO access logs might only need a few days of hot retention before dropping to cold storage or expiring outright, WARN/ERROR logs are worth keeping searchable longer since they're the ones actually pulled up during incident review, and audit-relevant events (auth, permission changes) may need their own longer, possibly separate retention track for compliance reasons regardless of how noisy the rest of the pipeline is. Building this as a per-signal-type policy in the stream processor (route to different index lifecycle rules based on level or a dedicated audit tag) keeps hot-tier cost proportional to what's actually worth keeping hot, rather than uniformly retaining everything for the same window.

Dashboards have their own latency budget, separate from ingest. A pipeline that ingests everything correctly can still fail operators if dashboard queries against the hot index take seconds instead of being near-instant during an incident; that's a property of hot-tier sizing and query patterns (how many fields are indexed, how wide the time range a typical dashboard panel scans), not just of whether the pipeline keeps up with ingest volume, and it's worth setting an explicit target (for example, sub-second p95 for a standard incident dashboard) and load-testing the hot tier against it directly, rather than assuming healthy ingest implies healthy dashboards.

Worked example

Baseline: ~1,000 log events/sec across a few hundred services, average event size 500 bytes, occasional spikes to 5x normal (5,000 events/sec).

Per-shipper buffer sizing is not the bottleneck. With, say, 300 services sharing that 1,000 events/sec baseline evenly, each service averages $1000 / 300 \approx 3.3$ events/sec, or $16.7$ events/sec at a 5x spike. To tolerate a 5-minute Kafka outage without losing data, a single shipper needs a local buffer of:

That's trivially small for local disk, which is exactly the point: at this scale, per-host buffering is not where the design risk lives.

The shared Kafka-to-indexer boundary is where the risk lives. Daily baseline volume is $1000 \times 86{,}400 = 86{,}400{,}000$ events/day, or at 500 bytes each, $43.2$ GB/day. At 30 days hot retention, that's roughly 1.3 TB of raw hot data before any compression, and the hot index has to sustain ingest at spike rate (5,000 events/sec, $2.5$ MB/sec), not just baseline, because a spike that outpaces indexing turns into growing consumer lag and stale search results right when something is probably going wrong and people are searching the most.

Trade-offs and pitfalls

• Provisioning the hot index for the 5x spike rate permanently is safe but wastes capacity most of the time; the alternative, autoscaling ingest nodes on lag/CPU signals, is cheaper on average but adds operational complexity (scale-up lag, cold-start time on new nodes) that has to be tuned so it reacts before the queue backs up too far.
• The Kafka buffer is doing real work here (durability, replay, decoupling), but it's also a new stateful system to operate: partition count, retention, and replica count all need capacity planning, and a misconfigured retention window can silently start dropping the very data you added Kafka to protect.
• Splitting hot/cold means any query spanning both (a 45-day lookback, say, when hot retention is 30 days) needs query-routing logic that stitches the two together, which is more moving parts than a single store and a common place for "the dashboard shows a gap" bugs to hide.
• A local shipper disk buffer only protects against outages shorter than the buffer window; monitor buffer fill level explicitly, because silent data loss when a buffer fills and starts dropping is worse than an alert that fires early.

Direct answer
For a user-facing operation composing data from several backend services, calling them all synchronously in parallel and waiting for every response is simplest to build but ties the caller's response time and availability to the slowest and least reliable of the services it calls; handling the composition asynchronously (via caching, pre-computation, or a read model built ahead of time) trades a small amount of freshness for a response time and availability that don't depend on every backend being fast and up at the exact moment of the request.

Structured elaboration
Synchronous fan-out (calling every needed service in parallel and waiting for all responses before replying) is the most straightforward approach and gives the freshest possible data, but it means the composed response is only as fast as the slowest call and only as available as the least reliable service in the fan-out; a single flaky downstream service degrades every request that needs its data, even if the other services involved are healthy. It works reasonably well when the number of services being composed is small, their latency is predictable and low, and there's a clear timeout-and-fallback policy for when one of them is slow (returning a partial result rather than hanging the whole request). The asynchronous alternative pre-computes or caches the composed result ahead of time (for example, a background job periodically assembles the combined view and stores it somewhere fast to read), so the request path only ever reads from one fast, always-available store rather than fanning out live; the cost is that the data shown can be stale by however long the pre-computation lag is.

Worked example
A product page that needs to show price (from a Pricing service), stock availability (from an Inventory service), and personalized recommendations (from a Recommendations service) at once: doing this via a live synchronous fan-out on every page view means a slow Recommendations call (even one that just times out and falls back to "no recommendations shown") shouldn't be allowed to delay or fail the whole page; a well-designed synchronous approach here still uses per-call timeouts and treats each piece as optional with a sensible fallback. An asynchronous alternative would pre-compute the recommendations for a user periodically (recommendations don't need to reflect the last five seconds of behavior) while keeping price and stock as live synchronous calls, since those two genuinely need to be current at the moment of purchase.

Trade-offs and pitfalls
The risk with synchronous fan-out is treating every piece of the composed response as equally critical, so a failure in a genuinely non-essential piece (like recommendations) takes down the whole page instead of just that one section; the fix is deciding, per piece of data, whether it's essential (block on it, with a tight timeout) or optional (fetch it with a shorter timeout and a graceful fallback if it's slow or unavailable). The risk with pre-computed or cached data is using it for something that actually needed to be current (like showing a stale "in stock" status right before checkout), which can produce a worse user experience than a slightly slower live check would have.

Direct answer

The deliverable itself is what's being tested here, so the answer is to actually produce it: a first paragraph stating mission and composition with assumptions clearly flagged, a second paragraph connecting the role to product and engineering objectives, followed by naming the two or three assumptions that matter most and exactly how you'd confirm them with the hiring manager.

Structured elaboration

• Reconstructing composition from public sources: cross-reference LinkedIn's people list (filtered by title keywords) against the company page, plus phrases like "you'll report to" or "you'll work closely with" across several job postings, plus an org-chart tool if the company is large enough to appear on one. This lets you build a probable reporting structure without ever having seen an internal chart.
• Flagging assumptions honestly: use explicitly hedged language ("I'm assuming X, based on Y") instead of presenting an inference as settled fact. A hiring manager who catches an inference stated as certainty will discount the rest of what you say.
• Turning assumptions into real questions: convert each one into a short, specific question rather than a vague "tell me about the team," for example, "I saw the team grew by three people over the last two quarters on LinkedIn, is that mostly backfill or net-new headcount?"

Worked example

Paragraph 1: "Based on your careers page and recent postings, I'm assuming this team sits within the Payments organization and has roughly ten engineers split across a Core Payments pod (a small, semi-autonomous sub-team owning a specific area) and a Fraud and Risk pod, reporting to an Engineering Manager who reports to a Director of Payments Engineering. I'm inferring this from three of the last four postings listing 'Payments Engineering' as the department and two mentioning partnering with a Fraud team."

Paragraph 2: "The team's mission looks like keeping transaction success rates high while reducing fraud losses, which connects to your recently announced plan to expand into higher-risk international markets, since that expansion will put pressure on both systems at once."

Assumptions to confirm: "Two things I'd want to check directly with you: whether Fraud and Risk is really a separate pod or a shared responsibility inside Core Payments, and whether this role is a backfill or a net-new seat created specifically for the international expansion."

Trade-offs and pitfalls

Resist the urge to sound fully certain, a summary this specific will be wrong somewhere, and naming that up front, rather than being caught out later, is what turns "did some research" into "can operate with incomplete information," which is the actual thing being evaluated. Don't let the assumption list balloon into hedging on everything either, pick the two or three that would most change your approach, not every minor uncertainty you noticed.

Situation: On a launch project, my team owned the API work and the partner team owned the customer-facing workflow. We both wanted the same release date, but their priority was polish while mine was integration stability.

Task: I needed to keep both sides moving even as trade-offs came up.

Action: I set up a shared plan with one clear owner per dependency, then separated must-have work from nice-to-have work. I also defined the term "trade-off" for the group as a choice where we gain one benefit by giving up another, so the conversation stayed concrete. When design wanted an extra step and engineering needed more time for testing, I asked, "What is the smallest version that still protects the user and the launch date?" We agreed to ship the core flow first, keep one optional enhancement for later, and review progress twice a week.

Result: We delivered the shared goal on time with a smaller scope, and both teams felt heard. I learned that progress keeps moving when you make the decision criteria explicit instead of debating opinions.

Direct answer
Difficult feedback to a mentee works best delivered privately, tied to a specific, observed behavior and its concrete impact, not to the person's character, and followed up on to confirm the message landed and something changed. The delivery mechanics matter less than getting three things right: timing (soon after the behavior, not saved up), specificity (a real example, not a vague pattern), and follow-through (checking back in, not treating the conversation itself as the fix).

Structured elaboration

Before the conversation

• Get the facts straight: what exactly happened, what was the impact, and is this a one-off or a pattern. Vague feedback ("you need to be more careful") is unusable; a mentee can't act on a mood, only on a specific instance.
• Decide the stakes. Not all critical feedback carries the same weight:
  • A routine performance gap (missed a deadline, sloppy code style) can wait for the next scheduled 1:1.
  • An ethical or safety concern (something in the person's work poses real risk if it ships) changes the calculus: it needs to happen immediately, privately, and is often paired with a concrete containment step (pause the change, get a second reviewer), not just a conversation.
• Decide the medium: a private 1:1, not written feedback and not in front of the team, unless the finding also needs to be logged for safety or compliance reasons.

Delivering it

• Lead with the specific behavior and its impact, not a label: "this change would have introduced X" is usable; "this was careless" is not.
• Ask before you assert. The mentee may know something you don't (a constraint you weren't aware of); asking "walk me through the reasoning" often surfaces that before you've over-committed to a judgment.
• Separate the person from the work. The message is "this output has a problem," not "you are the problem."

When the power dynamic is reversed

Feedback isn't always flowing to someone junior. Giving critical feedback to a mentee who is more senior, more tenured, or simply more confident than you requires the same content but a different frame: lead with genuine respect for their experience, be more explicit that you're not questioning their general competence, and expect (and plan for) more pushback. Defensiveness here is a normal reaction to a status threat, not necessarily a sign the feedback was wrong; the skill is staying anchored to the specific evidence instead of either escalating or backing down.

After the conversation

• Confirm shared understanding before ending: ask them to restate what they heard.
• Agree on a concrete next step and a checkpoint to revisit it, not just "let's see how it goes."
• Follow up. Feedback that isn't revisited quietly signals it wasn't actually important.

Worked example

Situation

During a code review, I found a race condition in an interrupt service routine (an ISR, the block of code that runs automatically when a hardware event interrupts normal execution) a mentee had written: a shared buffer was being written from the ISR without disabling interrupts around the critical section (the stretch of code that touches shared data and must not be interrupted mid-update), so a preemption (the interrupt firing and pausing the main code at an unpredictable moment) at the wrong moment could corrupt data intermittently and unpredictably.

Why this wasn't routine feedback

This wasn't a style nitpick. Left unaddressed it was a latent, hard-to-reproduce bug that could surface in the field. That pushed it from "note it for next time" to "we talk today, and the change doesn't merge until it's fixed."

The conversation

I asked the mentee to walk me through what happens if the interrupt fires mid-write, rather than telling them the bug outright. They found the failure mode themselves partway through the explanation, so the fix landed as their own understanding rather than my correction. We then talked through the general pattern (anything touching state shared between an ISR and main-line code needs an explicit critical section) so it would generalize past this one bug.

Follow-through

I asked them to check two other places in the codebase where similar shared state existed, as a way to prove the concept had stuck rather than just fixing the one instance. Both had the same latent issue.

Result

The immediate bug was fixed before merge, and the mentee started flagging similar patterns unprompted in their own future changes, which was the real signal the feedback had generalized rather than just been complied with once.

Trade-offs & pitfalls

• The feedback sandwich dilutes the message. Padding critical feedback between two compliments is a common junior instinct; it often causes the actual point to get lost. Genuine positive feedback is worth giving, but on its own merits, not as camouflage for the critical part.
• Waiting to "collect examples" delays too long. A senior mentor gives feedback close to the event; batching several issues into one big conversation later makes it feel like an ambush and makes each point harder to act on.
• Not distinguishing skill gap from something more serious. A performance gap and an ethical or safety issue call for different urgency and different documentation; treating a safety issue as routine coaching is itself a failure mode worth naming.
• Confusing "they got defensive" with "I was wrong." Especially with a more senior or tenured mentee, defensiveness is a predictable reaction to a status threat. A junior mentor backs off; a senior one stays anchored to the specific evidence while still leaving room for the other person to be right about something they missed.

Direct answer

Prefer Union-Find when connectivity needs to be queried repeatedly against a growing (insert-only) graph, or when the graph arrives as a stream of edges that must never all be held in memory at once as an adjacency structure. Prefer DFS/BFS when you need the graph traversed only once (a single connected-components pass), when you need the actual member list or shape of each component (not just "are these two connected"), or when the graph already needs to be materialized in memory for other reasons anyway. Both give the same $O(V+E)$-class total cost for a one-shot components computation; the real difference is in incremental behavior and memory footprint, not raw speed for a single full pass.

Structured elaboration

Practical performance on large sparse graphs. For a single, one-time "compute all components now" pass, DFS/BFS and Union-Find are asymptotically equivalent: $O(V+E)$ for the traversal versus $O(E \cdot \alpha(n)) \approx O(E)$ for Union-Find (since $\alpha(n) \le 4$ in practice). On a sparse graph ($E = O(V)$, the common real-world case for social graphs, service topologies, and dependency graphs), both are effectively linear in $V$. The practical performance difference shows up in constant factors and access pattern: DFS/BFS needs random access into an adjacency structure (cache-friendly if built as compressed sparse row, less so as a hash-map-backed adjacency list), while Union-Find only ever touches two flat arrays indexed directly by node ID, which tends to have excellent cache locality regardless of how the edges happen to be ordered.

Memory.

Union-Find's memory advantage is specifically that it never needs random access back into the edge set once an edge has been processed; DFS/BFS inherently needs to revisit a node's neighbor list every time that node is reached, so the adjacency structure must stay resident for the traversal's duration.

Incremental update implications. This is the sharpest practical difference. Adding a new edge to an already-computed Union-Find result is cheap: one union call, $O(\alpha(n))$ amortized, no need to touch anything else. Adding a new edge after a DFS/BFS-computed component list, by contrast, requires either a fresh traversal from one of the edge's endpoints to merge two existing components (cheap if the components were tracked as explicit sets you can merge, expensive if you would naively recompute everything), or accepting that the previously-computed component list is now stale. Neither structure handles edge REMOVAL cheaply: DFS/BFS needs a full re-traversal of the affected component to determine if it split, and Union-Find has no "un-union" operation at all, so both degrade to "rebuild from scratch" (or a much more complex dedicated dynamic-connectivity structure) under deletions.

Worked example

Consider a service-dependency graph with 50,000 services (nodes) and 120,000 dependency edges (sparse, average out-degree 2.4), where the graph is rebuilt from a fresh edge list once per minute by a monitoring job, and additionally single new edges arrive in near-real-time as services register new dependencies between rebuilds.

• The once-per-minute full rebuild is a one-shot computation over the complete edge list either way: a DFS/BFS pass costs $O(V+E) = O(50{,}000 + 120{,}000) = O(170{,}000)$ node-and-edge touches; a Union-Find pass over the same 120,000 edges costs $O(E \cdot \alpha(n)) \approx O(120{,}000 \times 4) = O(480{,}000)$ amortized array operations, which is a larger constant-factor count of individual operations but each operation is a simple array read/write rather than a hash-map or adjacency-list lookup, so in practice the two approaches land in a similar wall-clock ballpark for this size; the choice would come down to which representation the rest of the monitoring job already needs (if it needs an adjacency structure for other purposes anyway, DFS/BFS reuses it for free).
• The near-real-time single-edge updates between rebuilds are where the two approaches diverge sharply: applying one new edge to an already-built Union-Find structure is a single $O(\alpha(n))$ union call, whereas correctly updating a DFS/BFS-derived component list for one new edge (without a full re-traversal) requires the monitoring job to have retained enough bookkeeping to merge two component records directly, which is realistically only clean to do if that bookkeeping itself amounts to reimplementing a Union-Find-like merge step on top of the traversal result.

Trade-offs and pitfalls

• Common mistake: treating this as purely an asymptotic-complexity question. For a single full pass, both are effectively $O(V+E)$-class; the decision in practice is almost always about incremental-update patterns and what auxiliary information (member lists vs a fast connectivity check) the downstream consumer actually needs, not about which is "faster."
• Common mistake: assuming Union-Find handles deletions just because it handles insertions cheaply. The asymmetry (cheap merge, no cheap split) is the single most important practical trade-off and the one most likely to bite a design that assumed both operations were equally supported.
• If the downstream need is "list every member of every component," DFS/BFS gives that as a direct byproduct of the traversal; Union-Find gives it only after an additional $O(V)$ grouping pass over all nodes by final root, a cost that is easy to forget when comparing the two approaches only on their headline complexity.
• On a graph that is already required to be memory-resident as an adjacency structure for other parts of the system (routing, querying neighbors, rendering the topology), DFS/BFS effectively costs nothing extra in memory, since it reuses that existing structure; Union-Find would then be introducing a second, redundant representation purely to get a connectivity answer.

Direct answer

Compute the total steady-state CPU and memory demand from the request rate and per-request footprint, divide each by what a single instance can safely absorb after applying the target utilization and headroom, round each up to a whole instance, and take the larger of the two counts, since whichever resource binds first determines how many instances are actually needed.

Structured elaboration

The given signature has no per-instance capacity input, and an instance count cannot be computed without knowing what one instance can hold. The practical fix is to make that assumption explicit as a documented constant (or an additional parameter in a real implementation) rather than leaving it implicit: this example standardizes the fleet on a 4 virtual-CPU (vCPU), 8,192 MB instance shape.

import math

# Fleet standardizes on one instance shape; documented explicitly because the
# function signature has no per-instance capacity input, and a total resource
# need cannot become an instance COUNT without one.
INSTANCE_CPU_CORES = 4
INSTANCE_MEM_MB = 8192


def compute_instances(requests_per_sec, cpu_per_request, mem_per_request_mb,
                       cpu_target_util, mem_headroom_pct):
    # Total steady-state resource demand (treat each request as holding its
    # footprint for about one second of occupancy, the standard back-of-
    # envelope simplification for this kind of sizing).
    total_cpu_cores = requests_per_sec * cpu_per_request
    total_mem_mb = requests_per_sec * mem_per_request_mb

    # Usable capacity per instance after applying the target ceiling and headroom.
    usable_cpu_per_instance = INSTANCE_CPU_CORES * cpu_target_util
    usable_mem_per_instance = INSTANCE_MEM_MB * (1 - mem_headroom_pct)

    instances_for_cpu = math.ceil(total_cpu_cores / usable_cpu_per_instance)
    instances_for_mem = math.ceil(total_mem_mb / usable_mem_per_instance)

    # The binding constraint wins; round UP because a fractional instance
    # cannot be provisioned and under-sizing the tighter resource defeats
    # the point of the target and headroom inputs.
    return max(instances_for_cpu, instances_for_mem)


if __name__ == "__main__":
    n = compute_instances(
        requests_per_sec=500,
        cpu_per_request=0.05,
        mem_per_request_mb=20,
        cpu_target_util=0.60,
        mem_headroom_pct=0.20,
    )
    print(f"total CPU cores needed: {500 * 0.05}")
    print(f"total memory needed: {500 * 20} MB")
    print(f"usable CPU per instance: {INSTANCE_CPU_CORES * 0.60}")
    print(f"usable memory per instance: {INSTANCE_MEM_MB * 0.80}")
    print(f"instances required (CPU-bound path): {math.ceil((500*0.05)/(INSTANCE_CPU_CORES*0.60))}")
    print(f"instances required (memory-bound path): {math.ceil((500*20)/(INSTANCE_MEM_MB*0.80))}")
    print(f"compute_instances(500, 0.05, 20, 0.60, 0.20) = {n}")

Output when run as shown:

total CPU cores needed: 25.0
total memory needed: 10000 MB
usable CPU per instance: 2.4
usable memory per instance: 6553.6
instances required (CPU-bound path): 11
instances required (memory-bound path): 2
compute_instances(500, 0.05, 20, 0.60, 0.20) = 11

Worked example

At 500 requests/sec, 0.05 CPU cores/request implies 25 total CPU cores of demand; 20MB/request implies 10,000MB of total memory demand. Each instance offers 4 x 0.60 = 2.4 usable cores at the 60% target, needing ceil(25/2.4) = 11 instances on the CPU path. Each instance offers 8,192 x 0.80 = 6,553.6MB of usable memory at 20% headroom, needing ceil(10,000/6,553.6) = 2 instances on the memory path. The function returns max(11, 2) = 11: this workload is CPU-bound, not memory-bound, so the memory headroom target is easily satisfied once enough instances are provisioned for CPU.

Complexity

Constant time and constant space, O(1). The function does a fixed handful of multiplications, two divisions, two ceilings, and a max, none of it scales with the input values themselves.

Edge cases

A cpu_target_util of 0 or a mem_headroom_pct of 1 makes the corresponding usable-capacity term zero, a division by zero; the function should validate 0 < cpu_target_util <= 1 and 0 <= mem_headroom_pct < 1 before dividing, and raise a clear error rather than let a stray configuration value crash the sizing job. A requests_per_sec of 0 makes both totals zero and the function returns 0 instances; whether that is correct depends on whether the service must stay warm with zero traffic, worth a deliberate max(1, ...) floor if so, not left as an accidental side effect of the arithmetic. Negative inputs (a negative headroom percentage typed by mistake) are not validated here either and would silently produce a nonsensical negative usable capacity; a production version should reject them explicitly.

Trade-offs and pitfalls

Rounding down anywhere in this calculation would leave the binding resource under-provisioned; every division that turns a resource total into an instance count must round up.

A fleet with a fixed instance shape hides the real trade-off between few large instances and many small ones; in production this function should take the instance's own CPU and memory capacity as parameters instead of a hardcoded constant, so it can be reused across instance families.

Little's-Law-style sizing (Little's Law states that the average number of items in a system equals the arrival rate times the average time each item spends in the system; here simplified to treating each request as occupying its footprint for about one second) is a simplification. A workload with request durations far from one second, very fast health checks or very slow long-polling requests, needs the actual average request duration folded into the formula rather than assumed away.

Direct answer: A production-grade transactional outbox service has three pieces: the application's database with an outbox table written in the same local transaction as the business change, a relay/reader that publishes unpublished rows to the broker (polling or CDC-based, CDC meaning change-data-capture), and enough retry/dedup discipline on both sides that the whole thing delivers at-least-once without ever silently losing an event.

Structured elaboration

Outbox table schema. At minimum: id (primary key, also usable as an idempotency/dedup token), aggregate_id (what business entity this event is about, often used as the broker partition key to preserve per-entity ordering), event_type, payload (serialized event body), created_at, and published (boolean, or a published_at timestamp, null until published). An index on (published, created_at) supports the "give me the oldest unpublished rows" query the reader needs.

Reader design: polling vs streaming. A polling reader periodically queries for published = false rows, simple to build and reason about, but adds latency bounded by the poll interval and puts a steady, small query load on the database. A CDC-based (streaming) reader instead taps the database's write-ahead log or replication stream (e.g. via Debezium) to be notified of new outbox rows as they're written, giving near-real-time delivery with no polling load, at the cost of running and operating CDC infrastructure and a slightly more complex failure story (the CDC connector itself can fall behind or need to resume from a checkpoint).

Ordering guarantees. If per-entity ordering matters (e.g. all events for order 501 must be delivered in the order they were created), the reader must publish using aggregate_id as the partition/routing key, so a partitioned broker (like Kafka) preserves order within that key even while allowing parallelism across different keys. Global ordering across all entities is usually not worth the throughput cost and isn't needed by most consumers.

Idempotency handling on the consumer side. Since the relay guarantees only at-least-once delivery, every event's payload includes a stable ID (the outbox row's id, or a domain-level event ID) that consumers use to deduplicate, typically via a small "seen event IDs" store with a bounded retention window matched to how long redelivery could realistically be delayed.

Scaling the reader for high throughput. A single polling reader eventually becomes a bottleneck; horizontal scaling requires either sharding the outbox table (e.g. by a hash of aggregate_id) with one reader instance per shard, or, for a CDC-based reader, letting the CDC connector's own partitioning handle parallelism. Batching reads and publishes (rather than one row per query/publish round-trip) is usually the first and cheapest throughput lever before reaching for sharding.

Worked example. An orders platform doing 2,000 writes/sec needs its outbox reader to keep pace. A single polling reader batching 200 rows per poll every 200ms caps out at 200 / 0.2s = 1,000 rows/sec, below the required rate regardless of how the rest of the system is tuned. Moving to a CDC-based reader (Debezium tailing the outbox table's WAL, write-ahead log) removes the poll-interval ceiling entirely, since new rows are picked up as they're written to the log rather than on a fixed cadence, at the cost of adding a Kafka Connect cluster and its own on-call burden that didn't exist with the simpler polling design; the actual sustained throughput and delivery latency the CDC path achieves would need to be measured against the real database's write-ahead log volume and Kafka Connect's own configured parallelism, not assumed from the pattern alone.

Trade-offs and pitfalls. Teams often under-invest in the outbox table's own cleanup: published rows accumulate forever unless there's a retention/archival job, which eventually degrades both the outbox query performance and, for CDC-based readers, the size of the write-ahead log the CDC connector has to process on any resume-from-checkpoint scenario.

Direct answer

In the first week, stop the bleeding and claim the system as yours in writing; over the following weeks and months, work through a small number of phases, understand failure patterns, fix the worst recurring one, then build back tests, monitoring, and documentation, each with a milestone you can show and a number that proves things are actually improving.

Structured elaboration

• First week, stabilize and establish ownership: pull the history of recent incidents or failures to find the actual pattern, not the loudest anecdote, put even crude monitoring or alerting in place if none exists, and write a short doc stating what the system does, that you now own it, and what the plan is; send that to stakeholders so "who owns this" stops being a question.
• The following weeks, phased remediation with milestones: a first phase of roughly the next 3 weeks fixes the single highest-frequency failure mode, the quick win that buys you credibility and breathing room for the rest of the plan; a second phase of roughly the next 4 weeks adds test coverage and documentation for the core paths people actually depend on daily, not full coverage everywhere; a third phase covering the remaining weeks up to about 3 months hardens the rest, removes workarounds people have built to route around the system's flakiness, and puts a real runbook in place.
• Measuring progress: track a simple, visible leading indicator, most naturally failures or incidents per week, from a clear baseline, so stakeholders see a trend line rather than taking your word for "it's better now."

Worked example

I inherit a system with a baseline of about 6 disruptive failures a week and no documentation of why. Week 1: reviewing the last 2 months of incident history shows over half of them trace back to one specific recurring cause; I add a basic alert on that specific failure mode so at least we're notified instead of surprised, and send a short ownership note to stakeholders. By week 4, end of phase 1, the highest-frequency failure mode is fixed, and the weekly failure count drops from about 6 to about 3. By week 8, end of phase 2, core-path tests and basic documentation are in place, and the count drops further to about 1 a week. By week 12, end of the roughly 3-month plan, the remaining workarounds are removed and a runbook is in place, and the system is stable at close to 0 failures a week, down from the original baseline of 6.

Trade-offs and pitfalls

A common mistake is spending the first week building the perfect long-term architecture instead of just stopping the most disruptive recurring failure and telling people you own it; credibility comes from visible short-term progress, not a beautiful plan no one has seen work yet. Another failure is skipping the ownership communication step, leaving stakeholders unsure whether problems are still being routed to the previous owner or a whole team. Watch also for declaring success once the failure count drops without also removing the manual workarounds people built around the old flakiness; those workarounds often hide the next failure mode until you take them away.