FAANG-Standard Interview Preparation Guide for Junior SRE

Trace and span

A trace is the full record of one request's journey through a distributed system, everything that happened to serve it end to end. A trace is made up of spans: each span is one unit of work, like a single RPC call or a single database query, with its own start time, end time, name, and metadata (tags/attributes).

Parent-child relationships

Spans form a tree (or more generally a DAG): a span is the child of whichever span invoked it. For example, an API gateway's span is the parent of the auth-service span it calls, which might itself be the parent of a database-query span. That tree structure is what makes a trace useful beyond "the total time was X": it shows whether work happened sequentially (one span starts only after its parent's other children finish) or in parallel (several child spans overlap in time), which is exactly what you need to reason about where wall-clock time actually went.

Why context propagation matters

Context propagation is passing the trace's identifying information (a trace ID, the current span ID, and the sampling decision) along with the request as it crosses process and network boundaries, typically as HTTP headers (like the W3C traceparent header) or gRPC metadata. Without it, each service only knows about its own local work; there's no way to stitch five separate services' telemetry back into one coherent trace of a single user request.

Common sampling strategies

• Head-based sampling: decide whether to keep a trace right at the start of the request (e.g., keep a random 1%). Cheap and simple, but you're gambling before you know whether the request turns out to be interesting.
• Tail-based sampling: buffer the full trace and decide whether to keep it only after it completes, so you can apply rules like "always keep traces with errors or above a latency threshold." Much better at catching rare, interesting events, but requires a collector that can buffer and hold spans before deciding.
• Adaptive/rate-limited sampling: target a roughly fixed volume of kept traces per second regardless of traffic swings, so sampling doesn't silently drop to near-zero during a traffic spike or flood a backend during a burst.

What tracing helps answer

Which specific hop a regression is coming from, why one particular slow request was slow (as opposed to just "the p99 is high"), whether time was spent sequentially or in parallel, and where a request spent idle/waiting time versus actually doing work.

Trade-offs of high sampling rates and high-cardinality data

Storage and ingestion cost scales roughly with how many traces (and how much detail per trace) you keep, so sampling everything at high fidelity gets expensive fast. High-cardinality attributes on spans (like a full URL with query parameters, or a raw user ID as a tag) multiply that cost further, since backends often index on tag values, and a tag with millions of distinct values blows up that index. The practical trade-off is: sample richly enough (often via tail-based sampling specifically for errors and slow traces) to reliably catch the rare tail events you actually care about, without paying full cost to store the enormous volume of unremarkable, fast, successful traces that make up the bulk of traffic.

Direct answer
At high incident volume, right-sizing postmortem effort means reviewing incidents proportionally to their severity and learning value rather than giving every incident the same heavyweight treatment, since a full deep-dive on every minor blip both burns out reviewers and dilutes attention from the incidents that actually deserve it.

Structured elaboration

• Tier the review depth by severity and novelty. High-severity or novel-pattern incidents get the full treatment: timeline reconstruction, root cause and contributing factors, cross-team facilitation. Low-severity, well-understood, or clearly one-off incidents get a much lighter 'mini' review: a short written summary with a root cause and, if warranted, one action item, no meeting required.
• Rotate reviewers rather than relying on the same few people. Concentrating review responsibility on a small group both burns them out and creates a bottleneck; distributing it (with a shared template and light training) keeps quality consistent while reducing individual load.
• Automate triage where the pattern is well understood. If a category of incident has occurred many times with the same known cause, an automated or templated mini-postmortem that flags it as a known, tracked pattern (rather than requiring fresh analysis every time) frees up reviewer time for genuinely novel incidents.
• Track a pattern-level view, not just per-incident. A large volume of small, similar incidents is itself a signal worth its own dedicated (heavier) review, even if none of them individually crossed the severity threshold for a full postmortem, since the aggregate pattern is often more informative than any single instance.
• Measure whether this is actually preserving learning value, not just reducing workload. Track whether recurrence rates for previously-reviewed incident classes stay flat or improve even as review depth for minor incidents drops, to confirm the lighter-touch approach isn't quietly letting real risk go unaddressed.

Worked example
An organization runs roughly 2,000 incidents a month and full postmortems have become a rubber-stamp exercise nobody has time to do well. The fix: define three tiers. Tier 1 (high severity or genuinely novel pattern, maybe 5% of incidents) gets a full facilitated postmortem within a defined turnaround. Tier 2 (moderate severity, somewhat familiar pattern, maybe 25%) gets a lightweight async writeup by the on-call responder, reviewed by a rotating peer within a week, no live meeting required unless something surprising surfaces. Tier 3 (low severity, well-understood and recurring pattern, the remaining ~70%) gets an automated, templated log entry tagging the known category, with no individual analysis required unless the volume of that specific category spikes, which triggers escalation to a full pattern-level review. Reviewer rotation is enforced across teams so no single person is doing more than a defined share of Tier 1 and Tier 2 reviews in a given month.

Trade-offs and pitfalls
The biggest risk of this approach is under-reviewing something that seemed minor in isolation but was actually an early instance of a bigger, developing problem; the pattern-level tracking (watching for a spike in a normally-quiet Tier 3 category) is what catches that, and skipping it is the most common mistake when teams implement tiering purely to save time.

Don't gate sensitive steps behind standing credentials a responder already holds. Gate them behind short-lived, narrowly scoped credentials issued by the runbook orchestrator at the moment of use, with a pre-authorized fast path for the highest severities so speed during a real incident doesn't require quietly bypassing the audit trail.

Comparing access models

Worked example: sizing the credential TTL

If the median observed time to complete a given remediation step across past incidents is 12 minutes, a 15-minute TTL leaves almost no margin:

a responder who hits a snag is interrupted 3 minutes short of done, exactly when stopping is most disruptive. A TTL of roughly 20 minutes, the median plus a working buffer rather than an open-ended grant, gives room to finish without leaving a long-lived credential outstanding. A 4-hour TTL "to be safe" instead means a credential compromised from a responder's terminal during that window stays valid for the rest of the shift; that's the trade being made for the extra convenience.

Keeping it auditable without slowing the responder down

• Runbooks reference secret IDs, never raw values, so the document itself is safe to read even if it leaks.
• The orchestrator executes the sensitive step server-side where practical, so the responder never sees the decrypted secret at all, only the outcome.
• Every credential issuance logs identity, ticket or incident ID, scope, and TTL to an immutable log, correlated automatically rather than reconstructed after the fact.
• The highest severities get pre-authorized issuance, no waiting on a human approver, precisely because the TTL and logging, not a manual gate, are what keep it auditable.

Trade-offs and pitfalls

A break-glass path needs more audit rigor than the normal path, not less; pair any emergency bypass with mandatory post-incident review and automatic rotation of whatever it touched. Auto-approval for the highest severities removes a human gate exactly during the highest-risk window (a real incident, adrenaline, and possibly an actor exploiting the chaos), so the TTL and logging have to carry that weight instead. Orchestrator-executed remediation is safer for the responder but adds its own risk surface; the automation itself now needs the same change-review rigor as production code, not less because "it's just a script."

Direct answer

Talk to each person privately before bringing them together, so you understand the facts and stakes from each side without an audience. Then classify the conflict as substantive (a genuine disagreement about the right call) versus interpersonal (friction dressed up as a substantive disagreement), because each needs a different resolution path. Bring them together around a shared goal and concrete evidence, not around who's right, and if it's genuinely undecidable in the room, use a time-boxed way to get more evidence rather than let the standoff continue to block the team.

Resolution framework

Never mediate cold in a group. Talk to each person separately first. You're listening for their read of the facts, what they think is at stake, and what "winning" would actually look like to them. This also surfaces things people won't say in front of the other person.

Classify before you intervene. A disagreement that looks technical or process-based on the surface is sometimes substantive and sometimes really about communication style or unresolved friction. Treating an interpersonal conflict as if it just needs more evidence wastes everyone's time; treating a real substantive disagreement as if it just needs better feelings management does too.

Reframe the joint conversation around the shared goal. Ask both people directly what evidence would change their mind. This shifts the conversation from defending a position to examining what's actually true, and it's a useful tell: someone who can't answer that question may be more attached to being right than to the outcome.

Use a time-boxed way to break a genuine deadlock. If the disagreement is real and evidence-based but neither side has enough information to concede, propose a small, bounded experiment or spike to settle it rather than let the argument continue indefinitely. If there's no time for that, make the call yourself and say plainly that you're doing so.

Follow through explicitly. Name who owns the resulting decision, document it somewhere durable, and check back in later to make sure the resolution actually held rather than just went quiet.

Worked example

Two people you mentor are at an impasse over a decision, and delivery is now stalled because of it. Separate conversations reveal the disagreement is mostly substantive, but there's real interpersonal friction layered on top, one of them has started talking over the other in shared meetings. You facilitate a joint session with explicit ground rules, focused on what evidence would resolve the substantive question, and propose a short timeboxed way to get that evidence rather than debate it further. Separately, and privately, you have a direct conversation with the person who'd been talking over the other about how that was landing on the team, independent of who turns out to be right on the substance.

Trade-offs and pitfalls

Mediating in a group before talking to each person privately risks blindsiding someone and getting performative, professional-sounding answers that hide what's actually going on.

Always pushing for consensus wastes time on disagreements that genuinely don't have a consensus answer. Sometimes the right move is a clean, owned decision, not more discussion.

Making the call yourself resolves the immediate block but has a cost: it can create resentment, or teach people to escalate disagreements to you instead of learning to resolve them with each other, so it's worth being deliberate about when you step in to decide versus when you keep facilitating.

A conflict that keeps recurring in slightly different forms is often a signal of a structural problem, unclear ownership boundaries between the two people, rather than a personality clash, and treating the symptom each time without noticing the pattern means you'll be back here again.

Direct answer. Register one cleanup function with trap cleanup EXIT. The EXIT pseudo-signal fires on every way the shell ends: normal completion, an exit call, a failure under set -e and, in Bash, death from SIGINT (Ctrl-C), SIGTERM (the default signal of kill) or SIGHUP, even when you have set no signal traps (executed on bash 5.2.21: an EXIT-only script that sent itself each of those signals printed its cleanup line and left with status 130, 143 and 129). Add trap 'exit 130' INT and trap 'exit 143' TERM anyway: they turn the signal into an ordinary exit with a documented status, so a parent such as Python's subprocess sees return code 143 instead of -15, and they keep the script correct under shells that do not run the EXIT trap on a signal (executed: dash printed nothing on SIGTERM). The one thing you cannot clean up after is SIGKILL (kill -9), which the kernel delivers without giving the process a chance to run anything.

Terms used in the examples

• Signal. A small numbered message the operating system delivers to a process. SIGINT (2) is what Ctrl-C sends, SIGTERM (15) is what a plain kill PID sends, and SIGKILL (9) cannot be caught or ignored.
• Pseudo-signal. EXIT and ERR are names Bash accepts in trap that are not real operating-system signals: EXIT means "this shell is about to end" and ERR means "a command just failed".
• $$. The process ID of the running script itself, so kill -INT $$ makes the script send itself the signal Ctrl-C would send.
• Subshell. A child copy of the shell, created by ( ... ), a pipeline stage or $( ... ). It has its own copy of the variables and its own exit.
• Process group. The set of processes (the script and the programs it started) that the terminal signals together when you press Ctrl-C.
• set -E (errtrace). Makes the ERR trap apply inside functions and subshells too.

Four hooks appear in the worked example, but only one of them does the cleaning: trap cleanup EXIT. The INT and TERM lines turn those signals into an ordinary exit with a recognisable status (Bash would run the EXIT trap for them anyway, as the direct answer shows), and the ERR line only logs which command failed. A script with just the EXIT line already cleans up after normal ends, exit calls, set -e failures and, in Bash, SIGINT, SIGTERM and SIGHUP.

Which hook for which event

The exit status inside and after the trap

At the moment the EXIT trap starts, $? still holds the status the script was exiting with, so read it first (local rc=$?). The trap's own commands then run, and when the trap finishes the script exits with the original status, unless the trap itself calls exit N, which replaces it. The safe pattern is to capture rc, clean up, and exit "$rc". Clear the traps inside the handler (trap - EXIT INT TERM) so a second signal during cleanup does not re-enter it.

Worked example

#!/usr/bin/env bash
set -Eeuo pipefail

tmpdir=$(mktemp -d)
cleanup() {
  local rc=$?                       # status the script was exiting with
  trap - EXIT INT TERM              # avoid re-entry while cleaning up
  rm -rf -- "$tmpdir"
  echo "cleanup: removed tmpdir, original status $rc"
  exit "$rc"                        # preserve it
}
trap cleanup EXIT
trap 'exit 130' INT                 # 128 + SIGINT(2); EXIT trap then runs
trap 'exit 143' TERM                # 128 + SIGTERM(15)
trap 'echo "ERR trap: line $LINENO: $BASH_COMMAND" >&2' ERR

case ${1:-ok} in
  ok)     echo "work done" ;;
  fail)   false; echo "not reached" ;;
  int)    kill -INT  $$; sleep 1; echo "not reached" ;;
  term)   kill -TERM $$; sleep 1; echo "not reached" ;;
  sub)    ( exit 5 ) || echo "subshell left with $?, parent cleanup has not run yet"
          ( trap 'echo "subshell own EXIT trap"' EXIT; exit 6 ) || echo "subshell left with $?" ;;
  func)   f() { false; }; f; echo "not reached" ;;
esac

Each mode, run separately (kill -INT $$ sends the signal to the script itself, which is what Ctrl-C does to a foreground script):

$ bash cleanup.sh ok
work done
cleanup: removed tmpdir, original status 0
exit=0
$ bash cleanup.sh fail
ERR trap: line 19: false
cleanup: removed tmpdir, original status 1
exit=1
$ bash cleanup.sh int
cleanup: removed tmpdir, original status 130
exit=130
$ bash cleanup.sh term
cleanup: removed tmpdir, original status 143
exit=143
$ bash cleanup.sh sub
subshell left with 5, parent cleanup has not run yet
subshell own EXIT trap
subshell left with 6
cleanup: removed tmpdir, original status 0
exit=0
$ bash cleanup.sh func
ERR trap: line 24: false
cleanup: removed tmpdir, original status 1
exit=1

Note that with set -E the ERR trap also fires inside functions (func case, line 24), and that the cleanup ran exactly once per script even though three traps were registered.

Reading the sub case: ( exit 5 ) ends only the subshell, so $? becomes 5 while the script itself keeps running, and the script's cleanup has not fired, because it fires when the parent shell ends. This matters for ownership: the temp directory belongs to the parent, so the trap that removes it must be registered by the parent. A trap registered inside a subshell runs when that subshell ends, which would delete the directory while the parent still needs it. In the second line the subshell registers its own EXIT trap, which fires at its own exit (subshell own EXIT trap).

Subshells and functions

• Subshells. A subshell (( ... ), a pipeline stage, $( )) resets caught traps to the default. The parent's EXIT trap does not run when the subshell exits: in the sub case, ( exit 5 ) returned and the parent's cleanup had not yet run. A subshell may set its own trap, as the second line shows. Put the cleanup in the process that owns the resource.
• Functions. Traps are process-wide, not function-scoped. An EXIT trap set inside a function replaces the script's one and fires at script exit, not when the function returns. (Checked: a trap set in f was still present, and ran, after f returned.)
• ERR trap and set -E. Without -E (errtrace), the ERR trap is not inherited by shell functions, command substitutions or subshells. With the trap set and set -e but no -E, a failing false inside a function exits the script with status 1 and the ERR trap never prints. Adding -E makes it print ERR trap fired: false first (both variants run, exit status 1 in each).

Timing traced with numbers

SECONDS is a Bash variable holding the whole seconds since the script started. Two scripts, each sent SIGTERM from outside one second after they start (SIGTERM is used so the demo can run in the background; any trapped signal follows the same rule):

# foreground.sh                        # background-wait.sh
trap 'echo "TERM trap ran at ${SECONDS}s"; exit 143' TERM
sleep 3                                 sleep 3 & wait $!

-- foreground.sh
TERM trap ran at 3s
-- background-wait.sh
TERM trap ran at 1s

The signal arrived at second 1 in both runs. With sleep 3 in the foreground, Bash held the trap until that command finished, at second 3, a two-second delay. With sleep 3 & followed by wait $!, the wait builtin is interrupted by the signal, the trap ran at second 1, and the script exited at once. A 10-minute child would delay the trap by up to 10 minutes in the first form.

Timing, trade-offs and pitfalls

• Bash runs a trap only after the current foreground command finishes. If the script is inside sleep 3 and receives SIGINT after 1 second, the INT trap runs at 3 seconds (measured), whereas the wait builtin returns at once. For long children, run them in the background and wait, or forward the signal to the child's process group.
• Ctrl-C goes to the whole foreground process group, so the child also gets SIGINT; your handler cleans the script's resources, not the child's.
• The same trap on EXIT for a temp directory is safer than putting rm -rf at the end of the script: use mktemp -d and rm -rf -- "$tmpdir" with the variable quoted and set before the trap is registered.
• SIGKILL and power loss leave debris. For resources that must not leak (locks, mounts, temp databases), also use a startup sweep, or a lock that the kernel releases when the process dies (flock).
• Give the INT and TERM traps an explicit status (exit 130, exit 143) so callers can tell an interrupted run from a failed one.

Direct answer

Correctness has to win outright for balance mutations: shard accounts across a consensus group (Raft is the standard choice for its operational clarity) with a single elected leader per shard, and require every write to reach a majority quorum of replicas before it's acknowledged as durable. Availability pays for this in write latency, since a quorum commit has to wait for the slowest replica in the majority, not just the leader, and during a leader failure there's a real (bounded, but nonzero) window where that shard's writes are unavailable while a new leader is elected. The way to minimize the damage is to make that window as short as possible through fast leader election and leader leases, and to make the latency cost as small as possible through deliberate quorum placement, not by weakening the consistency guarantee itself.

Architecture

flowchart TD
    LB[Global load balancer] --> A1[Region A: 2 replicas, leader here]
    LB --> B1[Region B: 2 replicas]
    LB --> C1[Region C: 1 replica]
    A1 --> RAFT[Raft group, N=5, quorum=3]
    B1 --> RAFT
    C1 --> RAFT
    RAFT --> LEDGER[Append-only transaction log]

Accounts are partitioned by account ID using consistent hashing (a hashing scheme where each account maps to a position on a ring of shards, chosen so that adding or removing a shard only reshuffles a small fraction of accounts instead of nearly all of them) into shards; each shard is its own independent Raft group with its own leader, so write throughput scales by adding shards rather than being bottlenecked by one global leader. Client writes route to the shard's current leader; reads can be served from the leader for strict linearizability (every read reflects the most recent completed write, as if all operations happened one at a time in real-time order), or from a follower with a leader-issued lease for bounded-staleness reads that don't need to pay quorum latency.

Deriving the quorum

For a Raft group of $N$ replicas, a write commits once it's replicated to a majority:

For $N = 5$:

The number of simultaneous replica failures the group tolerates while still having a live quorum is:

Placement and latency

Place the 5 replicas as 2 in the leader's home region (Region A), 2 in a second region (Region B), and 1 in a third region (Region C). This placement matters for two separate reasons. First, region-level fault tolerance: losing Region A or Region B (2 replicas each) leaves exactly 3 replicas, which is exactly quorum, so the group survives the loss of any single region without losing write availability, precisely because no single region holds more than $F_{\text{tolerated}} = 2$ replicas. If any region instead held 3 replicas, losing that region alone would drop the group below quorum.

Second, latency: because Raft only needs a majority, not all 5, the leader can commit as soon as 2 of the 4 followers acknowledge (its own copy plus 2 acks makes 3, quorum). The fastest 2 acks available are the local Region-A follower (low latency, same region) and whichever of Region B or Region C responds first:

With an assumed same-region RTT of 2ms and cross-region RTTs of 70ms (A to B) and 90ms (A to C):

The commit is gated by the faster of the two remote regions, not the slower one, and not by all four followers, which is the concrete reason majority quorum beats "wait for everyone": a design that required all 5 replicas to ack would pay the 90ms worst-case RTT on every write instead of the 70ms best-of-two.

Leader failover

On leader loss, followers detect the missing heartbeat after an election timeout and hold a new election; the group is unavailable for writes to that shard for roughly the election-timeout-plus-one-round-trip window, which is why election timeouts are tuned as short as reliably possible without causing spurious elections from ordinary network jitter. A leader lease (the leader holds exclusive write authority for a short renewable period) lets followers serve bounded-staleness reads without contacting the leader at all, and automated leader transfer for planned maintenance (the outgoing leader hands off cleanly before stepping down) avoids paying the full election-timeout cost for maintenance events, which are predictable and shouldn't cost the same as an unplanned crash.

Cross-shard transfers

A single-shard write (both accounts in a transfer live in the same shard) is a normal Raft-committed entry, no special handling needed. A cross-shard transfer needs coordination between two independent Raft groups: a coordinator writes a prepare entry to both shards' logs, and only commits the transfer once both shards' local quorums have accepted the prepare; if either shard fails to prepare within a timeout, the coordinator writes a compensating abort entry to whichever shard did succeed, rather than leaving the transfer half-applied. This is deliberately simpler than a general two-phase-commit protocol across arbitrary participants, because both sides are already individually linearizable Raft logs, so the coordinator's job is narrowly "get both to agree" rather than solving distributed consensus from scratch.

Trade-offs & pitfalls

The quorum design directly trades latency for correctness: every write pays at least the RTT to the nearest remote region in the placement, which is unavoidable if you actually want the durability guarantee (a write acknowledged locally-only in Region A could be silently lost if Region A is the one that fails), and no client-side retry logic fixes a fundamentally under-replicated design. A common mistake is treating replica count as the only lever, when placement matters just as much: 5 replicas concentrated so that one region holds 3 gives you the same $N=5$, $Q=3$ numbers on paper, but a single region failure then drops the group below quorum, which defeats the entire point of geographic redundancy even though nothing about the Raft math itself is wrong. Idempotency and fencing tokens are non-negotiable additions on top of the consensus layer: a Raft term number rejects stale writes from a leader that's since been superseded, and a client-supplied idempotency key prevents a transaction from being double-applied if a client retries after a timeout that turned out to be a lost acknowledgment rather than a real failure, not a rejected write.

Review checklist

• Identify large temporaries and peak memory sites
• Prefer in-place ops or fused kernels (a fused kernel is a single compiled operation that does several arithmetic steps in one pass over the data, e.g. a*b+c computed in one native loop, instead of allocating a separate full-size temporary array after every individual step)
• Use dtype minimization (float32 vs float64)
• Avoid creating many intermediate arrays in chained expressions
• Ensure readability and add comments explaining optimizations

Code suggestions

• Replace chained ops with single expression using NumPy ufuncs (ufunc, short for universal function: a NumPy function like np.exp or np.multiply that applies element-by-element across an array in a single compiled pass) or out= parameter (out=: tells the ufunc to write its result directly into an existing array's memory instead of allocating a brand-new array for the result):

# bad: many temporaries
a = np.exp(x)
b = a * w
c = np.where(b>0, b, 0)

# better: reuse buffers
tmp = np.empty_like(x)   # allocate the reusable buffer once, up front
np.exp(x, out=tmp)
np.multiply(tmp, w, out=tmp)
np.maximum(tmp, 0, out=tmp)

The tmp = np.empty_like(x) line is what makes the "better" version actually runnable: every out=tmp call below it writes its result back into that same pre-allocated buffer instead of allocating a fresh array each time, which is the entire memory saving, a, b, and c in the "bad" version are each a full new array; tmp in the "better" version is one array, reused three times.

Worked example, verified on CPython 3.12, confirming the rewrite produces identical values and showing the memory difference concretely with tracemalloc:

import numpy as np
import tracemalloc

x = np.array([1.0, -2.0, 3.0, -4.0])
w = np.array([0.5, 0.5, 0.5, 0.5])

def bad(x, w):
    a = np.exp(x)
    b = a * w
    c = np.where(b > 0, b, 0)
    return c

def better(x, w):
    tmp = np.empty_like(x)
    np.exp(x, out=tmp)
    np.multiply(tmp, w, out=tmp)
    np.maximum(tmp, 0, out=tmp)
    return tmp

print(np.allclose(bad(x, w), better(x, w)))
# True

tracemalloc.start()
_, before = tracemalloc.get_traced_memory()
for _ in range(1000):
    bad(x, w)
_, peak_bad = tracemalloc.get_traced_memory()
tracemalloc.stop()

tracemalloc.start()
_, before2 = tracemalloc.get_traced_memory()
for _ in range(1000):
    better(x, w)
_, peak_better = tracemalloc.get_traced_memory()
tracemalloc.stop()

print("bad peak > better peak:", peak_bad > peak_better)
# True: bad() allocates 3 new arrays (a, b, c) per call; better() allocates 1 (tmp) and reuses it

Both functions agree on every value (np.allclose is True); the peak-memory comparison confirms, directly, that bad() allocates strictly more per call, three fresh arrays (a, b, c) versus better()'s single reused buffer, which is the mechanism this checklist item is about, not just a naming convention.

• Use NumExpr (evaluates a whole array expression like a*b+c in one call without materializing each intermediate array) or Numba (compiles a Python function to machine code the first time it runs) for large elementwise chains to reduce temporaries.
• Use memory-mapped arrays (np.memmap, which lets an array live on disk and be accessed in pieces instead of fully loaded into RAM) for very large datasets.

Additional advice

• Add benchmarks demonstrating memory/regression
• Add comments and keep variable names meaningful
• Consider chunked processing for very large arrays to keep memory bounded

Direct answer
Catch an exception locally only when you can actually do something useful with it there (retry, fall back, translate it into a domain-specific error); otherwise let it propagate. Create a custom exception type when the caller needs to distinguish this failure from others programmatically; reuse a generic one when nobody downstream will branch on it.

Structured elaboration

• The 'can I do something' test: at each layer, ask whether you have enough context AND enough authority to resolve the failure. A repository layer that hits a broken connection can retry once; it usually can't decide whether the caller should show an error page or silently use cached data, so it should propagate (possibly wrapped) rather than swallow.
• Custom vs generic exceptions: define a custom type when a caller three layers up needs to branch (except PaymentDeclinedError: offer another card vs except PaymentGatewayTimeout: retry). A generic RuntimeError or bare Exception is fine for truly unexpected states nobody is meant to handle specially, but never use a bare except Exception: as a catch-all that discards the distinction.
• Avoiding pitfalls: never catch-and-discard (except Exception: pass) without at minimum logging with full context; never catch a broad type just to re-raise the same thing unchanged (that's dead code that hides which specific failures the layer actually expects); when you DO catch to translate, preserve the original exception as the cause (raise DatabaseError(...) from e in Python, wrapping in Java) so a debugger can still walk to the root cause. Also avoid duplicating log lines at every layer: if every layer that catches-and-rethrows also logs the same exception on its way up, one real failure produces several near-identical log entries by the time it reaches the top, which pages on-call with noisy duplicates and makes the true error rate impossible to read off log volume. The fix is to log ONCE, at the layer that either fully handles the error or is the outermost boundary (the request handler, the job runner), and let every intermediate layer that only wraps-and-rethrows do so silently, relying on the exception chain (from e) rather than a fresh log line to preserve context.

Worked example
Service-to-service call: a UserService.get_profile(id) calls an HTTP client that can raise ConnectionError or HTTPStatusError. UserService catches both, and translates HTTPStatusError with status 404 into its own UserNotFoundError (a real domain concept the caller branches on), but lets a 500 or ConnectionError propagate wrapped as UserServiceUnavailableError from e (preserving the cause) since it has no way to fix an unreachable dependency itself; it does NOT log at this layer, since it's not the final decision point.
Database operation: a repository's save(record) catches a unique-constraint violation and translates it to DuplicateRecordError (a business-meaningful case the caller likely wants to show the user), but lets a connection-pool-exhausted error propagate unmodified up to the request-handling layer, which is the only layer with enough context to decide 'shed load' vs 'retry', and is also the only layer that logs it.

Trade-offs and pitfalls
Over-translating (wrapping every low-level exception in a new custom type at every layer) produces an exception-type explosion that is as hard to reason about as no typing at all, and it can silently lose the original stack trace if you forget to chain the cause. Under-translating (letting raw library exceptions like ConnectionError leak all the way to a UI layer) forces every caller to know implementation details of a dependency two layers down. The right amount of translation happens at trust/ownership boundaries (crossing into a new bounded context, crossing a network call, crossing into user-facing code), not at every function call, and the same discipline applies to logging: log at the boundary that owns the decision, not reflexively at every catch block.

An Ingress object declares what HTTP(S) routing should happen (hostnames, path rules, which Service backs each path, which TLS secret to use); an Ingress Controller is the running component that watches Ingress objects and actually implements that routing on real infrastructure. Kubernetes splits these into two objects for the same reason it splits a Deployment's spec from the controller that reconciles it: the API object stays a stable, portable declaration, while the implementation (which varies enormously between environments) is free to be swapped out without changing what you wrote.

What the Ingress object itself declares

An Ingress is pure declaration, no execution: a list of host/path rules, each pointing at a backend Service and port, plus optional TLS configuration referencing a Secret containing a certificate and key. It has no opinion about how that routing gets enforced; by itself, an Ingress object sitting in the API server does nothing.

What the Ingress Controller does with it

The controller (for example nginx-ingress, Traefik, or a cloud provider's own controller such as GKE's or AGIC for Azure) watches for Ingress objects and translates the declared rules into a real, running configuration: it opens listener ports, terminates TLS using the referenced Secret, and applies the host/path routing logic, typically by configuring an underlying proxy (like nginx) or provisioning a cloud load balancer.

Why split the two, contrasted with a Service of type LoadBalancer

A Service of type LoadBalancer works at L4 (Layer 4, the TCP/UDP transport layer): it exposes one Service on one IP and port, with no visibility into HTTP hostnames or paths. Ingress works at L7 (Layer 7, the application layer): it can route many hostnames and paths to many different backend Services through a single entry point, but only because something understands HTTP well enough to look inside the request to make that decision, which is exactly the controller's job.

Why split the responsibility this way

• No single "correct" implementation exists. Different environments need genuinely different routing implementations (self-hosted nginx, a cloud provider's managed load balancer, a service mesh's gateway); a single hardcoded Ingress-to-infrastructure mapping baked into the API server couldn't serve all of them.
• The API object stays stable across environments. The same Ingress manifest can move between a local cluster running nginx-ingress and a cloud cluster running a managed controller, changing only which controller is installed, not the application team's YAML.
• It matches Kubernetes' general controller pattern. Objects declare desired state; a controller reconciles that state against reality. Ingress is just this pattern applied to L7 routing, the same separation Kubernetes already uses everywhere else (a Deployment declares desired replica state, the Deployment controller makes it real).

Trade-offs and pitfalls

• Because the controller does the real work, its feature set and its vendor-specific annotations determine what's actually possible; two clusters running different controllers can behave differently from the exact same Ingress YAML, which undermines the portability the split is meant to provide unless you stick to well-supported, portable fields.
• No Ingress Controller installed means Ingress objects sit inert; a common early mistake is writing correct Ingress rules and then wondering why nothing routes, when the real gap is a missing or misconfigured controller.
• Cross-cutting concerns that need to apply before traffic even reaches a specific controller instance (global rate limiting, a shared web application firewall) are often better handled at an edge layer in front of the cluster rather than pushed entirely into per-Ingress annotations.

Direct answer

Template-driven configuration (Jinja2 rendering into plain config files) and a declarative configuration-management tool (Ansible) both PREVENT drift better than raw ad hoc scripts, but they prevent it in different ways with different failure modes: Jinja2 templating guarantees the RENDERED file is internally consistent with its inputs, but nothing enforces that the rendered file is what is ACTUALLY DEPLOYED on a given server unless something ELSE (a separate push/apply mechanism) also runs reliably; Ansible's own idempotent modules provide BOTH the rendering AND the enforcement (a template task run against a server directly asserts and corrects that server's actual state), closing that gap the templating-alone approach leaves open.

Structured elaboration

Template-driven configuration (Jinja2). Strength: the rendered OUTPUT is always internally consistent given its inputs (variables, a base template), easy to review as plain text before deployment. Weakness: rendering a correct file and DEPLOYING it to 500 servers reliably are two SEPARATE steps; Jinja2 itself has no concept of "did server 347 actually receive this file," that enforcement has to be built separately (a copy/scp step, a configuration-management tool used JUST for file distribution), and if THAT separate step is not itself idempotent and reliably re-run, drift reappears through exactly the gap templating alone does not close.

Declarative configuration management (Ansible). Strength: a template (or copy, lineinfile) task is BOTH the rendering step AND the enforcement mechanism in one, Ansible connects to each server, checks current state, and corrects it if it differs, in one idempotent operation; running the SAME playbook repeatedly is safe and self-correcting by design. Weakness: real learning curve and tooling overhead relative to "just render some files," and Ansible's own execution model (a sequence of tasks run in order) is not as purely declarative as, say, Terraform's plan/apply model, though its individual modules ARE idempotent.

For preventing configuration drift across 500 servers specifically. The template-only approach's gap (rendering correctness without deployment-enforcement) becomes MORE consequential at 500-server scale, not less: a manual or semi-automated push process that works reliably for 10 servers has much more room to silently fail, partially apply, or simply be forgotten for SOME subset of 500; Ansible's built-in, idempotent, per-host enforcement closes exactly this gap at scale, which is why it is the stronger recommendation here specifically, not merely a stylistic preference.

Worked example

A concrete migration plan from ad hoc scripts to Ansible, with testing, gradual rollout, and rollback options:

1. Testing, on a small isolated set first. Convert the highest-drift-prone ad hoc scripts (identified via the same incident/change-history approach used for any similar migration) into Ansible playbooks; test against a small (5 to 10 host) isolated cohort, confirming the playbook is genuinely idempotent (running it twice produces no unexpected second-run changes, an explicit, checked property, not assumed).
2. Gradual rollout, risk-ordered cohorts. Expand in risk-ordered waves (lowest-risk hosts first), with an AUDIT-ONLY period (ansible-playbook --check --diff) before enforcement on each new cohort, the same discover-then-audit-then-enforce sequence any staged rollout follows.
3. Rollback options. Per-cohort: if a specific cohort's playbook run causes problems, that cohort reverts to its PREVIOUS (ad hoc script or manual) process temporarily while the playbook is fixed, without affecting already-successfully-migrated cohorts; playbook-level: since Ansible playbooks are themselves version-controlled, a bad playbook CHANGE is a git revert away from the last known-good playbook version, re-run against the affected hosts to correct them back.
4. Full rollout. Once all cohorts are on Ansible with a clean bake period, the ad hoc scripts are formally retired (not just unused, actually removed from any remaining cron/manual-run path), closing the door on the old, drift-prone process being accidentally reintroduced.

Trade-offs and pitfalls

• Common mistake: treating "we render our configs with Jinja2 now" as equivalent to "we've solved configuration drift," when templating alone says nothing about deployment enforcement. A team that adopts templating without ALSO building (or adopting) a reliable, idempotent deployment/enforcement mechanism has solved only HALF the problem, the rendered file being correct, while leaving the other half, the file actually being where it needs to be on every server, exactly where it was before.
• Ansible's idempotency guarantee is at the MODULE level, and a playbook that shells out via a raw command/shell task defeats it for that specific task, the same local-exec-equivalent gap any declarative tool has when it shells out to imperative code; migrating TO Ansible does not automatically mean every task in the resulting playbook is actually idempotent, that discipline still needs to be applied task by task.
• A migration plan that converts scripts to playbooks all at once, rather than testing idempotency explicitly on a small cohort first, risks discovering a non-idempotent task at 500-server scale instead of at 5-to-10-server scale, the same reason any pilot is chosen for genuine representativeness rather than convenience; the small-scale idempotency check in step 1 is not optional caution, it is what makes the subsequent staged rollout trustworthy.
• Retiring the ad hoc scripts (the final step) is easy to skip once the migration feels "done," and skipping it leaves a live, drift-reintroducing path available, an unenforced old path tends to get used again under pressure; formally removing access to the old scripts, not merely stopping their routine use, is what actually closes the loop.