Code Quality, Error Handling, and Defensive Programming Questions

Direct answer

For security-sensitive decisions like authorization, default to fail closed, deny the action when the check can't complete, because the cost of a false negative (briefly blocking a legitimate user) is almost always smaller than the cost of a false positive (an unauthorized action succeeding). The exception is when failing closed itself creates a worse safety or availability problem, which is a deliberate, domain-specific judgment call, not a blanket rule.

Structured elaboration

• Default posture: authorization and permission checks fail closed. If the permission service times out, deny the request with a clear error; don't silently treat "unknown" as "allowed."
• Where fail-open is sometimes deliberate: low-stakes, availability-critical paths where an outage of the check itself would be worse than the risk it guards against, and the exposure window is bounded and monitored.
• Make the decision explicit and documented per check, not an accident of how the code happens to be written; a try/catch that swallows the exception and falls through is an accidental fail-open.
• Instrument it: a fail-closed denial caused by an internal error should be logged and alerted distinctly from a legitimate permission denial, so an outage in the auth path shows up as an incident, not just a spike in 403s.

Worked example

An admin panel calls a permissions service to check if a user can delete a record. The service times out. If the code reads let allowed = true; try { allowed = permissionCheck(); } catch (e) { logger.warn(e); } followed by if (allowed) { allowDelete(); }, the caught exception is logged but never rethrown, so allowed is left at its optimistic default of true and the timeout results in the delete going through anyway, an accidental fail-open hiding inside code that looks defensive because it has a try/catch. The fix flips the default and the exception path: let allowed = false; try { allowed = permissionCheck(); } catch (e) { return deny(); }, so any failure to get a definitive answer denies the action.

Trade-offs and pitfalls

Fail-closed can turn a partial outage of a dependency into a full outage of everything gated behind it, so it needs to be paired with making that dependency itself highly available (caching the last known-good permission, short timeouts, circuit breakers) rather than accepting cascading denial as the cost of security. Fail-open, chosen without discussion, is the classic way an authorization bug ships silently, because everything still "works" in testing.

What the interviewer probes next

Whether the candidate can articulate the asymmetry between a false allow and a false deny, rather than reciting "fail closed is always right," and whether they'd catch an accidental fail-open hiding inside ordinary-looking exception handling.

Direct answer
A boundary check catches a specific class of bug (accessing an index, offset, or value outside the range the code actually handles correctly) at the moment it happens, instead of letting it silently produce wrong output or crash somewhere unrelated later; three concrete examples: array/list indexing, pagination offsets, and numeric limits.

Structured elaboration and worked examples

• Array indexing: the bug is an off-by-one or attacker-controlled index reading past the end of a buffer or list. The defensive check: validate 0 <= index < len(array) before accessing, raising a clear IndexError/custom exception instead of either crashing with a cryptic native error or, in an unsafe language, reading adjacent memory. A unit test: assert_raises(IndexError, get_item, [1,2,3], 5).
• Pagination offsets: the bug is a negative or absurdly large offset/limit from a client, which can either error confusingly deep in a SQL driver or, worse, silently return zero rows and look like 'no data' rather than 'bad request'. The defensive check: clamp or reject offset < 0 and cap limit to a sane maximum (say 1000) before it reaches the query layer. A unit test: assert paginate(items, offset=-5, limit=10) raises ValueError.
• Numeric limits: the bug is an integer overflow or an out-of-domain value (a negative quantity in an order, a percentage over 100) silently producing a nonsensical result instead of an error. The defensive check: validate the value's range explicitly before using it in a calculation. A unit test: assert_raises(ValueError, apply_discount, price=100, percent=150).

Trade-offs and pitfalls
Each of these checks is cheap individually, but the value comes from applying them CONSISTENTLY at every place the boundary is actually crossed (every array access from external input, not just the ones you happen to remember); a single unguarded pagination endpoint added six months later by someone who didn't see this pattern reintroduces the exact bug class. Treat these as patterns to lint for or wrap in a shared utility function, not as one-off checks to remember individually.

Direct answer

For malformed records, validate each record against a schema at ingestion and route failures to a dead-letter store instead of letting them stop the batch or corrupt downstream tables, turning 0.1% bad data into a bounded, monitored side channel. For crash-safe restarts, make the job idempotent by processing in checkpointed, replayable units, so a restart resumes from the last committed checkpoint instead of reprocessing everything or skipping records.

Structured elaboration

• Malformed-record isolation: validate required fields and types before transformation; a record that fails validation is written to a dead-letter table with the original payload and the specific error, and the job continues with the remaining records rather than failing the whole batch on one bad row.
• Alert on the dead-letter rate, not just its presence: 0.1% might be expected background noise or a new bug; alert when the rate deviates from baseline, since a hardcoded "zero bad records" threshold guarantees constant false alarms.
• Crash-safe restart via checkpointing: process in small, idempotent units, for example per-partition or per-batch, and commit a checkpoint only after that unit's output is durably written; on restart, resume from the last committed checkpoint.
• Idempotent writes: make the downstream write itself safe to repeat, using an upsert keyed by a natural id rather than an append, so if a batch is reprocessed after a crash, the result matches processing it once, not duplicated rows.

Worked example

A pipeline processes daily transaction files in batches of 10,000 records, checkpointing after each batch's output is durably upserted to the warehouse. If the job crashes after committing batch 47 but before finishing batch 48, the restart resumes at batch 48, not batch 1. Within batch 48, a record with an invalid date fails schema validation, is written to a dead-letter table with the error "invalid date format," and processing continues with the remaining 9,999 records.

Trade-offs and pitfalls

Checkpointing after every single record adds overhead for little benefit; checkpointing once per whole file means a crash near the end reprocesses almost everything, which is only safe if the writes are truly idempotent. A common pitfall is building dead-letter isolation for malformed records but forgetting to make the checkpoint-and-restart logic idempotent, so bad-data handling looks solved while a crash still silently duplicates good records.

What the interviewer probes next

Whether the candidate connects the two problems, idempotent writes are what make checkpointed restarts actually safe, rather than treating them as two unrelated asks.

Direct answer

The endpoint needs an idempotency key, a client-generated unique id sent with the request, so the server recognizes a retry as "the same request" and returns the original result instead of processing it twice. To test it, simulate the exact failure, the server processes the request but the response is lost, and the client retries with the same key, then assert only one charge exists.

Structured elaboration

• Implementation requirement: the server stores (idempotency key, request hash, result) and on a repeat key returns the stored result instead of re-executing side effects, typically with a TTL and a check that the retried body matches the original.
• Test 1, happy-path retry: send request A with key K, capture the result; resend identical request A with key K; assert the second call returns the same result and no second charge is created.
• Test 2, concurrent retry: fire two requests with the same key K at nearly the same time, simulating a client retrying before the first response returns; assert exactly one charge is created. This tests that the check is atomic, not "check then insert" with a race window.
• Test 3, key reuse with a different payload: send key K with amount 10, then key K with amount 20; assert the defined behavior (reject as a conflict) rather than silently using either amount.
• Test 4, TTL expiry: confirm the deliberately chosen, tested behavior when the same key is reused after the idempotency record has expired.

Worked example

A checkout service processes a 50-unit charge under key "order-4471-attempt-1". The client's connection drops after the charge succeeds but before the response arrives, so it retries with the same key. The test asserts the payment gateway shows exactly one 50-unit charge and the API returns the same transaction id both times, not a new one.

Trade-offs and pitfalls

The most common bug is a "check if key exists, then insert" pattern that isn't atomic; it passes sequential tests but fails under real concurrent retries, which is why Test 2 is the one that actually matters and the one teams skip. Storing idempotency records forever is a data-growth problem, but too short a TTL reopens the double-charge window during exactly the retry storms it exists to prevent.

What the interviewer probes next

Whether the candidate reaches for the concurrency test unprompted, since a sequential-only idempotency suite gives false confidence.

Direct answer

Fail-fast stops the operation immediately and surfaces the error the moment something is wrong, trading availability for correctness and a clear signal; graceful degradation keeps the system partially functional by falling back to reduced capability, trading some correctness or completeness for continued availability. The right choice depends on whether a wrong or incomplete answer is worse than no answer at all for this specific system.

Structured elaboration

Fail-fast: when correctness matters more than availability. An internal data pipeline computing financial reconciliation numbers should fail fast and loudly the moment its inputs look wrong, because a wrong number that looks plausible and gets used in a report is a much worse outcome than the pipeline simply not running today. Fail-fast systems are also easier to operate: a hard failure with a clear error is diagnosable immediately, whereas a system that silently degrades can mask a real problem for a long time before anyone notices the quality of its output has quietly dropped.

Graceful degradation: when partial availability beats a hard stop. A customer-facing product page that depends on a recommendation service should degrade to a generic, non-personalized set of recommendations if that service is slow or down, rather than showing the customer an error page, because a slightly-worse-but-functional page is a much better outcome for both the customer and the business than a hard failure on a page that otherwise works fine.

Instrumentation differs by approach. A fail-fast system needs strong alerting on the failure itself, since the failure IS the signal: an error rate spike, a specific exception type, or a circuit breaker opening. A gracefully-degrading system needs the opposite kind of visibility: a metric or log line specifically for "we are currently in degraded mode", because the degraded path, by design, does not look like a failure to a simple error-rate dashboard, and a team that isn't specifically tracking degraded-mode usage can be running in a permanently degraded state for months without noticing.

Communicating degraded functionality. For a user-facing system, this usually means a visible but non-alarming UI signal ("Showing popular items while personalized recommendations are unavailable") rather than silence, since silent degradation erodes trust once a user notices the quality difference without being told why. For a downstream service-to-service dependency, this means an explicit field or header in the response indicating degraded mode, so the calling service can make its own informed choice about whether to also degrade or to fail.

Worked example

An internal pipeline vs. a downstream API dependency, side by side: the internal pipeline computing quarterly revenue numbers for a financial report should fail fast and halt if a required upstream table is empty or a row count sanity check fails, alerting the on-call data engineer immediately, because publishing a subtly wrong number in a financial report is far worse than the report being late. The customer-facing recommendation widget on the same company's storefront, dependent on a separate ML service, should instead catch a timeout from that service and immediately serve a cached "most popular this week" list, log a degraded_mode=true metric tagged with the reason, and continue serving the page, because an incomplete page for one widget is a minor UX cost, not a correctness failure the business needs to halt over.

Trade-offs and pitfalls

The most common mistake is applying the wrong default to a whole system uniformly: treating every dependency as fail-fast produces a fragile product where one non-critical service outage takes down an entire page, while treating every dependency as gracefully-degradable risks quietly serving wrong financial or safety-relevant data with no alert ever firing. The decision should be made dependency by dependency, based on whether being wrong is worse than being unavailable for that specific piece of functionality, not applied as a single system-wide policy.