Code Quality, Error Handling, and Defensive Programming Questions

Direct answer

A health check that returns success just because the process is running, without checking anything it actually depends on, lies by reporting healthy while the service can't do its job, for example because its database connection pool is exhausted. Liveness checks should answer "should this process be restarted," and readiness checks should answer "can this instance serve traffic right now," and conflating the two causes an orchestrator to make the wrong decision.

Structured elaboration

• Liveness check: answers "should this be restarted?" It should catch deadlocks or unrecoverable internal state, but should not check external dependencies, because a database outage isn't fixed by restarting the app, and a liveness check that fails on a database outage causes the orchestrator to restart every instance simultaneously, making the outage worse.
• Readiness check: answers "should traffic go to this instance right now?" This one should check dependencies, database reachable, cache reachable, disk space available, because an instance that can't reach its database shouldn't receive traffic even though it doesn't need to be restarted.
• The lying health check anti-pattern: an endpoint that just returns success with no logic reports healthy even when every downstream dependency is down, the single most common health-check bug in production systems.
• Defensive design: a readiness check needs its own short timeout and shouldn't itself become a source of load, since orchestrators poll it frequently.

Worked example

A service's health endpoint returns success as long as the HTTP server thread is alive, with no check of its database connection. The database goes down; the app is still "alive" and keeps receiving traffic, returning errors to every real request, while the health check keeps reporting green the entire time, hiding the outage from the orchestrator.

Trade-offs and pitfalls

Making a liveness check too strict, checking dependencies, causes cascading restarts during a dependency outage, turning a partial outage into a full one. Making a readiness check too shallow, like the example above, means the orchestrator keeps sending traffic to instances that can't serve it, arguably worse than doing nothing.

What the interviewer probes next

Whether the candidate distinguishes liveness from readiness at all, since conflating them is extremely common and causes real incidents.

Direct answer

Treat it as a tooling and process gap, not a reviewer-attention gap: add automated secret scanning at commit time and in CI so a credential-shaped string never merges, and add a logging-layer guard that redacts known secret field names before anything is written out, regardless of what a developer typed. Rotate the exposed credential immediately once found, because a log line, once written, has to be treated as compromised even after the code is fixed.

Structured elaboration

• Prevention layer 1, secret scanning: tools like gitleaks or truffleHog run in pre-commit and CI, catching a credential-shaped string before it's even committed, earlier than code review would.
• Prevention layer 2, structured logging with a deny-list: a logging wrapper that inspects known sensitive field names (password, apiKey, token, authorization) and masks them, so even a developer who forgets is still protected by the framework.
• Prevention layer 3, log-destination scanning: tools on the aggregated log store itself that alert if a secret-shaped pattern appears in ingested logs, catching what slipped past the first two layers.
• Response when it happens anyway: rotate the credential immediately; a log line lives in multiple systems (aggregator, backups, a SIEM), so deleting the log line doesn't undo the exposure.

Worked example

A developer adds a debug line that logs "calling payment API with key: " plus the raw key. A pre-commit gitleaks hook configured with a pattern for the vendor's key format flags the commit before it's ever pushed, well before it would reach a human reviewer or production.

Trade-offs and pitfalls

Deny-list redaction by field name misses a secret logged under an unexpected key or embedded in free text, so it's a safety net, not a guarantee, which is why scanning before merge matters more than redaction at log time. Overly aggressive secret-pattern matching produces false positives that train developers to bypass the scanner, so the patterns need tuning to real credential formats.

What the interviewer probes next

Whether the candidate's answer is "we'd catch it in review," which is exactly what already failed here, versus a systemic, automated, multi-layer answer.

Direct answer

Fail-fast stops the operation immediately and surfaces the error the moment something is wrong, trading availability for correctness and a clear signal; graceful degradation keeps the system partially functional by falling back to reduced capability, trading some correctness or completeness for continued availability. The right choice depends on whether a wrong or incomplete answer is worse than no answer at all for this specific system.

Structured elaboration

Fail-fast: when correctness matters more than availability. An internal data pipeline computing financial reconciliation numbers should fail fast and loudly the moment its inputs look wrong, because a wrong number that looks plausible and gets used in a report is a much worse outcome than the pipeline simply not running today. Fail-fast systems are also easier to operate: a hard failure with a clear error is diagnosable immediately, whereas a system that silently degrades can mask a real problem for a long time before anyone notices the quality of its output has quietly dropped.

Graceful degradation: when partial availability beats a hard stop. A customer-facing product page that depends on a recommendation service should degrade to a generic, non-personalized set of recommendations if that service is slow or down, rather than showing the customer an error page, because a slightly-worse-but-functional page is a much better outcome for both the customer and the business than a hard failure on a page that otherwise works fine.

Instrumentation differs by approach. A fail-fast system needs strong alerting on the failure itself, since the failure IS the signal: an error rate spike, a specific exception type, or a circuit breaker opening. A gracefully-degrading system needs the opposite kind of visibility: a metric or log line specifically for "we are currently in degraded mode", because the degraded path, by design, does not look like a failure to a simple error-rate dashboard, and a team that isn't specifically tracking degraded-mode usage can be running in a permanently degraded state for months without noticing.

Communicating degraded functionality. For a user-facing system, this usually means a visible but non-alarming UI signal ("Showing popular items while personalized recommendations are unavailable") rather than silence, since silent degradation erodes trust once a user notices the quality difference without being told why. For a downstream service-to-service dependency, this means an explicit field or header in the response indicating degraded mode, so the calling service can make its own informed choice about whether to also degrade or to fail.

Worked example

An internal pipeline vs. a downstream API dependency, side by side: the internal pipeline computing quarterly revenue numbers for a financial report should fail fast and halt if a required upstream table is empty or a row count sanity check fails, alerting the on-call data engineer immediately, because publishing a subtly wrong number in a financial report is far worse than the report being late. The customer-facing recommendation widget on the same company's storefront, dependent on a separate ML service, should instead catch a timeout from that service and immediately serve a cached "most popular this week" list, log a degraded_mode=true metric tagged with the reason, and continue serving the page, because an incomplete page for one widget is a minor UX cost, not a correctness failure the business needs to halt over.

Trade-offs and pitfalls

The most common mistake is applying the wrong default to a whole system uniformly: treating every dependency as fail-fast produces a fragile product where one non-critical service outage takes down an entire page, while treating every dependency as gracefully-degradable risks quietly serving wrong financial or safety-relevant data with no alert ever firing. The decision should be made dependency by dependency, based on whether being wrong is worse than being unavailable for that specific piece of functionality, not applied as a single system-wide policy.

Direct answer
Decide what error detail reaches a client by defaulting to the minimum that's actually actionable for THAT client (a correlation id and a stable error code, always; a human-readable message only if it's genuinely safe and useful; never a stack trace or internal identifiers), keeping the full detail in internal logs correlated by the same id.

Structured elaboration

• Safe client-facing format: {error_code, message, correlation_id} at minimum; the message should describe what went wrong from the CLIENT's perspective ("the email field is required") never from the server's internal perspective ("NullPointerException in UserValidator.java line 42").
• Correlation ids without leaking internals: a correlation id is safe to expose (it's an opaque token, not information about your system) and is exactly what lets support/engineering find the FULL internal detail later, without the client ever seeing that detail directly.
• Localized user messages: keep the machine-readable error_code stable and English-invariant; localize the human-readable message separately based on the client's locale, so client code branching on error_code never breaks when message wording/translation changes.
• Automated tests for no leakage: a test suite that deliberately triggers every known internal exception type and asserts the CLIENT-FACING response contains none of a blocklist of sensitive patterns (stack trace markers, internal hostnames, SQL fragments, raw exception class names for internal errors) catches this class of leak before it ships, since manual review alone reliably misses it under time pressure.

Worked example
An internal psycopg2.OperationalError: could not connect to server: Connection refused... host "10.2.4.19" must never reach a client; the sanitized response is {"error_code": "internal_error", "message": "Something went wrong on our end. Please try again.", "correlation_id": "7f3e-9c"}, while the full raw exception (including the internal hostname) is logged server-side, findable by an engineer searching for correlation_id: 7f3e-9c.

Trade-offs and pitfalls
The hardest cases are 5xx errors that ARE genuinely useful for the client to know more about (a specific downstream service being down, which the client's own retry logic might want to know about specifically); resist the urge to pass through the raw exception message even here, and instead define a small, deliberate set of STRUCTURED, safe detail fields ({"error_code": "dependency_unavailable", "dependency": "payment_gateway"}) rather than either a blanket generic message or a raw leak.

Direct answer
Under active error-budget exhaustion from cascading 5xx errors, prioritize the mitigation that stops the cascade fastest with the least new risk: circuit breakers and request timeouts first (they cut the feedback loop immediately and are usually already-tested code paths), then backpressure/bulkheads to protect what's left, with retries-with-backoff and graceful degradation as the follow-up once the bleeding has stopped, not the first move.

Structured elaboration

• Circuit breakers, first: if a downstream change is causing cascading failures, the fastest way to stop the cascade is to stop CALLING the failing dependency; a circuit breaker (or a manual, config-driven kill switch if none exists yet for this path) halts the cascade immediately, faster than any code change can ship.
• Timeouts, immediately after: if calls to the failing dependency are hanging rather than failing fast, tightening the timeout (even a temporary, aggressive config change) frees up resources (threads, connections) being held hostage, which is often what's actually driving the cascade beyond the original failing dependency.
• Bulkheads/isolated thread pools: if the resource exhaustion has already spread to starve unrelated requests (see the bulkhead survivor), isolating pools limits further blast radius, though retrofitting a bulkhead mid-incident is a bigger, riskier change than flipping an existing breaker or timeout config.
• Retries with backoff and jitter: valuable for RECOVERY once the dependency is coming back, but retries added or left ENABLED during the active cascade make it worse, not better, by adding more load onto an already-struggling dependency; this is often the first thing to actively DISABLE, not add, during the incident.
• Backpressure: shedding load at the edge (rejecting a percentage of incoming requests outright, with a clear 503) protects the system's remaining capacity for the traffic it CAN serve, at the direct, visible cost of intentionally failing some requests.
• Graceful degradation: the longer-term fix (serve a fallback/cached response instead of failing) usually requires a code change that can't ship instantly during an active incident, so it's the follow-up hardening work, not the immediate mitigation.
• Measuring effectiveness: track the downstream dependency's own error rate and the error BUDGET burn rate in real time as each mitigation is applied; a mitigation is working if the burn rate visibly slows within minutes, not hours.

Worked example
During the incident: (1) immediately flip the circuit breaker for the failing downstream to force-open (or disable retries against it if no breaker exists) to stop the cascade; (2) tighten the client timeout for that dependency from 30s to 2s to stop threads from being held hostage; (3) if capacity is still degraded, enable load shedding (reject 20% of lowest-priority traffic) to protect the rest; (4) once the dependency confirms recovery, re-enable the breaker and retries gradually, watching the error rate as you do, rather than flipping everything back on at once.

Trade-offs and pitfalls
The instinctive first move during many outages is to add MORE retries ('the requests are failing, let's retry them harder'), which is exactly backwards during a cascading-failure incident: more retries onto an already-overloaded dependency deepens the cascade. The discipline that prevents this: stop calling the failing thing first, THEN worry about graceful recovery.