Code Quality, Error Handling, and Defensive Programming Questions
Writing robust, high-quality code that fails safely. Covers defensive programming, input validation, error handling and fault tolerance, logging for diagnosability, and general engineering-quality standards. Includes anticipating failure modes and making code resilient to bad inputs and unexpected states.
Describe the secure and defensive coding practices a data scientist must follow when shipping model code and data pipelines to production. Provide an example of how you would manage secrets in a CI/CD pipeline as part of your answer.
Sample Answer
Direct answer
A data scientist shipping model code and pipelines into production carries the same secure- and defensive-coding obligations as any other engineer touching production systems, specifically around secret management, input validation, least-privilege access, dependency scanning, and avoiding sensitive data in logs, even though these concerns are less often emphasized in a typical data-science education than modeling technique itself.
Structured elaboration
Secret management. Credentials for a data warehouse, an object store, or a third-party API must never be hardcoded in a notebook or a script, and never committed to version control, including in a notebook's saved output cells, which is a common and easy-to-miss leak specific to notebook-based workflows; instead, secrets are injected at runtime from a secrets manager or environment variables provided by the CI/CD or orchestration platform.
Input validation. Data read from any external or upstream source (a raw CSV drop, an API response, a message queue) should be validated for expected shape, types, and ranges before being fed into a model or a transformation step, the same discipline as validating an API request in a traditional backend service, since a silently malformed input to a training or inference pipeline can produce a subtly wrong model or prediction rather than a visible crash.
Least privilege and role-based access. A pipeline's service account or credential should have only the specific read/write permissions its actual job requires (read access to the specific input tables, write access to the specific output location), not a broad, convenient "can access everything in the data warehouse" credential reused across many pipelines, since a bug or a compromised credential in one pipeline should not be able to affect unrelated data it never needed to touch.
Dependency vulnerability scanning. The Python (or other language) package ecosystem commonly used in data science pulls in a large, often under-audited dependency tree; running an automated vulnerability scanner against the pipeline's dependencies as part of CI, and having a defined process for updating a flagged dependency, closes a gap that is frequently overlooked specifically in data-science-originated code compared to more traditionally security-reviewed backend services.
Avoiding sensitive information in logs. Logging the full contents of a row for debugging (a common, convenient habit during model development) risks logging PII or other sensitive fields into a log aggregation system that may have much broader read access than the original data source itself; log identifiers, counts, and derived statistics instead of raw sensitive field values, and specifically review any debug-level logging left over from development before it ships to production.
Worked example
A secrets-in-CI/CD example: a nightly training pipeline needs a database credential to read training data and an object-store credential to write the resulting model artifact. Rather than embedding these in the pipeline's own configuration file (which would be committed to the same repository as the pipeline code, or worse, saved into a notebook's own output), the CI/CD platform's secrets manager injects them as environment variables at pipeline-execution time, scoped specifically to that pipeline's own service account, which has read-only access to exactly the training-data tables it needs and write-only access to exactly the model-artifact bucket, nothing broader. The pipeline code reads os.environ["DB_CREDENTIAL"] at runtime and never persists it to disk or logs it, even at debug log level, and a separate, automated scan of the pipeline's requirements.txt runs on every CI build, flagging any dependency with a known CVE before the pipeline is allowed to deploy.
Trade-offs and pitfalls
The most common and specifically notebook-native leak is a credential or a sensitive data sample accidentally saved into a notebook's own OUTPUT cells (from a print() or a displayed DataFrame during exploratory work) and then committed to version control along with the notebook file itself, since the credential or data isn't visible in the input cells' code at all, only in the saved output, which many code reviewers don't think to check as carefully as the code itself. A second common mistake specific to data science workflows is treating dependency and secret hygiene as "someone else's job" (the platform or security team's), rather than as a baseline expectation for any code being shipped to production, regardless of whether the author's primary background is modeling or traditional software engineering.
Describe the role of assertions and invariants in maintaining code correctness. When should assertions be used versus throwing exceptions? Provide an example where an assertion detects a developer error early and avoids a costly runtime check in production, and describe how this maps to design-by-contract thinking (preconditions, postconditions, invariants).
Sample Answer
Direct answer
An assertion checks a condition that your own code's logic guarantees should always be true if nothing upstream has a bug; an exception handles a condition that the outside world (a user, a file system, a network) can legitimately produce regardless of whether your code is correct. Assertions catch developer errors early and cheaply; exceptions handle the world being unpredictable.
Structured elaboration
What an assertion is for. An assertion encodes an invariant: a statement that, given correct code, must hold at this point in the program no matter what valid input arrives. If it fails, the bug is in the code that led to this point, not in the input or the environment. Because of this, assertions are cheap to reason about (you never need a recovery path for them, the correct response to a failed assertion is to fix the bug) and, in several languages, can be compiled out entirely in optimized production builds, which is precisely why they must never be relied on for something the outside world can trigger.
What an exception is for. An exception handles a condition that is a normal, expected possibility given a correct program: a file that does not exist, a network call that times out, a user who submits invalid input. These require an actual recovery path (retry, a default, informing the user) because they will happen in production no matter how correct the code is.
Preconditions, postconditions, and invariants (design by contract). A precondition is what a function requires to be true of its inputs to behave correctly; a postcondition is what it guarantees to be true of its output if the precondition held; an invariant is a condition that must hold at every observable point in an object's lifetime. Assertions are the natural implementation mechanism for all three inside a single codebase's own internal logic: asserting a precondition at function entry catches a caller who violated the contract due to a bug in their own code, which is different from validating an input that arrived from outside the trust boundary and might be malformed for entirely legitimate reasons.
The line between the two. The test is not "is this input bad", it's "could this input be bad even if every line of my own code is correct". A negative array length passed internally between two functions you wrote and control, where nothing external can produce that value if your code is right, is an assertion case. A negative quantity field parsed from a JSON request body is an exception (or validation-error) case, because a malicious or buggy client can produce it no matter how correct your server code is.
Worked example
A function withdraw(account, amount) internal to a ledger system might assert assert account.balance >= 0, "invariant violated: account balance went negative" right after debiting, because if the debit logic is correct, the balance should never go negative; if this assertion fires, there's a bug in the debit logic itself, and the fix is to find that bug, not to add a check that reacts gracefully to a negative balance in production. Contrast this with the same function's very first line, which must instead RAISE an exception (not assert) if the caller passes a negative amount: a negative withdrawal amount is exactly the kind of thing an upstream caller (a request handler parsing user input) can produce, whether or not the ledger's own internal logic has any bugs at all, so it needs a real, always-active check, and the right response is to reject the request, not to crash a debug build's assertion and silently no-op in production.
Trade-offs and pitfalls
The most costly mistake is using an assertion to validate something a real caller can trigger: because assertions can be disabled in optimized builds in several languages (C's NDEBUG, Python's -O flag), a check that only exists as an assertion can silently vanish in production, meaning the exact case it was meant to guard against reaches production code entirely unchecked. The opposite mistake, wrapping every internal invariant in a full exception with a try/catch elsewhere in the codebase, adds real performance and readability cost for something that, if your own code is correct, should genuinely never happen and does not need a recovery path at all, only a way to fail loudly and immediately during development and testing.
Your feature store becomes temporarily unavailable. Describe specific strategies you would use to keep online model predictions available in a degraded but safe form. Explain the trade-offs for accuracy, fairness, and user experience, and how you would test the fallback behavior.
Sample Answer
Direct answer
When the feature store is unavailable, an online model-serving path should never simply fail the request: it should fall back through a small, pre-agreed hierarchy (cached recent features, then a safe default prediction with a lowered confidence flag) so the caller always gets a usable response, while the fact that a fallback was used is logged and monitored so the degradation is visible rather than silent.
Structured elaboration
The fallback hierarchy, in priority order. First choice: serve the most recent successfully-computed features for this entity from a local or short-TTL (time-to-live) cache, since slightly-stale features usually produce a prediction close enough to correct for most use cases. Second choice, if no cached features exist for this entity (a genuinely new user or item): fall back to population-level default feature values (an average or a reasonable prior) and mark the resulting prediction with a lower confidence score, or route it to a simpler, feature-light backup model if one exists. Last resort, if neither is available and the caller cannot tolerate any uncertainty: return a clearly-flagged "prediction unavailable" response rather than a fabricated number, for use cases where acting on a low-confidence guess is worse than not acting at all (for example, a hard compliance decision, as opposed to a UI recommendation widget).
Trade-offs across accuracy, fairness, and user experience. Cached features trade some accuracy for continuity, and this is usually the right trade for continuously-updating profiles like browsing behavior. Population-level defaults introduce a fairness risk: an average feature value systematically favors the "typical" user in the training data and can under-serve users whose real behavior differs from that average, which matters more for some decisions (loan or hiring-adjacent predictions) than others (a media recommendation). The user experience angle: whether to show a degraded result silently or to flag it depends on the stakes; a slightly-stale recommendation can be shown without comment, but a lending or eligibility decision made on default features should be flagged for a human review rather than treated as equivalent to a normal decision.
Testing the fallback path. The fallback logic should be tested the same way the primary path is: with an explicit test that simulates the feature store being unavailable (not just tested manually during an actual outage), asserting that the service still returns a response within its latency budget, that the response is correctly flagged as degraded, and that a monitoring signal fires for the fallback having been used.
Worked example
A fraud-scoring endpoint depends on a feature store that goes down. Design: on a feature-store timeout, the request handler checks a local, short-TTL (60 second) cache for this user's last-known feature vector; if present, it scores with those features and returns a normal-looking response with an internal feature_source: "cached" field for observability, not shown to the end user since a slightly-stale fraud score is not itself a problem. If no cached features exist (a first-time user, for example), the handler falls back to a simpler rules-based check rather than the full ML model, and marks the decision feature_source: "fallback_rules", confidence: "low", which downstream systems can use to route the transaction to manual review rather than an automatic approval, since a fabricated ML confidence score on default features would be actively misleading here.
Trade-offs and pitfalls
The single biggest pitfall is a fallback path that works technically (the request does not error) but is never monitored, so a feature-store outage that lasts three days produces three days of quietly degraded predictions with nobody aware it happened until someone notices a downstream metric drifted. A second pitfall is applying the same fallback strategy regardless of the decision's stakes: silently substituting a default feature vector is a reasonable trade for a content-recommendation widget and a much riskier one for a decision with fairness or compliance implications, where the safer choice may be to flag for manual review rather than to auto-approve on a guess.
Design a Java logging helper that redacts common PII, such as email addresses, Social Security numbers, and credit-card numbers, from log messages before they are written. State your assumptions, show the use of compiled regular-expression patterns, discuss the performance considerations, explain how you would configure the helper to extend the redaction patterns, and describe how you would test and validate it at scale. Also cover what structured fields you would include (for example a correlation ID and a job or request identifier), what you would log at INFO versus DEBUG level, and how the approach differs for a nightly batch scoring job versus a real-time service.
Sample Answer
Direct answer
A PII-safe logging helper redacts common sensitive patterns (emails, Social Security numbers, credit card numbers) from a message before it is ever written to a log, using compiled regular expressions applied consistently across every log call, while also enforcing structured fields (a correlation ID, a job or request identifier), log-level discipline, and an extensibility point for adding new redaction patterns as new sensitive-data types are identified.
Structured elaboration
Redaction via compiled regex patterns. A small set of well-tested regular expressions, compiled once and reused (not recompiled on every log call, which would be wasteful), match common PII shapes: an email address pattern, a Social Security number pattern (\d{3}-\d{2}-\d{4}), and a credit-card-like sequence of 13 to 16 digits (allowing spaces or dashes as separators, since real card numbers are often written with them). Each match is replaced with a fixed [REDACTED] marker before the message is passed to the underlying logging framework.
Performance considerations. Compiling patterns once at startup (not per-call) and running them against every log message adds a small, roughly-constant regex-matching cost per log call; for a very high-throughput logging path this is measurable but usually acceptable, since the alternative (an actual PII leak into a log aggregation system with far broader read access than the original data source) is a materially worse outcome, and the specific cost can be validated by simply measuring log throughput with and without the redaction step in a realistic load test.
Configuration to extend redaction patterns. New PII patterns identified over time (an internal account-number format specific to the business, for instance) should be addable without modifying the core logging class, via a constructor parameter or configuration file accepting additional patterns, so the redaction logic can grow as new sensitive-data types are identified in practice, not require a code change and redeploy of the core logging utility itself for every new pattern.
Testing and validating at scale. Beyond unit tests confirming each pattern redacts correctly and that non-sensitive messages pass through unchanged, validating "at scale" means running the redaction logic against a genuinely large, realistic sample of actual (or realistically-synthetic) log messages and manually or statistically auditing a sample of the OUTPUT for anything that looks like it should have been redacted but wasn't, since regex patterns can have false negatives on real-world data that a small, hand-written unit test suite won't surface (an international phone number format, a differently-formatted SSN with no dashes).
Structured fields, log levels, and INFO versus DEBUG. Beyond redaction, the logging helper should ensure every log entry carries a correlation ID (tying related log lines from the same request or job together) and a job/run identifier where relevant (for a batch context specifically). INFO-level logging should capture what a normal operator needs to see to understand system behavior (a job started, a job completed, a summary count); DEBUG-level logging can include more granular detail useful only during active troubleshooting, but should still be redacted with exactly the same discipline as INFO, since a DEBUG log accidentally left enabled in production is a very common real-world path by which sensitive data actually ends up in logs.
Differences for a batch job versus a real-time service. A nightly batch scoring job's logging strategy centers on a per-run summary (start time, record count processed, error count, completion status) tagged with a job_id and run_id, since a human reviewing a batch job's logs the next morning wants an overview, not necessarily a line per record. A real-time service's logging centers on a per-REQUEST correlation ID and typically much higher log volume, and needs sampling strategies for very high-traffic endpoints (logging a representative fraction of successful requests at INFO, while still logging every failure) to keep log volume and cost manageable without losing visibility into failures specifically.
Worked example
public class PiiSafeLogger {
private static final Pattern EMAIL = Pattern.compile("[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}");
private static final Pattern SSN = Pattern.compile("\\b\\d{3}-\\d{2}-\\d{4}\\b");
private static final Pattern CREDIT_CARD = Pattern.compile("\\b(?:\\d[ -]*?){13,16}\\b");
private final List<Pattern> patterns = new ArrayList<>(List.of(EMAIL, SSN, CREDIT_CARD));
public PiiSafeLogger(List<Pattern> extraPatterns) { patterns.addAll(extraPatterns); }
public String redact(String message) {
if (message == null) return null;
String result = message;
for (Pattern p : patterns) result = p.matcher(result).replaceAll("[REDACTED]");
return result;
}
}
Verification note: a Java Development Kit was not available in this execution sandbox, so the exact class above was not compiled directly; the identical regular-expression patterns and replacement logic were instead executed against an equivalent Python translation (Python's re module uses the same pattern syntax for these specific expressions) and confirmed: an email address embedded in a sentence is fully redacted; a dashed Social Security number is fully redacted; a spaced 16-digit credit-card-like number is fully redacted; a message containing no PII passes through completely unchanged; and a null input returns null rather than throwing. This confirms the REGEX LOGIC is correct; it does not confirm Java-specific behavior (for example, java.util.regex.Pattern's exact semantics versus Python's re), which is a disclosed limitation of this verification, not a claim of full Java execution.
Trade-offs and pitfalls
Regex-based redaction is a strong first line of defense but is not exhaustive: a credit card number with unusual formatting, a non-US identification number format, or a PII field that doesn't match any recognizable pattern at all (a person's name in a free-text field, which has no distinguishing shape a regex can reliably catch) can pass through unredacted; this is exactly why validating "at scale" against a realistic log sample, not just a handful of unit tests, matters, and why some organizations pair regex-based redaction with an explicit ALLOWLIST discipline (only log fields you've deliberately reviewed) for the highest-sensitivity data flows, rather than relying on a denylist-style redaction pattern to catch everything. The most common real-world failure mode is a DEBUG-level log statement, written during development and forgotten, that logs an entire request or record object directly (bypassing the redaction helper entirely, since it wasn't routed through it) rather than going through this centralized logging path, which is why enforcing that ALL logging goes through a single, redacting logger (not calling a raw print or an unwrapped logging framework call directly) is as important as the redaction logic itself.
That is every published Code Quality, Error Handling, and Defensive Programming question for Data Scientist so far. Browse the other topics in this category, or practice this one interactively.