Direct answer
Design a 60-minute error-handling interview exercise around a small, realistic scenario (not a puzzle) that has genuine ambiguity requiring judgment calls, with a rubric that scores correctness, defensive design, testability, and error-message clarity SEPARATELY, so a candidate can score well on some dimensions and poorly on others rather than a single pass/fail gut call.
Structured elaboration
- The prompt: a realistic, bounded task (e.g., 'implement a function that processes a batch of user-submitted records, handling malformed records gracefully') with deliberately underspecified edge cases, since how a candidate HANDLES ambiguity (asking clarifying questions, stating assumptions explicitly) is itself a signal worth observing, not noise to eliminate from the prompt.
- Constraints: a realistic time-box (45 minutes coding, 15 minutes discussion) and explicit permission to use documentation/search, since defensive-coding judgment (not memorized API trivia) is the actual skill being assessed.
- Expected deliverables: working code (even if incomplete, given the time-box) plus a brief verbal explanation of the design decisions made, specifically the error-handling choices and why.
- Scoring rubric, four separate dimensions: (1) correctness: does the happy path work; (2) defensive design: are malformed inputs handled explicitly and sensibly (not just 'does it crash'); (3) testability: is the code structured so the error paths COULD be unit tested (dependency injection, small functions), even if time didn't allow writing every test; (4) clarity of error messages: are errors actionable and specific, or generic.
- Grading borderline solutions: a candidate who writes clean, correct happy-path code but handles only ONE of several plausible malformed-input cases, and can ARTICULATE the others when asked in discussion, should score differently than one who neither handles nor recognizes the other cases exist; the VERBAL discussion component is what distinguishes 'ran out of time but understands the gaps' from 'doesn't see the gaps at all', which is why the discussion time is deliberately built into the 60 minutes, not treated as optional overflow.
Worked example
Prompt: 'Write a function that parses a batch of JSON-encoded user records and returns the valid ones, given the batch may contain some malformed entries.' A strong candidate handles the obvious malformed-JSON case, explicitly discusses (even if not fully implementing) the partial-batch-isolation question (does one bad record fail the whole batch or just itself), and proposes a specific error-reporting shape for the malformed entries; a weaker candidate writes code that crashes on the first malformed record with no discussion of the trade-off, scoring low on defensive design and error-message clarity even if their happy-path code is clean.
Trade-offs and pitfalls
A rubric with only a single overall score collapses genuinely independent signals (a candidate can be excellent at defensive design but rusty on a specific syntax detail, which a single holistic score would unfairly conflate); scoring the four dimensions separately, then discussing them together as a panel, produces a much more calibrated, defensible hiring decision than a single gut-feel number.