Test Strategy, Planning, and Risk-Based Prioritization Questions

Direct answer

Two short interview tasks that genuinely probe automation-versus-manual judgment: a coding task where the candidate implements a scoring function given ambiguous, realistic constraints, and a design/theory task where the candidate is presented with a specific, somewhat ambiguous scenario and asked to decide and justify whether and how to automate it, with strong answers distinguished by whether the candidate reasons explicitly about trade-offs or simply defaults to "automate everything" or "automate nothing."

Structured elaboration

Coding task: give the candidate a small, realistic function to implement, similar in spirit to scoring a set of tests for automation-worthiness given metadata like frequency, cost, and stability, deliberately leaving some ambiguity in the requirements (for example, not fully specifying every threshold) to see whether the candidate asks clarifying questions or makes and states a reasonable assumption, rather than silently guessing. Scoring criteria: correctness of the implementation against the stated requirements, code clarity and testability of the solution itself, and critically, how the candidate handles the ambiguity, do they ask, state an assumption, or just guess silently.

Design/theory task: present a realistic, moderately ambiguous scenario (for example, "a test runs weekly, takes an engineer 45 minutes, and the underlying feature was redesigned two months ago and might be redesigned again soon, should you automate it") and ask the candidate to decide and justify their recommendation. Scoring criteria: does the candidate identify and weigh the actual relevant trade-offs (frequency versus cost versus stability) rather than reciting a generic definition of automation benefits; do they reach a clear, defensible recommendation rather than an unresolved "it depends" with no actual conclusion; and do they show awareness of what information they do not yet have and would want to gather before fully committing.

What strong answers look like: for the coding task, a strong candidate implements a working, reasonably clean solution, explicitly states the assumption they made about the unclear threshold rather than silently picking one, and can explain why they chose that assumption if asked. For the design task, a strong candidate walks through the specific trade-offs present IN THIS SCENARIO (the recent redesign specifically argues for caution despite the meaningful weekly time cost), reaches a clear recommendation ("defer automation for now given the redesign risk, revisit once the feature stabilizes"), and names what would change their answer (if the redesign is confirmed complete and no further changes are planned).

What weak answers look like: for the coding task, a candidate who silently guesses at the ambiguous requirement without flagging it, or produces a working but poorly-structured solution that would be hard for someone else to adjust later. For the design task, a candidate who reaches for a generic, scenario-independent answer ("automation is always good practice, so yes") without engaging with the specific tension the scenario was designed to surface, or who lands on an unresolved "it depends" without actually committing to a reasoned recommendation.

Worked example

A concrete design-task scenario: "A test covering a checkout confirmation email runs about 15 times a week, takes 10 minutes manually each time, and the underlying email template has not changed in six months. Should you automate it, and if so, what would you automate first?" A strong candidate reasons: frequency is moderate-high (15x/week), manual cost per run is low individually but adds up (roughly 2.5 hours/week), and stability is high (unchanged for six months), landing on "yes, automate, and given the low per-run cost, prioritize confirming the email is triggered and contains correct key fields (order number, total) over pixel-perfect visual verification, since that gives most of the value cheaply." A weak candidate might jump straight to "yes automate everything about this test" without distinguishing what specifically to automate first, or "10 minutes isn't that long, don't bother" without actually computing or considering the weekly cumulative cost.

Trade-offs and pitfalls

The main risk in designing tasks like these is making the scenario TOO obviously one-sided (a test that is clearly a slam-dunk automate or clearly not worth it), which fails to actually differentiate candidates since almost anyone reaches the same answer; the value is specifically in a scenario with a genuine, reasoned tension (like the recent-redesign example), where the reasoning process, not just the final answer, is what reveals judgment quality. The second risk is over-weighting the coding task's raw correctness and under-weighting how the candidate handled ambiguity, which misses that in real automation work, judgment about ambiguous, incompletely-specified situations is often the more important, harder-to-teach skill.

Direct answer

Automated acceptance tests verify that a feature meets its defined functional requirements in a repeatable, scripted way; manual User Acceptance Testing (UAT) is a human, often business-stakeholder-driven review that confirms the feature actually solves the intended real-world problem and feels right to use, which is inherently harder to fully script.

Structured elaboration

The two serve different purposes and should not be treated as interchangeable:

• Automated acceptance tests: owned by engineering or QA automation, they check specific, defined acceptance criteria (given this input, the system produces this output) and run continuously, typically as part of CI, well before a human ever looks at the feature. Success looks like: every defined acceptance criterion passes consistently, and the suite catches a regression the moment it is introduced.
• Manual UAT: owned by business stakeholders or a designated UAT team who represent the actual end users, run close to release (once the feature is functionally stable), and evaluate things that are hard to fully specify as pass/fail assertions: does this workflow match how the business actually operates, is anything confusing or subtly wrong in a way automated checks would not catch. Success looks like: stakeholders sign off that the feature is fit for their actual use, not just that it matches a written spec.

Timing matters: automated acceptance tests should run early and often, ideally on every change, since they are cheap to re-run. UAT should happen once the feature is functionally stable and close to release, since it is expensive (it consumes real stakeholders' time) and repeating it on every minor change would not scale.

Worked example

For a B2B admin dashboard feature that lets a customer's admin manage user permissions: automated acceptance tests would verify things like "granting the editor role to a user allows them to edit records but not delete them," "removing a user's access immediately revokes their active session," and "the audit log records every permission change," run on every commit. These are engineering-owned, deterministic, and fast.

Manual UAT for the same feature would involve an actual customer-facing stakeholder (or an internal proxy for one) walking through the real workflow they would use: onboarding a new team member, adjusting permissions after a role change, auditing who has access before a compliance review, and judging whether the flow makes sense, whether terminology matches how their organization actually thinks about roles, and whether anything feels like it would generate support tickets. This happens once, close to release, after the automated suite is already green.

For long-running or judgment-heavy UAT scenarios, such as an acceptance test that requires subtle human judgement of layout or wording quality, a workable middle ground is combining automated checks with human review: let automation confirm the mechanical parts (the right data appears, the right permissions apply) and reserve a shorter, targeted human review pass for the parts that genuinely require judgment, rather than trying to force the whole scenario into either fully automated or fully manual.

Trade-offs and pitfalls

The common mistake is treating automated acceptance tests as a substitute for UAT, assuming that if the scripted criteria pass, the feature must be acceptable to users; automated tests can only check what someone thought to specify in advance, and UAT exists precisely to catch what nobody specified. The opposite mistake is skipping automated acceptance tests and relying entirely on UAT for functional correctness, which is slow, expensive, and does not scale to catching regressions on every change.

Direct answer

When asked to cut QA time to speed up a release, the right response is not to test less everywhere equally, but to prioritize edge cases by a repeatable likelihood-times-impact method and present the resulting cuts explicitly to stakeholders as a deliberate trade-off, so the decision to skip certain tests is informed and documented rather than an unstated risk nobody agreed to.

Structured elaboration

A repeatable method: score each candidate edge case on likelihood (1-5: how probable is this scenario in real usage or in this specific change) and business impact (1-5: what happens if it goes wrong), multiply for a combined score on a 1-25 scale, and rank descending. This produces a simple risk matrix with an explicit gating rule rather than a case-by-case judgment call: a combined score of 9 or above is test now, 4 through 8 is defer to a fast-follow automation effort, and below 4 is explicitly accept as untested for this release, with the bands set low enough that one high-impact factor (for example, likelihood 1 times impact 4 equals 4) can still pull a case out of the accept tier on its own.

Gating criteria to present to stakeholders: rather than a vague "we're cutting some testing," present the specific tiers and what falls into each, the raw metrics behind the scoring (why a given edge case landed where it did, not just the final tier label), and what would change the decision (if usage data later shows a "low likelihood" case happening more than expected, it gets reprioritized). This turns the cut from an unexplained risk into an explicit, defensible decision stakeholders can weigh in on and revisit.

Worked example

For a release under time pressure, five candidate edge cases might score:

Presented to stakeholders: "we are testing the two highest-scored cases now given their combined likelihood and impact; the two mid-scored cases move to an automated regression test scheduled for next sprint rather than manual testing this week; the lowest-scored case is explicitly accepted as untested for this release, and we will revisit if it turns out to matter more than expected." This gives the product manager a specific, reasoned trade-off to approve rather than an unstated gap in coverage.

Trade-offs and pitfalls

The main risk in this kind of negotiation is caving to time pressure and cutting testing without a repeatable method behind it, which produces an ad hoc, hard-to-defend set of gaps that erode trust the first time one of them causes a production issue. The scoring method and the explicit stakeholder presentation are what convert "we tested less" into "we made a specific, informed trade-off," which is a meaningfully different and more defensible position when something does eventually go wrong in a deferred area.

Direct answer

For a freshly released product with limited engineering capacity, automate unit and integration tests on the account and payment logic first, since those are the highest-risk, most stable, and cheapest-to-automate layers; leave end-to-end and exploratory testing manual until the product's user flows stabilize and capacity grows.

Structured elaboration

Apply three criteria to decide what to automate first:

• Risk: what would hurt the business most if it broke silently. For a product handling accounts and payments, authentication and payment-processing logic carries the highest risk.
• ROI: unit and integration tests are cheap to write and run in seconds, so even modest test volume pays back quickly; end-to-end tests are expensive to build and maintain, and for a freshly-released product whose UI and flows are still evolving, that cost is even higher.
• Stability: automate what is least likely to be rewritten soon. Core business logic (does this password hash correctly, does this payment amount calculate correctly) tends to stabilize faster than UI and UX, which typically keeps changing based on early user feedback.

Exploratory testing stays manual by design regardless of capacity, since its entire value is a human noticing something unexpected; it does not compete for the same automation budget as the other test types.

Worked example

For a web application handling user accounts and payments, good early automation candidates:

• Unit tests: password hashing and validation logic, payment amount and tax calculation, input validation on account creation fields. Fast, deterministic, cheap to write.
• Integration tests: the account-creation flow hitting a real (sandboxed) database, the payment-authorization call against a payment gateway sandbox, session-token generation and validation against the auth service.

Deferred for now: full end-to-end automation of the sign-up-to-first-purchase user journey (expensive to build and the UI is still evolving post-launch), and any UI-level regression suite (high maintenance cost against a product still finding its shape). Exploratory testing of the new-user onboarding experience continues manually throughout, since that is exactly the phase where unexpected user behavior is most valuable to catch by human observation.

Trade-offs and pitfalls

The common mistake for a freshly-launched product is jumping straight to end-to-end UI automation because it "tests like a real user would," without accounting for how expensive it is to maintain against a UI that is still actively changing post-launch. The safer sequencing is bottom-up: unit and integration first, since they are cheap and change less, then end-to-end once the product's core flows have stabilized enough that the automation investment will not be rewritten every sprint.

Direct answer

Even a feature that seldom fails can have a strong automation business case if the manual execution cost is high enough: at 8 hours and two engineers per manual run (16 engineer-hours), a one-time automation investment of around 80 engineer-hours breaks even in roughly five to six nightly runs, meaning under two weeks, well before "seldom fails" becomes a reason to skip automating it.

Structured elaboration

The business case needs three explicit numbers and a validation plan for each:

1. Manual cost per run: hours per run times number of people involved. Validate by actually timing a few real manual runs rather than trusting an estimate, since manual processes are frequently slower in practice than remembered.
2. Automation build cost: a one-time estimate in engineer-hours. Validate by spiking a small proof-of-concept for the hardest part of the automation (often environment setup or a flaky dependency) before committing to a full estimate, since build-cost estimates are the most commonly wrong number in these business cases.
3. Ongoing maintenance cost per run: automated tests are not free after they are built; assume a small per-run overhead for monitoring and occasional fixes. Validate this after the first month of real operation rather than assuming it stays at the initial estimate.

Break-even is the point where cumulative manual cost, extrapolated forward, exceeds cumulative automation cost (build plus ongoing maintenance).

Worked example

Assumptions: manual run costs 8 hours times 2 engineers = 16 engineer-hours per run. Automation build cost estimated at 80 engineer-hours (a moderate-complexity nightly job). Ongoing maintenance estimated at 0.25 engineer-hours per run (roughly one hour every four runs, for monitoring and occasional fixes).

$\text{break-even runs} = \frac{\text{build cost}}{\text{manual cost per run} - \text{maintenance cost per run}} = \frac{80}{16 - 0.25} \approx 5.1 \text{ runs}$

At 5 runs: manual cumulative cost is 5 x 16 = 80 hours; automation cumulative cost is 80 + 5 x 0.25 = 81.25 hours, automation is essentially at parity. At 6 runs: manual cumulative cost is 96 hours versus automation's 81.5 hours, automation is now clearly ahead. Since the job runs nightly, break-even arrives in under a week of operation. Sensitivity check: even if the build estimate were significantly more pessimistic, at 200 engineer-hours instead of 80, break-even still arrives at roughly 13 runs (under two weeks), because the manual cost per run (16 hours) is high enough that the payback is robust to a meaningfully wrong build-cost estimate.

Trade-offs and pitfalls

The business case can mislead if the manual-cost estimate is based on how long the process is SUPPOSED to take rather than how long it actually takes in practice; measuring a few real runs before finalizing the case avoids building a business case on an optimistic number. The other pitfall is ignoring maintenance cost entirely and comparing only build cost against manual cost, which overstates how fast the payback arrives and sets an unrealistic expectation for when the investment pays for itself.