Meta QA Engineer (Staff Level) Interview Preparation Guide

Direct answer

Measuring automation ROI across multiple product lines and attributing it causally to automation changes specifically, rather than to other concurrent factors, requires a staggered rollout design across product lines (since a true randomized experiment across whole product lines is rarely practical) combined with a pre-registered set of metrics and an explicit plan for ruling out the most likely confounding explanations.

Structured elaboration

Metrics: defect reduction (production defect rate per release, per product line), release cycle time (time from code-complete to release, per product line), and maintenance cost (engineer-hours spent maintaining the automated suite, per product line), tracked consistently across all product lines both before and after their respective automation investments.

Experiments and analysis plan: since randomly assigning entire product lines to "get automation investment" versus "do not" is usually not something the business will accept (it means deliberately under-investing in some product lines), a staggered rollout design is more practical: automation investment rolls out to different product lines at different, staggered times (driven by real prioritization, not randomization), and each product line's own before/after change is compared, then aggregated across product lines, which gives some of the benefits of replication (multiple independent before/after comparisons) without requiring an ethically or practically difficult randomized design.

Control groups: product lines not yet reached by the rollout at a given point in time serve as a concurrent comparison group, helping distinguish an organization-wide trend (an overall improvement in release practices unrelated to automation) from a change specific to the product lines that received the automation investment.

Attributing causality: look for a consistent pattern across the staggered rollout, if each product line's metrics improve specifically around the time IT received automation investment, rather than all product lines improving simultaneously regardless of their individual rollout timing, that pattern is much stronger evidence of a real causal effect than a single before/after comparison on one product line alone, since it would be a coincidence for an unrelated confounder to happen to align with each product line's own staggered rollout timing.

Worked example

Four product lines receive automation investment at different times: Product Line A in month 1, B in month 4, C in month 7, D in month 10, over a total 15-month observation window. If defect rate drops specifically in the 1-3 months following EACH product line's own rollout, month 1-3 for A, month 4-6 for B, and so on, rather than all four dropping together at some unrelated organization-wide moment (say, all dropping in month 8 regardless of individual rollout timing), that staggered pattern, aligned with each line's own specific rollout timing, provides meaningfully stronger causal evidence than any single line's before/after comparison alone. Release cycle time and maintenance cost are tracked the same way, checking whether their improvement (or, for maintenance cost, initial increase followed by longer-term decrease) also aligns with each line's individual rollout timing.

Trade-offs and pitfalls

The most common mistake is treating a single product line's before/after improvement as strong causal evidence on its own, when a staggered, multi-product-line design with each line's improvement aligning to its own specific rollout timing is a substantially stronger, more defensible standard of evidence. The second mistake is ignoring maintenance cost as a metric and only reporting defect reduction and cycle-time improvement, which can make automation investment look like a pure win while hiding a real, ongoing cost that affects the true net ROI picture.

Plan overview
Design a pipeline that never uses production PII directly: use masking/anonymization for prod-like records, synthesize non-PII datasets for most tests, and seed CI databases deterministically to keep tests repeatable and debuggable.

Anonymization / masking

• Use reversible tokenization only for authorized devops (vaulted keys). Prefer irreversible masking for test environments.
• Techniques: deterministic hashing with salt per-environment for unique-but-non-identifying values; format-preserving masking for phone/SSN to keep schema/validation.
• Trade-offs: masking preserves realism but risks leakage if keys are compromised; irreversible masking reduces risk but can break uniqueness or validation.

Synthetic-data generation

• Create generators (Faker + domain rules) that obey constraints and edge-cases; generate user flows with realistic relationships.
• Trade-offs: safe and flexible, but may miss subtle production distributions causing false positives/negatives.

Seeding CI databases

• Store canonical fixtures as versioned, small JSON/SQL fixtures; load via migrations or test setup hooks.
• Use deterministic RNG seeds and timestamp anchors so generated data is reproducible.
• Example: seed users, accounts, transactions with fixed UUIDs derived from a namespace + index.

Referential integrity

• Generate data via factories that build parent-child relationships in a single transaction; validate FK constraints post-seed.
• Use DB transactions/foreign key checks and checksum comparisons to detect corruption.

Deterministic tests for debugging

• Fix RNG seeds, deterministic timestamps (freeze time), and id generation (stable UUIDs). Log seed and fixture version on test failure.
• Keep small, focused fixtures for unit tests; broader scenario fixtures for integration tests.

Governance & automation

• Enforce CI gate that prohibits tests referencing production exports; scan test data for PII patterns; rotate masking keys; audit logs.
• Trade-offs: stricter controls increase safety but add maintenance overhead.

This plan balances safety, realism, and reproducibility for reliable automated testing without exposing PII.

Direct answer
The situation I'd describe is a mid-sized project where my manager wanted me to present a set of results to a client as more conclusive than the underlying data actually supported, because the client relationship was under strain and a confident-sounding update would help. My personal value was straightforward accuracy in what I present, even when the more cautious version is less comfortable to deliver; my manager's approach prioritized relationship repair over precision in that specific moment. I did not treat it as a fight to win outright; I looked for a version of the update that was honest and still served the relationship.

Structured elaboration

1. Name the actual tension precisely, not just "we disagreed." In this case it was not that my manager wanted me to lie; it was a difference in where to draw the line between appropriately confident communication and overstating certainty, which is a much more common and more defensible kind of workplace values conflict than an outright integrity violation.
2. Raise the concern directly and early, privately, before the moment it would matter (the client meeting), rather than either silently complying or making it a public confrontation. I asked my manager one on one what specifically in the data supported the stronger framing, which turned the conversation from a disagreement about values into a conversation about evidence.
3. Offer an alternative that serves the underlying goal your manager actually cares about. My manager's real goal was preserving the client relationship, not the specific wording; I proposed a version that led with the two results we were genuinely confident in, was transparent about the one metric still trending in the wrong direction, and paired it with a concrete next step and timeline. This served the relationship-repair goal without requiring me to overstate anything.
4. Be honest about what you would do if the answer had been no. If my manager had insisted on the original framing after that conversation, my actual next step would have been to ask to attach a short written appendix with the caveated numbers, so the honest version existed in the record even if it wasn't the headline; if that had also been refused, I would have escalated to my manager's manager rather than either comply silently or refuse outright, because the stakes (client trust, and my own credibility if the caveated number surfaced later) were high enough to warrant it.
5. Reflect honestly on what you learned, including about your own judgment, not only about the other person. I learned that raising the concern as a specific evidentiary question ("what supports this framing") got further, faster, than raising it as a values statement ("I'm not comfortable with this") would have, because it gave my manager something concrete to respond to.

Worked example
The client update, as originally proposed, said: "engagement is up and the rollout is on track." What the underlying data actually showed: two of three key metrics had improved meaningfully, but the third (a retention metric the client cared about specifically) had been flat to slightly down for three weeks running, with a plausible but unconfirmed hypothesis for why. The version I proposed and we ultimately sent said: "engagement and adoption are both up meaningfully this period; retention is currently flat, and we have identified a likely cause we're testing a fix for over the next two weeks, with a follow-up update once we have results." The client's actual reaction was more positive than my manager expected, specifically because the concrete next step read as more credible than an unqualified "on track" would have.

Trade-offs & pitfalls
The common failure in answering this question is picking an example that is really just "I disagreed with a decision," with no genuine values dimension, or the opposite extreme, an example so severe (fraud, safety, legal risk) that it reads as a one-time crisis story rather than the kind of ordinary, recurring tension this question is actually probing for. Another pitfall is describing the resolution as pure capitulation ("I raised it once, they said no, I dropped it") or pure martyrdom ("I refused and it cost me"), neither of which shows the judgment interviewers are actually testing for: the ability to find a version of the truth that serves both your own integrity and the legitimate underlying goal the other person had.

Direct answer

I turned a nightly "wall of red" into a one-screen summary that showed failures grouped by root cause instead of one row per failing test. The result was that people reviewed a handful of causes each morning instead of hundreds of individual failures, and the team started fixing causes instead of re-running builds. (Below is a story skeleton; swap in your own real numbers, but keep the shape.)

Situation and task

• Our CI (continuous integration, the automated build-and-test system) ran about 2,000 automated tests overnight. The default report was a flat list of failing test names, so on a bad night 240 tests failed and nobody could tell whether that was 240 problems or one.
• I was asked to make the report usable for QA, developers and the engineering manager, who each wanted something different.

Action: metrics I put on it (each tied to a decision)

1. Failures grouped by signature (a signature is a short fingerprint of a failure's error and location, so identical failures group together). Decision: which cause to fix first.
2. Pass rate on the main branch only (the trunk, the shared branch releases are cut from), trended over 14 days. Decision: is the trunk healthy enough to release from?
3. Flake rate per test (a flaky test is one that passes and fails on the same code; rate = runs whose result differs from the previous run divided by total runs). Decision: which tests to quarantine (temporarily stop them from blocking the build while they keep running) or repair.
4. Time from first red build to first fix. Decision: is triage keeping up?

Metrics I deliberately left off

• Total test count and code coverage percentage (they go up whether or not quality does).
• Overall pass rate across every branch (draft branches drown the trunk signal).
• Per-person failure counts (it turns a data tool into a blame tool and invites people to game it).

Technical decision

I added a signature column to the results table (one row per test per run) and a second table linking a signature to its ticket. The dashboard's top panel was a simple query grouping failed rows by signature, with a count of distinct tests and builds. That schema choice, not the chart styling, is what made the rest possible. Here is that query run on a tiny sample (7 result rows, builds 41 and 42, the ticket table linking a signature to its ticket):

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE results (test TEXT, status TEXT, signature TEXT, build INTEGER);
CREATE TABLE signature_ticket (signature TEXT PRIMARY KEY, ticket TEXT);
""")
db.executemany("INSERT INTO results VALUES (?,?,?,?)", [
    ("test_login",    "fail", "expired-credential", 41),
    ("test_profile",  "fail", "expired-credential", 41),
    ("test_export",   "fail", "expired-credential", 41),
    ("test_login",    "fail", "expired-credential", 42),
    ("test_search",   "fail", "timeout-search-db",  42),
    ("test_cart_tax", "fail", "assert-rounding",    42),
    ("test_billing",  "pass", None,                 42),
])
db.execute("INSERT INTO signature_ticket VALUES ('expired-credential', 'QA-311')")

query = """
SELECT r.signature,
       COUNT(*)                 AS failed_results,
       COUNT(DISTINCT r.test)   AS distinct_tests,
       COUNT(DISTINCT r.build)  AS builds,
       COALESCE(t.ticket, 'none yet') AS ticket
FROM results r
LEFT JOIN signature_ticket t ON t.signature = r.signature
WHERE r.status = 'fail'
GROUP BY r.signature
ORDER BY failed_results DESC, r.signature
"""
for row in db.execute(query):
    print(row)

('expired-credential', 4, 3, 2, 'QA-311')
('assert-rounding', 1, 1, 1, 'none yet')
('timeout-search-db', 1, 1, 1, 'none yet')

Six failed rows became three lines: GROUP BY signature collapses rows with the same signature, and the counts show how big each cause is. The same query over the real 240 failed rows is what produced the 9 signatures below.

Result (derived from the example data, not a benchmark)

On the first night we ran it, 240 failing results collapsed into 9 signatures, and the top signature (one expired test credential) explained 150 of them, so 240 items to read became 9. Beyond that, the 15-minute weekly review shifted from arguing about which tests to blame to assigning owners to signatures.

Trade-offs and pitfalls

• Over-grouping can hide a second, unrelated bug behind a large signature, so I kept a drill-down to individual tests.
• A metric nobody acts on is decoration. I kept a metric only if I could name the decision it feeds.
• Be ready to say what you would do differently (for example, I would have interviewed the consumers before choosing metrics, not after).

Benefits (QA perspective)

• Faster feedback to devs, earlier bug detection, higher CI capacity, parallel environment coverage.
• Measurable goals & metric to evaluate success:
  • Reduce wall-clock time — Metric: 95th-percentile pipeline duration (minutes).
  • Increase throughput — Metric: tests completed per hour (or pipelines/hour).
  • Maintain determinism / reduce flakiness — Metric: flake rate = % of flaky test runs (tests with intermittent failures per 1,000 runs).
  • Control resource cost — Metric: average CPU-hours (or cloud cost $) per pipeline.

Definitions

• Wall-clock time: real elapsed time from job start to finish.
• Throughput: total amount of test work finished per unit time.
• Determinism: tests produce same pass/fail given same code and environment.
• Flakiness: intermittent, non-deterministic failures not caused by code changes.
• Resource cost: compute/memory/CI runner costs consumed to run tests.

Main risks & mitigations

• Increased flakiness due to concurrency (race conditions, shared state) — add sandboxing, isolate services, use unique test data.
• Resource contention and higher cost — set sensible shard sizes, autoscale runners, spot instances, budget alarms.
• Non-deterministic ordering masking bugs — run nightly serial/regression runs; add reproducible logs.
• Test orchestration complexity — invest in robust test runner and idempotent setup/teardown.

This gives measurable signals to decide if parallelization yields faster, reliable, and cost-effective CI.

Approach (summary)
Treat tests as items with runtime cost and detection value for current changed lines. Model detection probability for a set S of tests assuming independent failures using per-test historical failure probability p_t and coverage: a line is "caught" if any covering test fails. The objective is maximize overall probability of catching at least one regression across changed lines under a time budget. This objective is monotone submodular, so a greedy marginal-gain-per-time selection gives a (1 - 1/e)-approximation.

Key ideas

• Compute per-line risk (uniform or weighted from commit metadata).
• For a candidate set S, line L caught probability = 1 - product_{t in cov(L) ∩ S} (1 - p_t).
• Objective = 1 - product over lines of (1 - caught_prob(L))? (If lines independent we can sum expected caught-lines instead; use expected detected-lines as surrogate.)
• Use greedy: repeatedly pick test with highest marginal expected-detections per second until budget exhausted.

Pseudocode

# inputs:
# changed_lines: set of lines
# tests: list of {id, time, p_fail, covers: set(lines)}
budget = B
S = set()
remaining = B

# precompute per-test contribution to each line
while True:
    best = None
    best_ratio = 0
    for t in tests - S:
        if t.time > remaining: continue
        # marginal expected new detected lines
        marginal = 0
        for L in (t.covers & changed_lines):
            # current prob line caught by S
            cur = 1 - product((1 - s.p_fail) for s in S if L in s.covers)
            # new prob if add t
            new = 1 - (1 - t.p_fail) * (1 - cur)
            marginal += (new - cur)  # expected additional detection for line L
        ratio = marginal / t.time
        if ratio > best_ratio:
            best_ratio = ratio; best = t
    if not best: break
    S.add(best)
    remaining -= best.time
return S

Complexity

• Let n = #tests, m = #changed_lines. Each greedy iteration costs O(n * avg_coverage). Worst-case iterations ≤ n so O(n^2 * avg_coverage). With indexing per-line covering-tests, compute marginals in O(sum coverage) per iteration.

Limitations & practical notes

• Assumes independence of test failures; real failures correlate—consider using joint-failure statistics or clustering to adjust p_fail.
• Coverage granularity matters (line vs function). Use file/function-level if sparse.
• Surrogate objective (expected lines caught) may differ from true multi-line regression model—tune with historical A/B.
• Cold-start tests have unreliable p_fail; use Bayesian priors.
• For large scale, use lazy-greedy (priority queue) to reduce evaluations, and parallel marginal computations.

Direct answer. The recurring anti-patterns at scale are monolithic tests, sleep-based waits, heavy UI-driven setup, brittle selectors, and no clear test ownership, and each has both a specific consequence and a specific, distinct fix - treating them as one generic "flaky tests" problem misses that the fixes don't transfer between them.

Structured elaboration, per anti-pattern:

• Monolithic tests (one test asserting many unrelated things end-to-end). Consequence: one failure gives no signal about WHICH assertion actually broke, and a single flaky step fails the whole test. Fix: split into focused tests, each asserting one behavior, with shared setup extracted to a fixture rather than repeated inline.
• Sleep-based waits (sleep(3) instead of an explicit wait). Consequence: either flaky (too short under load) or needlessly slow (padded "to be safe"), and the padding compounds across thousands of tests into real CI minutes. Fix: replace with explicit waits on a specific condition (wait_until_visible), never a fixed duration.
• Heavy UI setup (driving the UI to reach a state a direct API/DB call could establish instantly). Consequence: setup time and setup flakiness dominate the actual test. Fix: seed state via API/DB/fixtures, and reserve UI-driven steps for what you're actually testing.
• Brittle selectors (XPath keyed to DOM structure/position, or to translated text). Consequence: any unrelated markup or copy change breaks tests that never touched that feature. Fix: stable attribute-based locators (data-testid), centralized in a locator layer so a real breakage is a one-place fix.
• No clear ownership (a shared suite nobody feels responsible for). Consequence: flaky tests accumulate because fixing someone else's failing test isn't anyone's job; the suite trends toward "always red," and a real regression hides in the noise. Fix: assign per-area ownership (even informally) and a policy that a newly-introduced flaky test blocks the PR that introduced it, not the next person's PR.

Worked example. A monolithic checkout test asserting cart total, tax calculation, AND shipping-address validation in one test body: when it fails, the team cannot tell from CI alone which of the three broke without opening the trace. Splitting it into three focused tests, each seeded via API to skip the earlier UI steps, turns one ambiguous 45-second failure into three fast, specific ones - and only the actually-broken one goes red.

Trade-offs and pitfalls. Fixing brittle selectors and sleep-based waits usually gets prioritized first because they are mechanically easy to grep for; "no clear ownership" is the anti-pattern most often left unaddressed because it's a people/process fix, not a code fix - and it's also the one that lets the other four recur after the initial cleanup.

A regulated, critical-flow mobile banking app should weight testing more conservatively than a typical consumer app: heavier end-to-end and integration coverage on the flows regulators and customers cannot forgive a mistake on, while still keeping a large unit-test base for everything else.

Proposed mix and reasoning

A reasonable balance is roughly 55-60% unit tests, 25-30% integration tests, and 12-15% end-to-end tests, a meaningfully larger end-to-end share than a typical consumer app's roughly 10%. The reasoning: unit tests remain the cheapest way to verify the large volume of calculation and validation logic (balance calculations, transaction limits, fraud-rule evaluation), so they still deserve the majority share; but the CONSEQUENCE of a wiring or integration bug in a banking app (an incorrect balance shown, a transfer that silently fails, a security check that's bypassed) is severe enough, both financially and regulatorily, to justify pulling more of the remaining budget toward integration and end-to-end coverage than a lower-stakes consumer app would.

Where manual testing is still required

Manual testing remains necessary for scenarios that are either too rare, too destructive, or too judgment-dependent to safely automate: security-focused exploratory testing looking for unanticipated vulnerabilities (penetration-style probing rather than a fixed script), regulatory-compliance review where a human needs to confirm the app's behavior matches a written legal requirement's INTENT rather than a literal test assertion, and edge-case account states (fraud holds, disputed transactions, closed-account edge cases) that are expensive to construct realistically in an automated environment and occur rarely enough that automating them may not pay back the investment.

Which areas deserve heavier end-to-end automation

Prioritize end-to-end automation for the flows where a failure is both high-frequency and high-consequence: login and authentication (including biometric and multi-factor paths), balance display accuracy, money movement (transfers, bill pay), and any flow touching regulatory disclosures (required consent screens, mandated notices). These are the flows where "it passed our integration tests" is not sufficient reassurance, precisely because the real risk is in how the FULL assembled system, including the UI layer showing a customer their money, behaves for a real user.

Justifying the investment to stakeholders

To product stakeholders, frame the case around trust and retention: a single visible bug in balance accuracy or a failed transfer does disproportionate damage to a banking app's core value proposition, trust, compared to an equivalent bug in a less consequential app category, so preventing it protects the product's fundamental reason to exist. To security and compliance stakeholders, frame the case around audit and regulatory posture: documented, repeatable automated coverage of critical flows is evidence a regulator can review, and it materially reduces the likelihood of a compliance incident that carries real financial and reputational cost, which is typically the argument that resonates most directly with that audience.

Trade-offs and pitfalls

The pitfall in a regulated context is over-correcting toward end-to-end tests for EVERYTHING out of risk-aversion, which reproduces the ice-cream-cone anti-pattern under a different justification and slows the team down without a corresponding safety benefit for the many lower-stakes flows (help-center content, non-critical settings) that don't carry the same regulatory weight. Apply the heavier end-to-end investment specifically to the flows identified above, not uniformly across the whole app.

Direct answer

The partnerships that pay off under deadline pressure are almost never built in the moment you need them. They come from investing time in a working relationship with a team before there's a specific ask attached, understanding their priorities and vocabulary well enough that when you do need something urgent, they already trust your judgment and don't need to re-derive context from scratch.

Structured elaboration

1. Choose deliberately where to invest. You can't build deep relationships with every team you might someday depend on. Invest ahead of need in the teams whose dependencies are likely to become recurring or critical-path (on the chain of dependent work that directly determines a deadline), based on how your roadmap or their roadmap is shaping up.
2. Invest with no immediate ask attached. Show up to their planning or triage occasionally, offer help on something low-stakes, or spend time understanding how they prioritize their own queue. The absence of a request is what makes it relationship-building rather than a transaction.
3. Learn their vocabulary and criteria, not just their org chart. Knowing how a team actually decides what's urgent (their SLA, or service level agreement, tiers, meaning their committed response and turnaround times, and their escalation triggers) is what lets you frame a future ask in terms they'll immediately recognize as legitimate.
4. Share your own context too. A partnership that pays off later is two-directional: they should understand your team's constraints and cadence well enough that an urgent ask from you doesn't sound out of character.
5. When the moment comes, lean on the relationship, not authority. The payoff isn't that they're obligated to help, it's that they already trust your scoping and don't need to independently verify the ask is real before acting on it.

Worked example

As a backend engineer, I noticed my team periodically needed fast turnaround from the support team but had no real relationship with them beyond ticket queues. Over a few months, with no active request pending, I started sitting in on their triage session once a month, just listening and asking questions about how they decided what jumped the queue. In one of those sessions I noticed a complaint that kept resurfacing: a specific error support couldn't explain, so they were closing the tickets as "can't reproduce." I flagged it to the engineer on our side who owned that area, and made sure support knew we were looking into it even though nothing was urgent yet.

Months later, that same underlying issue caused a customer escalation with a tight deadline attached. I reached out directly to the support lead I'd built rapport with, framed the ask using the same triage language they used internally, and was specific about why it was time-sensitive. Because they already trusted that I didn't cry wolf and that my scoping was accurate, they fast-tracked the escalation ahead of their standard queue without needing the usual back-and-forth to validate it was real.

(Swap the domains freely: the same pattern works with a platform team, a design team, or a data team in place of support, as long as the investment happens before there's an active ask.)

Trade-offs & pitfalls

• Pitfall: relationship-building that's transparently transactional (showing up only when you're about to need something) reads as insincere and doesn't produce the trust you're after.
• Pitfall: investing broadly and shallowly across every team instead of selectively where dependencies are likely to matter. That spreads your own team's time thin for little return.
• Pitfall: treating the payoff as owed. A relationship earns goodwill; it doesn't guarantee compliance, and presuming it does damages the very trust you built.
• Senior differentiator: recognizing which dependencies are likely to become critical-path before they do, and investing ahead of the need rather than starting the relationship the day you first need a favor.

Direct answer
No. 100% code coverage means every line (or branch, depending on the metric) executed at least once during the test run; it says nothing about whether the ASSERTIONS in those tests were correct, whether the code handles every valid input correctly, or whether requirements were even translated into tests in the first place.

Structured elaboration: what coverage tells you vs. what it doesn't

What it tells you: which lines of your existing code your test suite never touches at all. This is genuinely useful as a MINIMUM bar and a gap-finder: a function at 40% coverage almost certainly has untested logic, and that is worth knowing.

What it does NOT tell you:

• Whether the tests that DID run the code actually asserted the correct behavior. A test that calls a function and asserts nothing (or asserts something trivially true) contributes to coverage while verifying nothing.
• Whether every BEHAVIOR was tested, only every LINE. A single test with one input value can cover 100% of a function's statements while testing exactly one equivalence class out of many.
• Whether the code is missing logic entirely. Coverage can only measure code that exists; a forgotten validation check, an unhandled error path that was never written, or a requirement that was silently dropped produces no coverage gap at all, because there is no line to fail to cover.
• Whether combinations of conditions interact correctly (this needs branch or MC/DC coverage, and even those don't guarantee correctness, only that every individual outcome was exercised at least once, not every combination).

Worked example
Consider def safe_divide(a, b):\n if b != 0:\n result = a / b\n return result. A single test calling safe_divide(10, 2) achieves 100% statement coverage for this function (every line runs). That same function crashes with an UnboundLocalError on safe_divide(10, 0), a case the 100%-covered suite never exercised, because the if line "counts" as covered whether its body runs or not. The coverage number was a perfect 100% while a real, reachable bug shipped.

Trade-offs & pitfalls
Treating coverage as a target to hit ("get us to 90%") rather than a diagnostic to read creates a predictable failure mode: developers write tests that execute code paths without meaningfully asserting on their behavior, purely to move the number, which is coverage-metric gaming rather than quality improvement. The healthier framing is coverage as a FLOOR, not a CEILING: low coverage reliably indicates risk, but high coverage is only weak evidence of correctness, and it must be paired with a deliberate test-DESIGN discipline (equivalence partitioning, boundary value analysis, negative testing) to actually reason about what SHOULD be tested, rather than simply exercising whatever the code already does.