HardTechnical

34 practiced

You have to build a predictive model that recommends automation candidates from a pool of manual tests using historical data (test-run frequency, failures found, manual-hours spent, feature churn). Outline feature engineering, model choice, labels, evaluation metrics, and how you'd deploy and retrain the model safely.

Sample Answer

**Approach overview**Build a supervised ranking/classification model that scores manual tests by automation ROI (likelihood automation will save effort/defects). Combine domain features, label defining expected value, conservative deployment and continuous monitoring.

**Feature engineering**- Static: test id, component, owner, priority, flakiness history (consecutive failures, failure variance).- Usage: test-run frequency per week, percent of CI vs. nightly, historical execution time (manual-hours aggregated).- Value: number of unique defects found by test, severity-weighted defects, time-to-detect median.- Maintainability: feature churn (lines changed, commits touching related module), last-modified timestamp.- Derived: ROI proxy = (manual-hours * run-frequency * avg-severity) / maintenance-cost-estimate.- Preprocessing: encode categorical with target/embedding, log-transform skewed numeric, fill recent windows, avoid leakage by using only data available before labeling cutoff.

**Labels**- Regression: estimated annual saved manual-hours or defect-severity-hours if automated.- Binary/ranking: positive if estimated ROI > threshold (e.g., payback < 3 months).- Create labels using historical outcomes: when tests were automated, measure subsequent reduction in manual-hours and defects; use that to derive ground truth.

**Model choice**- Gradient boosting (XGBoost/LightGBM/CatBoost) for tabular ranking with monotonic constraints on sensible features.- Optionally pairwise ranking or LambdaMART for top-K prioritization.- Calibrate outputs to probability-like scores.

**Evaluation**- Offline: precision@k, recall@k, MAP, NDCG for ranking; MAE/RMSE for regression; AUC and PR for binary.- Business metrics: actual manual-hours saved in A/B or retrospective validation, reduction in flaky false positives.- Cross-validation: time-based splits to prevent temporal leakage.

**Deployment & safe retraining**- Feature store and inference service integrated into CI/CD.- Phased rollout: shadow mode → canary on small team → full rollout. Show recommendations in UI for human approval before automatic scheduling.- Monitoring: data drift, model performance (precision@k), calibration, and alerting. Log predictions and outcomes for causal analysis.- Retraining cadence: automated trigger on drift or monthly retrain with holdout evaluation; maintain baseline model and rollback path.- Safeguards: conservative thresholds, human-in-loop gating for high-impact tests, explainability (feature importances, SHAP), and audit trail for decision traceability.

This pipeline ensures prioritized, explainable, and safe automation recommendations tailored to QA operational goals.

HardTechnical

60 practiced

You need to decide whether to automate tests that verify multi-step financial transactions for a regulated product. Discuss regulatory, auditability, and traceability factors that affect the automation decision, and describe the additional controls you'd include in automated tests to meet compliance requirements.

Sample Answer

**Approach (role context)** As a Test Automation Engineer I weigh risk, compliance needs, and operational cost before automating multi-step financial flows. Automation is desirable for repeatability and speed but must satisfy regulators’ expectations for auditability and traceability.

**Regulatory / auditability / traceability factors** - Legal/regulatory guidance: whether law requires manual sign-off, witness, or retention of original records. - Evidence requirements: what artifacts auditors expect (screenshots, payloads, timestamps, approvals). - Non-repudiation: ability to prove who initiated/approved tests and results weren’t tampered with. - Environment parity & data handling: use of synthetic or scrubbed PII; segregation between test and production. - Reproducibility: deterministic test inputs, seeded randomness, and versioned test data. - Segregation of duties: separation between those who write tests and those who approve release-related results. - Retention and exportability: long-term storage, WORM or immutable logs, and standard formats for auditors.

**Controls to include in automated tests** - Immutable audit trail: append-only, timestamped logs stored in WORM storage or a secure log service (e.g., cloud audit logs) with cryptographic checksums. - Evidence capture: for each transaction step capture request/response payloads, screenshots, transaction IDs, and persisted state snapshots; bundle per-test evidence packages. - Signed artifacts & provenance: sign test binaries/configs and record SCM commit hashes and CI build IDs in the report. - RBAC & approvals: require role-based gating for running high-risk suites; integrate manual approval steps in CI for end-to-end runs. - Test data management: use tokenized or synthetic financial data, mask PII, and store test data versions with access controls. - Deterministic & idempotent tests: use fixtures, fixed timestamps or time-mocking, and cleanup routines to avoid flakiness affecting audit outcomes. - Monitoring & alerts: record metrics, anomalous result detection, and notify compliance owners on failures. - Replayability: store inputs so auditors can replay the test against a snapshot environment; include infrastructure-as-code versions. - Retention & export: keep evidence for required retention windows and provide export in standard formats (PDF + JSON manifest) for auditors. - Encryption & key management: encrypt stored evidence and manage keys with HSM/KMS; log key usage.

**Example workflow** Automated pipeline triggers E2E suite in a controlled test tenant → pipeline records SCM commit, build ID → runner executes tests with tokenized data → each step logs signed payloads and screenshots to immutable storage → on completion, CI produces a signed evidence bundle and notifies stakeholders; high-risk runs require an approver to release results to release board.

**Why this meets compliance** These controls provide reproducible, tamper-evident evidence, clear ownership, and controlled data handling—addressing auditor expectations while preserving benefits of automation.

MediumTechnical

33 practiced

Create a protocol for 'quarantining' failing automated tests that cause noise in the CI signal. The protocol should define criteria for quarantine, steps to communicate, temporary gating policies, and metrics to decide when to reinstate or retire the test.

HardTechnical

31 practiced

Design a lightweight policy for when to automate tests that cover flaky third-party integrations (e.g., payment gateway, SMS provider). Consider reliability of provider, error injection capability in test environments, cost per transaction, and SLA-criticality. Provide a decision flowchart (describe steps) and a sample policy decision for a payments system.

Sample Answer

**Approach summary**I propose a lightweight, rule-based decision flow that balances test coverage, cost, and safety. It prioritizes deterministic local tests, controlled integration tests with error-injection, and conservative production-side checks.

**Decision flow (steps)**1. Assess provider reliability: use SLA, historical error rate, and incident frequency.2. Check environment capabilities: can test sandbox inject errors (timeouts, 4xx/5xx, latency)?3. Calculate cost per transaction and rate limits.4. Determine SLA-criticality: does failure block payment completion or user funds?5. Decision branches: - If SLA-critical AND provider unreliable OR historical flakiness > threshold => automate integration tests in CI with error-injection and run nightly against sandbox; add lightweight synthetic probes in production (low-rate). - If sandbox supports safe error-injection AND cost acceptable => include negative-path automated tests in CI; mock for fast unit tests. - If provider reliable, no error-injection, high cost => use mocked/contract tests in CI + canary synthetic checks in production. - Always add circuit-breaker and retry behavior tests (unit + integration where possible).

**Sample policy for Payments system**- Criteria: provider error rate > 0.5% OR any P1 incident in last 90 days => require CI integration tests with error-injection (simulate 3xx/4xx/5xx, latency, duplicate webhook), run nightly.- If cost per transaction > $0.10 and sandbox lacks error injection => use contract tests + 10 synthetic production probes/day behind rate limit and escalation on failure.- SLA-critical flag (payment settlement, refunds): require test coverage for happy path + 3 negative scenarios; failures trigger automatic incident ticket and rollback of deployments touching payment flow.

**Rationale & automation details**- Implement error-injection via provider sandbox or test proxy (WireMock) in pipeline.- Keep heavy integration tests gated to nightly/feature-branch runs to control cost.- Monitor synthetic probes and maintain runbook for provider failures.

EasyTechnical

42 practiced

You have metadata for each test: runs_per_week, last_code_change_days, ui_stability_days, estimated_automation_hours, manual_time_per_run_minutes. In Python, write a function decide_automate(test_metadata) that returns True/False using a simple rule-based heuristic (choose thresholds and explain them in comments). The function should be easily adjustable for threshold tuning.

Sample Answer

**Approach**- Simple weighted-rule heuristic: prefer automating tests that run frequently, have low UI instability, high manual runtime, and reasonable automation effort. Thresholds are configurable constants at top for easy tuning.

python

def decide_automate(meta):
    """
    meta: dict with keys:
      - runs_per_week (int)
      - last_code_change_days (int)
      - ui_stability_days (int)
      - estimated_automation_hours (float)
      - manual_time_per_run_minutes (float)
    Returns True if test should be automated, False otherwise.
    """
    # Configurable thresholds (tune these as project needs)
    MIN_RUNS_PER_WEEK = 3           # automation pays off if run >= this weekly
    MAX_LAST_CHANGE_DAYS = 14       # avoid automating if code changes very frequently
    MIN_UI_STABILITY_DAYS = 7       # UI should be stable enough to avoid flakiness
    MAX_AUTOMATION_HOURS = 8        # cap on acceptable automation effort
    MIN_MANUAL_TIME_MIN = 10        # manual time saved per run

    # Simple scoring / guard checks
    if meta.get("runs_per_week",0) < MIN_RUNS_PER_WEEK:
        return False
    if meta.get("last_code_change_days",0) < MAX_LAST_CHANGE_DAYS:
        return False
    if meta.get("ui_stability_days",0) < MIN_UI_STABILITY_DAYS:
        return False
    if meta.get("estimated_automation_hours", float("inf")) > MAX_AUTOMATION_HOURS:
        return False
    if meta.get("manual_time_per_run_minutes",0) < MIN_MANUAL_TIME_MIN:
        return False
    # If all guards pass, automate
    return True

**Explanation & Tuning**- Thresholds chosen to balance ROI vs maintenance risk: frequent runs and longer manual time justify automation; recent code churn and unstable UI increase maintenance cost; keep automation effort bounded.- To tune: adjust constants, or replace final boolean with a score (weight factors) to rank candidates.

**Edge cases**- Missing keys default conservatively to False- For flaky UI or very high automation hours, prefer manual or exploratory testing first

Manual vs. Automated Testing Decision Making Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank