Testing Strategy and Error Handling Questions

Combines broader testing strategy for systems with a focus on error handling, resilience, and user experience when failures occur. Topics include designing a testing strategy that covers unit tests, integration tests, end to end tests and exploratory testing, applying the testing pyramid, defining error boundaries and recovery paths, graceful degradation and fallback strategies, user feedback and error messaging, fault injection and resilience testing, logging and observability to detect and reproduce errors, and validating error handling behavior across environments and edge cases.

MediumTechnical

31 practiced

Explain strategies for managing test data while preserving user privacy: compare synthetic data generation, partial anonymization, and production data cloning with masking. For each approach mention pros/cons, compliance considerations, and one tool or technique you'd use.

Sample Answer

High-level summary: managing test data requires balancing fidelity (realistic behavior), developer productivity, and user privacy/compliance. Below I compare three common approaches: synthetic data generation, partial anonymization, and production cloning with masking.

1) Synthetic data generation- Pros: No real PII, can create balanced edge cases, scalable and deterministic for tests, safer for sharing with external teams.- Cons: May miss subtle production correlations and distributional nuances; building realistic generators can be time-consuming.- Compliance: Often lowest compliance risk if generators never reference real records; document generation rules and prove no re-identification paths.- Tool/technique: Faker + property-based generators or tools like Synthpop (for tabular) or Mockaroo; for complex distributions use differential privacy or generative models (CTGAN).

2) Partial anonymization (tokenization, hashing, generalization)- Pros: Keeps real-world distributions and relationships, easier to reproduce production bugs.- Cons: Risk of re-identification if not done correctly (linking attacks); choice of fields to redact is tricky.- Compliance: Must follow minimization principles, keep mapping keys secure, log access; consider irreversible transformations for high-risk fields.- Tool/technique: Format-preserving tokenization (Vault Tokens, Protegrity) or irreversible hashing with salts stored separately; use k-anonymity checks.

3) Production cloning with masking- Pros: Fast to provide realistic, end-to-end datasets; high fidelity for integration and performance tests.- Cons: If masking is weak or missed, high privacy risk; operational overhead for automated masking pipelines and storage controls.- Compliance: Treat as production-level process—access controls, audit trails, data retention policies, and validation that masking is irreversible for PII; consider legal approvals for any sensitive retention.- Tool/technique: ETL-based masking pipelines (Airflow + custom masking jobs) or specialized tools (Dataguise, IBM InfoSphere Optim, AWS Glue + hashing/masking routines).

Recommendation: For unit and CI tests use synthetic data; for integration/repro of production issues use masked clones with strict controls; for analytics/testing of model behavior consider hybrid: seed synthetic data using production-derived statistical models. Always enforce least privilege, encryption at rest/in transit, audit logging, and periodic privacy/QA checks (re-identification testing).

MediumTechnical

34 practiced

Plan a fault injection test for a payment-processing flow in staging. Your plan should include safety controls (to avoid charging real customers), which faults to inject (latency, 5xx, dropped messages), what metrics/alerts to monitor, how to roll back, and success criteria for the exercise.

Sample Answer

Situation/Scope: Run a targeted fault-injection (chaos) exercise in the staging payment-processing flow (authorization → capture → ledger update → notification) to validate resilience without affecting real customers or production data.

Plan:1. Safety controls (must before any injection)- Environment: staging isolated network with test doubles for payment gateways and external services. Ensure staging uses sandbox payment credentials and synthetic customer accounts flagged as "chaos-test".- Data: scrubbed/anonymized dataset; payments limited to $0.01 or synthetic tokens; block any staging-to-production outbound routes via firewall/policy.- Approval & runbook: sign-off from SRE, payments PM, and compliance. Scheduled low-impact window and a designated incident commander.- Circuit breakers: feature-flag toggles to disable injections quickly; kill switch that reverts network rules or restores service from snapshot.

2. Faults to inject (gradual, scoped)- Latency: add 200–2000ms latency to gateway and internal microservice RPCs for 10–30s windows.- 5xx errors: return controlled 500/502 from gateway and ledger service at varying rates (5%, 20%, 80%).- Dropped messages: drop X% of messages on the message bus (e.g., 10–50%) and simulate duplicate deliveries.- Partial outage: shutdown one instance of the payments service to test autoscaling/failover.

3. Metrics & alerts to monitor- Business metrics: payment success rate, authorization-to-capture ratio, average time-to-complete payment.- System metrics: request latency P50/P95/P99, error rate (4xx/5xx), queue length, consumer lag, DB write failures.- SLO/SLA indicators and any on-call alerts (pager flood).- Observability: distributed traces showing error spans, logs with correlation IDs, health-check dashboards.

4. Rollback / Mitigation- Immediate: trigger kill switch to stop fault injector; revert network rules; flip feature flag to bypass affected components (e.g., route to mock).- Secondary: scale up healthy replicas, restart failing pods, rollback recent deployments if correlated.- Postmortem: collect traces/logs, preserve state snapshots.

5. Success criteria- Safety: no real customer charges, no staging-to-production contamination.- Resilience: payment success rate degrades within acceptable SLO (e.g., ≥95% for low-impact faults, clear graceful degradation for severe faults).- Observability: faults are detected by alerts within target time; traces and logs allow root-cause identification.- Recovery: system automatically or operator-assisted recovers to baseline within defined RTO (e.g., <5 minutes) and without data loss or duplicate charges.- Learnings: documented mitigations, added tests, and at least 2 follow-up actions (config or code) assigned.

Run incremental experiments (start small fault percentage/latency, increase), validate checkpoints at each step, and produce a blameless postmortem with actionable remediation.

EasyTechnical

34 practiced

How would you test that error messages and interactive error controls are accessible to keyboard and screen-reader users? Provide a short plan including tools, example assertions, and how you'd include these tests in CI without manual QA.

Sample Answer

Situation: I need to ensure form error messages and interactive error controls (e.g., "retry", "dismiss") are usable by keyboard and screen-reader users, and prevent regressions via CI.

Plan (high level):- Automated unit/accessibility checks for component-level behavior.- End-to-end tests that exercise keyboard flows and screen-reader-relevant attributes.- Run these tests in CI and fail the build on regressions.

Tools:- axe-core / jest-axe for unit/component tests- Playwright (or Cypress) + @axe-core/playwright for E2E + keyboard automation- Testing Library (@testing-library/react) for component assertions- CI: GitHub Actions / GitLab CI to run headless browser tests- Optional manual verification: NVDA/VoiceOver for spot checks

Example checks / assertions:- Error message exists in DOM and is programmatically associated: - expect(input).toHaveAttribute('aria-describedby', 'email-error') - expect(document.getElementById('email-error').textContent).toMatch(/required/)- Error is announced to screen readers: - error container has role="alert" or aria-live="polite" - expect(errorEl).toHaveAttribute('role','alert')- Keyboard focus and interactive control behavior: - Tab to input, submit invalid -> focus moves to first invalid field or to the error summary - Simulate keyboard: page.keyboard.press('Tab'); page.keyboard.press('Enter'); expect(page.evaluate(() => document.activeElement.id)).toBe('email')- Interactive controls are reachable by keyboard and accessible: - expect(button).toHaveAttribute('tabindex','0') or naturally tabbable; expect(button).toHaveAccessibleName('Retry sending email')- Axe assertions: - await expect(await axe.run(page)).toHaveNoViolations() (or assert zero violations for relevant rules like aria-allowed-attr, color-contrast, etc.)

Example Playwright + axe snippet:

javascript

// playwright-axe example
const { test, expect } = require('@playwright/test');
const { injectAxe, checkA11y } = require('axe-playwright');

test('error messages are accessible', async ({ page }) => {
  await page.goto('http://localhost:3000/form');
  await page.click('button[type=submit]'); // trigger validation
  await injectAxe(page);
  await checkA11y(page, null, {
    detailedReport: true,
    rules: { 'color-contrast': { enabled: true } }
  });
  // keyboard focus assertion
  await page.keyboard.press('Tab');
  expect(await page.evaluate(() => document.activeElement.getAttribute('aria-describedby'))).not.toBeNull();
  // check role/aria-live
  const role = await page.getAttribute('#email-error','role');
  expect(role).toBe('alert');
});

CI integration:- Add a job that installs dependencies, starts the app (or uses test build), runs Playwright tests headless.- Fail on any test or axe violation. Use caching and parallel jobs to keep runtime reasonable.- Consider baselining: if temporary known issues exist, record them and require explicit triage before allowing CI to ignore.- Run full accessibility E2E nightly if heavy; run lightweight unit/axe checks on every PR.

Why this works:- Combines programmatic checks (aria relationships, role, focus) with automated screen-reader–relevant testing (aria-live/role and keyboard flows) and axe rules to catch common a11y regressions, and enforces them in CI so no manual QA step is required for regressions.

MediumSystem Design

36 practiced

Design a testing strategy for a new stateless microservice that exposes a REST API and depends on a Redis cache and a downstream auth service. List which tests you'd write (unit/integration/e2e/exploratory), where they run (local/CI/staging), and how you'd validate error handling paths (timeouts, 5xx from auth, Redis failures).

Sample Answer

Requirements clarification:- Stateless REST microservice that reads/writes Redis and calls downstream auth service.- Need correctness, resilience (timeouts, 5xx), and graceful degradation when Redis/auth fail.

Testing strategy (types, examples, where they run):

1) Unit tests- What: handler logic, input validation, routing, serialization, small helpers.- How: mock Redis and auth client (use test doubles/fakes).- Where: local + CI (fast, on every PR).- Examples: JWT parsing returns 401 when token malformed; request body validation rejects missing fields.

2) Integration tests (component-level)- What: service wired to real Redis (test instance) and a stubbed auth service (or contract mock).- How: use testcontainers or ephemeral Redis, and a controllable auth stub that can emit 200/401/500/delay.- Where: CI (nightly or per-merge) and staging.- Examples: successful flow writes cache; cache miss triggers auth call; cache hit avoids auth.

3) End-to-end (e2e)- What: full system with real auth service in staging or production-like environment.- How: run HTTP-level tests hitting deployed endpoints, assert behavior and telemetry.- Where: staging (pre-prod).- Examples: happy path, retries, auth token expiry flows, TTL behavior.

4) Chaos / resilience tests & exploratory- What: simulate network timeouts, Redis unavailability, auth 5xx, high latency, Redis evictions.- How: use fault injection (Istio/chaos monkey), kill Redis, or use proxy to delay responses.- Where: staging (and periodic in pre-prod), possibly scheduled chaos in prod on small scope.- Examples: auth 5xx -> service returns 503 with retry/backoff; Redis down -> fall back to stateless behavior or return degraded response.

Validating error handling paths:- Timeouts: test by configuring auth stub to delay beyond client timeout; assert service times out, logs metric, returns configured 504 or retry policy applied. Include unit tests for timeout logic and integration tests with delayed stub.- 5xx from auth: stub 500/502; assert exponential backoff, circuit breaker opens after threshold, client receives 502/503 or well-defined error JSON. Verify metrics (error count, circuit state) in integration/staging.- Redis failures: simulate connection errors and slow responses; assert service either retries short number of times, falls back (if possible) or returns consistent error. Test that cache write failures don't corrupt responses and are logged/metricized.

Additional practices:- Contract tests for auth client (consumer-driven pact tests) to catch breaking changes.- CI gates: unit + integration smoke must pass; e2e gated on staging approval.- Observability: assert that errors produce metrics/traces in tests (use in-memory exporters) so monitoring detects regressions.

This coverage balances fast feedback (unit/CI), realistic behavior (integration), production-like validation (staging/e2e), and resilience (chaos/exploratory).

HardTechnical

34 practiced

A service experiences cascading failures when downstream latency spikes. Describe how you'd design and test bulkhead isolation and circuit-breaker strategies to prevent such cascades. Include test scenarios that prove isolation between components and show how you'd measure effectiveness.

Sample Answer

Clarify scope and goals:- Goal: prevent a latency spike in downstream service B from degrading service A or other downstreams; preserve availability for unaffected flows and enable graceful recovery.- Constraints: service SLAs, platform (k8s, JVM, node), traffic patterns, observable telemetry.

Design (high level):1. Bulkheads- Partition resources by logical traffic or downstream: dedicate thread pools / worker pools / connection pools and timeouts per downstream or per tenant. For example, JVM: a dedicated executor + bounded queue for calls to B; k8s: separate deployment/replicasets or CPU/IO limits by path.- Apply quotas/rate limits plus priority for critical flows.2. Circuit Breakers- Per downstream circuit breakers with states (CLOSED -> OPEN -> HALF-OPEN) and metrics: sliding window for failures, latency, and timeouts. Use adaptive thresholds (p95/p99 latency exceed or error rate > X% over N requests).- Fast-fail on OPEN to preserve resources; HALF-OPEN probes limited concurrency.

Implementation notes:- Timeouts shorter than resource queues; use fencing so blocked threads don’t accumulate.- Integrate with retries/backoff but only for idempotent operations.- Centralized policy but local enforcement for low-latency decision.

Testing strategy (scenarios to prove isolation + measure effectiveness):1. Baseline measurement- Measure p50/p95/p99 latency, error rate, throughput of service A and B under normal load.2. Downstream latency spike (isolation test)- Inject latency into B (chaos experiment) while directing normal traffic to other downstreams.- Expected: calls to B hit its bulkhead queue and/or breaker opens; other downstreams' latency and error rate remain within baseline SLA.- Metrics: compare p95/p99 of other downstreams vs baseline; measure thread-pool saturation and queue length.3. Downstream failure / error surge (circuit-breaker test)- Make B return errors at configurable rate. Verify breaker opens when configured threshold crossed; subsequent requests to B fail-fast; circuit transitions to HALF-OPEN and recovers on success.- Metrics: time-to-open, time-to-recover, error budget consumed, number of retries attempted.4. Resource exhaustion (bulkhead capacity test)- Exhaust executor threads for B; verify only B is throttled and A continues to serve other routes. Measure blocked threads and dropped requests.5. Mixed load with critical vs best-effort traffic- Ensure bulkheads and rate limits prioritize critical flows; measure SLA adherence for critical flows.6. End-to-end resilience under load- Run load test where both traffic volume and induced latencies vary; validate overall system SLOs remain acceptable (error budget, latency percentiles).

How to measure effectiveness (KPIs):- Isolation effectiveness: delta in p95/p99 latency and error rate for non-impacted components compared to baseline (target: < 10% degradation).- Time-to-failure-containment: time from injected fault to breaker open or throttling enforcement.- Recovery time: time from fault removal to healthy state (< configured half-open window + successful probes).- Resource metrics: thread pool utilization, queue drop counts, CPU/memory.- User impact: percentage of requests served successfully vs failed-fast; SLO adherence (e.g., <1% error budget burn).

Test tooling and automation:- Use Chaos Monkey / Gremlin to inject latency/errors; k6 or Gatling for load; Jaeger/Zipkin for traces; Prometheus/Grafana for metrics; automated test pipelines to run scenarios in staging and periodically in production "blast radius" with canary.- Add synthetic transactions to verify critical paths.

Trade-offs and tuning:- Aggressive thresholds protect system but increase user-facing failures; tune thresholds using historical traffic and error distributions.- Bulkhead granularity: per-endpoint is safer but costs resources; group logically where appropriate.- Prefer observability-first: rich metrics and distributed tracing to diagnose when breakers trigger.

Example quick config (conceptual):- Circuit: sliding-window 120s, minRequests=20, failureRateThreshold=25%, openDuration=30s, halfOpenMaxRequests=5.- Bulkhead: maxThreads=50, queueSize=100, rejectionPolicy=fail-fast.

This combination, validated by the test scenarios and KPIs above, demonstrates isolation and measurable resilience against cascading latency failures.

Unlock Full Question Bank

Get access to hundreds of Testing Strategy and Error Handling interview questions and detailed answers.

Join thousands of developers preparing for their dream job.