Software Testing and Assertions Questions

Core software testing and debugging practices, including designing tests that exercise normal, edge, boundary, and invalid inputs, writing clear and maintainable unit tests and integration tests, and applying debugging techniques to trace and fix defects. Candidates should demonstrate how to reason about correctness, create reproducible minimal failing examples, and verify solutions before marking them complete. This topic also covers writing effective assertions and verification statements within tests: choosing appropriate assertion methods, composing multiple assertions safely, producing descriptive assertion messages that aid debugging, and structuring tests for clarity and failure isolation. Familiarity with test design principles such as test case selection, test granularity, test data management, and test automation best practices is expected.

HardSystem Design

64 practiced

Design an accessibility testing strategy for a mobile app that includes automated checks (color contrast, semantic labels), manual exploratory checks with assistive tech, CI integration, test data for different accessibility settings (font size, high contrast), and a triage process for accessibility regressions reported by users.

Sample Answer

Requirements & constraints:- Functional: detect a11y regressions for mobile app (iOS/Android) including color contrast, semantic labels, focus/order, dynamic type, screen reader output.- Non-functional: CI gating for PRs (fast feedback), nightly full-suite, maintainable, actionable bug reports, support for variable device settings (font sizes, high-contrast, RTL).

High-level architecture:- Local/CI pipelines → Automated static & runtime checks → Manual exploratory runbooks → Triage & bug lifecycle- Artifacts: test reports, screenshots/videos, accessibility logs, semantic tree dumps

Core components:1. Automated checks (fast, run on PRs) - Static: lint rules (Android Lint, Xcode Accessibility Audit, custom linters) to catch missing contentDescription/label, duplicate IDs. - Visual/color contrast: integrate tools (axe-core for webviews, custom contrast checks for native using screenshot + pixel sampling, or use Tenon/ColorContrastAnalyzer). Run against color tokens and rendered screens. - Semantic verification: UI snapshot + semantic tree diff (AccessibilityNodeInfo/AXHierarchy) to assert labels, traits, hints. - Focus/order: unit tests that validate navigation order for complex screens. - Implementation: test harness using Appium / Espresso / XCTest + accessibility SDKs; tests run headless/emulator for speed.

2. Manual exploratory checks (weekly / release) - Testers run guided checklist with assistive tech: VoiceOver, TalkBack, Switch Control, TalkBack + TalkBack Verbose mode, dynamic type large sizes, inverted/high-contrast modes, magnifier. - Provide scripts: flows, expected behavior, and capture steps (screen recording, semantic tree). - Accessibility champions in PM/design for UX edge cases.

3. CI integration - PR-level: run static linters + a small set of smoke accessibility UI tests (<5s per test) — block merge on critical failures. - Post-merge: nightly full suite on matrix of emulator/device profiles (iOS/Android × font sizes × high-contrast × RTL) producing artifacts saved to S3 and linked in issue. - Use parallelization and device farm (Firebase Test Lab / AWS Device Farm) for scale.

4. Test data & device profiles - Profiles: Normal, Large, Extra Large, XXL; High Contrast/Invert; RTL; Magnification. - Seed data: varied content lengths, localized strings, emojis, long accessibility labels. - Automate environment setup (ADB/UIAutomation scripts / simctl) to toggle settings before tests.

5. Triage & regression process - In-app feedback routing: users can report with "Accessibility issue" → collect reproduction steps, device/profile metadata, logs, optional screen recording. - Triage team (QA eng + accessibility engineer + PM): classify severity (Blocker/Critical/Minor), reproduce, attach logs and semantic dumps. - Workflow: create labeled bug in tracker with required fields (steps, profile, semantic tree, screenshot/video). Critical → hotfix branch with CI priority. - Regression detection: nightly diff of semantic trees/screenshots against golden baseline; fuzzy image diff + semantic diffs raise automated issues to triage.

Metrics & monitoring:- Key metrics: number of a11y regressions per release, time-to-fix, test coverage (screens with automated checks), user-reported a11y bugs.- Dashboards displaying failing rules, device/profile failure rate.

Trade-offs & considerations:- Emulators faster but may miss device-specific a11y behaviors — include real-device farm for nightly.- Balance between strict PR blocking rules (prevents regressions) and developer velocity — enforce critical rules, warn on non-critical.- Maintainability: keep rules small, add tests with feature PRs, provide onboarding docs and pre-commit hooks.

This strategy provides layered defense: fast automated gates, deep nightly checks across accessibility settings, rigorous manual evaluation with assistive tech, and a clear user-focused triage process to resolve regressions quickly.

HardTechnical

60 practiced

Design a strategy for non-deterministic performance testing where noise from environment is high. Explain sample collection, statistical tests to detect regressions, baseline management, automated comparisons for PRs, and how to minimize false positives using confidence intervals and repeated runs.

Sample Answer

Requirements & goals:- Detect real performance regressions despite high environmental noise.- Keep false positives low for CI/PR feedback.- Maintain usable baselines and automate comparisons.

Sampling strategy:- Run microbenchmarks in isolation where possible (cpu affinity, cgroups, no other jobs) and collect many samples per configuration (n ≥ 30). For noisy integration tests, use longer runs and record per-iteration latencies, GC/CPU/mem metrics, and timestamps.- Randomize order and warm up to remove cold-start effects. Capture environment tags (kernel, CPU freq governor, cloud instance id).

Statistical testing:- Use non-parametric tests robust to non-normal noise: Mann–Whitney U for median shifts, and Kolmogorov–Smirnov to detect distribution changes. For mean-sensitive metrics use Welch’s t-test if normality roughly holds (after log-transform).- Report effect size (Cliff’s delta or Cohen's d) and p-value. Prefer two-sided tests with alpha=0.01 for CI runs; for CI gating use alpha=0.001 to reduce false positives.

Baseline management:- Maintain time-stamped baselines per branch/commit range with metadata. Use rolling baselines (e.g., last 10 stable commits) and a “golden” baseline from release tags. Store raw samples and summary stats.

Automated PR comparisons:- On PR, run a fixed benchmark matrix: repeat k times (e.g., 5 independent CI jobs × 30 iterations). Aggregate samples and compare against the rolling baseline using the chosen tests. Gate PRs only if: - p-value < threshold AND - effect size > practical threshold (e.g., ≥5% latency increase) AND - change persists across independent runs.

Minimizing false positives:- Use confidence intervals: compute 95% CI for median (bootstrap) and only flag if CI bounds exclude baseline median with required effect size.- Require replication: regression declared only if observed in multiple independent CI runs (e.g., 3/5).- Apply multiple-testing correction (Benjamini-Hochberg) when checking many metrics.- Stabilize environment: pin CPU governor, disable hyperthreading if possible, and run on reserved instances to reduce variance.

Example automation flow:1. PR triggers N parallel CI benchmark jobs.2. Each job: warmup → 30 iter samples → upload raw data.3. Aggregator fetches baseline, computes bootstrapped medians, CIs, effect sizes, p-values, corrects for multiple tests.4. Post comment on PR with visual plots (violin + CI), verdict, and raw-data links.

Trade-offs:- More repeats increase signal but cost time; choose risk-based thresholds for critical paths.- Non-parametric tests are robust but less powerful; combine with effect-size checks to avoid sensitivity loss.

This yields reproducible, statistically defensible regressions with low false-positive rate while remaining practical for CI.

EasyTechnical

51 practiced

Describe the test pyramid and how it applies to a complex SaaS product. Explain recommended test distribution between unit, integration, and end-to-end tests, and discuss exceptions where you may intentionally deviate from the pyramid (for example, critical UI flows or external compliance requirements).

MediumTechnical

50 practiced

How would you test time-dependent behavior? For JavaScript code that uses setTimeout to delay a callback, show how to write deterministic tests using fake timers (for example with Jest or sinon). Include an example of advancing the fake clock and asserting that callbacks have been invoked.

Sample Answer

To test time-dependent behavior deterministically, replace real timers with fake timers so your test controls time. Use Jest’s built-in fake timers or sinon’s fake timers to advance the clock and assert that delayed callbacks run exactly when expected. Key ideas: install fake timers, trigger the code under test, advance time (or run all timers), assert side effects, and restore real timers.

Jest example (modern timer API):

javascript

// myTimer.js
export function runAfterDelay(cb, ms) {
  const id = setTimeout(() => cb('done'), ms);
  return () => clearTimeout(id);
}

javascript

// myTimer.test.js
import { runAfterDelay } from './myTimer';

beforeEach(() => {
  jest.useFakeTimers(); // install fake timers
});

afterEach(() => {
  jest.useRealTimers(); // restore
});

test('calls callback after delay', () => {
  const cb = jest.fn();
  runAfterDelay(cb, 1000);

  // callback not called immediately
  expect(cb).not.toHaveBeenCalled();

  // advance time by 500ms -> still not called
  jest.advanceTimersByTime(500);
  expect(cb).not.toHaveBeenCalled();

  // advance remaining 500ms -> callback should be called once
  jest.advanceTimersByTime(500);
  expect(cb).toHaveBeenCalledTimes(1);
  expect(cb).toHaveBeenCalledWith('done');
});

Sinon example:

javascript

import sinon from 'sinon';
import { runAfterDelay } from './myTimer';

test('sinon fake timers example', () => {
  const clock = sinon.useFakeTimers();
  try {
    const cb = sinon.spy();
    runAfterDelay(cb, 1000);

    clock.tick(999);
    sinon.assert.notCalled(cb);

    clock.tick(1);
    sinon.assert.calledOnce(cb);
    sinon.assert.calledWith(cb, 'done');
  } finally {
    clock.restore();
  }
});

Key points:- Prefer deterministic advances (advanceTimersByTime / clock.tick) over real waits.- Clean up fake timers to avoid leaking into other tests.- For promises or requestAnimationFrame, use the appropriate helpers (jest.runAllTimers, jest.advanceTimersToNextTimer) and consider async/await when code schedules microtasks.- This yields fast, reliable tests unaffected by environment timing variability.

MediumTechnical

58 practiced

When rolling out a new feature, describe a comprehensive test plan using feature flags and canary releases. Include pre-rollout tests (unit, integration, canary smoke tests), monitoring signals to watch during canary, rollback criteria, and how to automate promotion of the flag when metrics are healthy.

Unlock Full Question Bank

Get access to hundreds of Software Testing and Assertions interview questions and detailed answers.

Join thousands of developers preparing for their dream job.