Assertions and Behavior Verification Questions

Unit tests vs Integration tests (backend web service)

Unit tests

• Purpose: Verify a single unit (function, class, or handler) in isolation.
• Examples: controller method returns correct response for given input; service layer computes pricing with discounts; utility that parses JWTs.
• Techniques: mock external dependencies (DB, network, caches), use fast in-memory assertions.
• Pros: very fast (ms), deterministic, easy to run locally, pinpoint failures.
• Cons: can miss interaction bugs; over-mocking hides contract issues; maintenance if internal design changes.

Integration tests

• Purpose: Verify components work together and integration contracts hold.
• Examples: API endpoint -> service -> repository writes to DB and returns expected JSON; end-to-end flow that includes auth, DB write/read, and message queue publish.
• Techniques: use test DB (dockerized), real message broker or an embedded alternative, or lightweight containerized infra.
• Pros: catch real-world issues (SQL schema, serialization, transaction boundaries); higher confidence.
• Cons: slower (seconds–minutes), can be flaky (timing, network), higher maintenance.

Trade-offs and practices

• Pyramid approach: many unit tests, fewer integration tests, minimal end-to-end tests.
• Speed vs coverage: unit tests for logic correctness and quick feedback; integration tests for contract and environment issues.
• Flakiness: reduce nondeterminism in integration tests (fixed seeds, retries, deterministic test data); isolate tests (DB reset per test).

When to use real DB vs mocks in CI

• Use mocked DB or in-memory DB for most unit tests to ensure speed and isolation.
• Use a real database in integration tests run in CI to validate migrations, schema, query performance, and SQL dialect behaviors. Run these in a dedicated test stage (nightly or pre-merge for release branches) or in CI with stable containerized DB instances.
• Fast feedback loop: run unit tests + lightweight integration tests (using sqlite or in-memory alternatives if behavior matches) on every push; run full integration suite with the real DB (Postgres/MySQL) and other services on PR gates or nightly builds.

Example rule of thumb:

• Commit/PR → run unit tests + lint + quick integration subset (mock external APIs, real DB optional).
• Merge to main → full integration tests against real infra.

Golden-master (snapshot) testing captures the expected output of a module and compares future runs against that canonical "golden" to detect regressions. For nested JSON outputs this requires stable, comparable artifacts and deterministic normalization.

How to create golden files

• Run the module in a controlled environment, produce pretty-printed JSON, and store as ./testdata/goldens/<feature>/<version>.golden.json.
• Commit goldens to source control in a dedicated directory and reference them from tests.

Masking/normalizing nondeterministic fields

• Before comparison, parse JSON and replace or remove fields like timestamps, UUIDs, or ephemeral IDs.
• Normalize ordering for arrays/objects where order is not significant (sort keys, sort arrays by stable key).
• Example Python normalizer:

import json
def normalize(obj):
    if isinstance(obj, dict):
        return {k: normalize(v) for k,v in sorted(obj.items()) if k not in ("created_at","id","request_id")}
    if isinstance(obj, list):
        return sorted([normalize(e) for e in obj], key=lambda x: json.dumps(x, sort_keys=True))
    if isinstance(obj, str) and looks_like_uuid_or_ts(obj):
        return "<MASKED>"
    return obj

Time-based fields can be replaced with fixed tokens or truncated to coarse granularity.

Versioning golden files

• Include a version in the path or filename (feature_v1.golden.json) and document schema/contract changes in PRs.
• Use semantic versioning for major behavioral changes: v1 → v2 when shape/meaning changes.
• Keep history; do not overwrite without review.

Safe review/update process

• Require a test failure demonstrating the diff.
• In PR, include: failing test output, reason for change, and updated golden file.
• Run CI with deterministic seeds and normalized comparisons; add human review step to inspect diffs (use json-diff tools with ignored paths).
• For large updates, add automated migration scripts and incremental commits to ease review.

Best practices

• Keep goldens small and focused per feature.
• Combine unit snapshots (small, deterministic) and integration snapshots (end-to-end).
• Automate normalization in test harness; never rely on manual edits alone.

Requirements & goals:

• Detect real performance regressions despite high environmental noise.
• Keep false positives low for CI/PR feedback.
• Maintain usable baselines and automate comparisons.

Sampling strategy:

• Run microbenchmarks in isolation where possible (cpu affinity, cgroups, no other jobs) and collect many samples per configuration (n ≥ 30). For noisy integration tests, use longer runs and record per-iteration latencies, GC/CPU/mem metrics, and timestamps.
• Randomize order and warm up to remove cold-start effects. Capture environment tags (kernel, CPU freq governor, cloud instance id).

Statistical testing:

• Use non-parametric tests robust to non-normal noise: Mann–Whitney U for median shifts, and Kolmogorov–Smirnov to detect distribution changes. For mean-sensitive metrics use Welch’s t-test if normality roughly holds (after log-transform).
• Report effect size (Cliff’s delta or Cohen's d) and p-value. Prefer two-sided tests with alpha=0.01 for CI runs; for CI gating use alpha=0.001 to reduce false positives.

Baseline management:

• Maintain time-stamped baselines per branch/commit range with metadata. Use rolling baselines (e.g., last 10 stable commits) and a “golden” baseline from release tags. Store raw samples and summary stats.

Automated PR comparisons:

• On PR, run a fixed benchmark matrix: repeat k times (e.g., 5 independent CI jobs × 30 iterations). Aggregate samples and compare against the rolling baseline using the chosen tests. Gate PRs only if:
  • p-value < threshold AND
  • effect size > practical threshold (e.g., ≥5% latency increase) AND
  • change persists across independent runs.

Minimizing false positives:

• Use confidence intervals: compute 95% CI for median (bootstrap) and only flag if CI bounds exclude baseline median with required effect size.
• Require replication: regression declared only if observed in multiple independent CI runs (e.g., 3/5).
• Apply multiple-testing correction (Benjamini-Hochberg) when checking many metrics.
• Stabilize environment: pin CPU governor, disable hyperthreading if possible, and run on reserved instances to reduce variance.

Example automation flow:

1. PR triggers N parallel CI benchmark jobs.
2. Each job: warmup → 30 iter samples → upload raw data.
3. Aggregator fetches baseline, computes bootstrapped medians, CIs, effect sizes, p-values, corrects for multiple tests.
4. Post comment on PR with visual plots (violin + CI), verdict, and raw-data links.

Trade-offs:

• More repeats increase signal but cost time; choose risk-based thresholds for critical paths.
• Non-parametric tests are robust but less powerful; combine with effect-size checks to avoid sensitivity loss.

This yields reproducible, statistically defensible regressions with low false-positive rate while remaining practical for CI.

Requirements:

• Run unit, integration, and UI (end-to-end) tests on every PR.
• Fast feedback (<10–20 min for PRs ideally), reliable gating, flaky detection, artifact retention.
• Tech: React frontend, Node backend. CI: GitHub Actions (example), artifacts in S3/GitHub Actions artifacts, test results to a dashboard (JUnit/Allure).

High-level pipeline:

1. Pre-checks: lint, typecheck (fast).
2. Unit tests (frontend + backend) — parallel shards.
3. Integration tests (API + DB) — in isolated ephemeral DB containers.
4. UI tests (Playwright/Cypress headless) — run in parallel runners or browser shards.
5. Post: collect JUnit results, coverage, store artifacts (builds, screenshots, videos) to S3; notify.

Parallelization strategy:

• Use job matrix: separate jobs for frontend-unit, backend-unit, integration, ui.
• Shard large suites: run Jest with --maxWorkers and --shard (or custom file-based sharding). Example matrix: frontend-unit shards 1..N.
• Run UI tests in parallel browser instances (Playwright parallelization).

Caching and artifact storage:

• Cache node_modules, build outputs, and Jest cache between runs (GitHub Actions cache keyed by package-lock hash).
• Persist test caches per-branch to speed incremental runs.
• Upload artifacts (built bundles, coverage reports, failing screenshots, video) to S3 or GitHub artifacts with lifecycle policy.

Flaky-test detection:

• Run each flaky-prone test with a small retry policy (e.g., 1 retry) but record retries in metadata.
• Maintain a failing-history DB: tests that fail intermittently flagged by trend analysis (daily job aggregates JUnit results).
• Automatically label tests as flaky after threshold (e.g., passed <70% in last 10 runs) and notify owners.
• For flagged flaky tests: require quarantine or rewrite before removing gating.

Gating rules for PR merges:

• Required checks: lint, unit shards pass, integration smoke tests pass, coverage thresholds enforced (e.g., no drop >2%), security scan.
• Full UI suite optional for fast merges but required on main branch or nightly; alternatively require a passing “UI smoke” subset on PRs.
• Enforce minimum approvals and passing CI via branch protection.

Techniques to reduce CI time while maintaining confidence:

• Test selection: run only affected unit tests in PR by mapping changed files to test files (use Jest --findRelatedTests).
• Split suites: run fast smoke tests on PR; schedule full regression (UI + full integration) on merge or nightly.
• Parallelize aggressively (shards + multiple runners).
• Use incremental builds and cache build artifacts to skip reinstall/build steps.
• Run expensive tests selectively: e.g., run UI full suite only when labels or size of change warrants, or on pushes to main.
• Use container snapshots or warm runners to avoid cold starts.

Example snippet (GitHub Actions matrix for Jest shards):

jobs:
  frontend-unit:
    strategy:
      matrix:
        shard: [1,2,3] # split tests into 3 shards
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
      - run: npm ci
      - run: npm run test:unit -- --shard=${{ matrix.shard }} --reporters=jest-junit
      - uses: actions/upload-artifact@v4
        with:
          name: junit-frontend-shard-${{ matrix.shard }}
          path: junit.xml

Trade-offs:

• Running full UI in PRs increases confidence but slows feedback; mitigate with smoke vs full split.
• Aggressive caching speeds builds but can hide dependency issues; use periodic clean runs.
• Automatic retries hide transient infra issues but can mask real flakiness—must pair retries with telemetry and quarantine.

Monitoring & maintenance:

• Dashboards for test pass rates, flaky test trends.
• Alert owners when a test becomes flaky or coverage drops.
• Periodic health job that runs full pipeline on main nightly.

This design balances fast PR feedback with rigorous full-suite verification, uses sharding and caching to reduce CI time, and adds mechanisms to detect and remediate flaky tests.