Assertions and Behavior Verification Questions

Mocks, stubs, fakes, and spies are types of test doubles used to isolate units under test. Short definitions, a tiny language-agnostic example, and when you’d prefer each:

1. Stub

• Definition: Provides canned responses to calls the unit under test depends on; not concerned with verifying interactions.
• Example: A PaymentGateway.stub.authorize(card) → returns {status: "approved"}.
• When to prefer: Use when the dependency is slow, non-deterministic, or hard to set up (third-party API) and you only need predictable return values.

2. Mock

• Definition: A double that you program with expected interactions (calls, order, arguments) and which fails the test if expectations aren’t met.
• Example: MockMailer.expect.send(email with subject "Welcome") → once.
• When to prefer: Use when behavior depends on specific interactions (e.g., ensuring an email is sent after signup).

3. Fake

• Definition: A working implementation with simplified or in-memory behavior (lighter-weight real implementation).
• Example: InMemoryUserRepository implements UserRepository storing users in a map.
• When to prefer: Use when the real implementation is heavy (database) but you want realistic behavior (queries, persistence semantics) in unit tests.

4. Spy

• Definition: A wrapper around a real object that records information about calls (can also let calls pass through).
• Example: SpyLogger wraps Logger and records call counts and arguments while still logging.
• When to prefer: Use when you want to assert that certain calls occurred but also exercise the dependency’s real behavior.

Key trade-offs: stubs/fakes improve determinism and speed; mocks/spies verify interactions and can make tests brittle if over-specified. Choose the double that balances test clarity, speed, and confidence.

Start with goals: ensure query-building never concatenates untrusted input, ORM/DB layers always use parameterization, and sanitizers/fallbacks don't give false sense of safety. Combine unit, integration, fuzzing, and static analysis into CI gating.

1. Unit tests for query-building logic

• Test that all helper functions return parameterized templates, not concatenated strings.
• Example (Python pytest with a query-builder):

def test_build_user_query_uses_placeholders():
    q, params = build_user_query(name="alice", age=30)
    assert "%s" in q or ":" in q  # template uses placeholders
    assert params == {"name":"alice","age":30}

Explain: focus on small pure functions; assert both SQL template shape and parameter values.

2. Integration tests verifying ORM parameterization

• Run tests against a test DB (in-memory or ephemeral container) and capture executed SQL (e.g., SQLAlchemy echo or DB proxy) to assert no literals of untrusted input appear.

# pseudo: capture last_executed_sql from SQLAlchemy event
assert "alice" not in last_executed_sql  # name should be bound param, not literal

Explain: ensures framework binding works end-to-end.

3. Fuzzing approaches

• Target layers that parse/format queries and any sanitizers. Use generation-based fuzzers with SQL-intent corpora plus mutation (sqlmap payloads): boofuzz, python-requests + AFL-style mutator, or Hypothesis for property-based tests:

from hypothesis import given, strategies as st
@given(st.text(min_size=1, max_size=200))
def test_query_builder_handles_arbitrary_input(s):
    q, params = build_user_query(name=s)
    execute_on_test_db(q, params)  # must not change schema or run injected SQL

Explain: monitor for errors, unexpected side-effects, or schema changes.

4. Static analysis & CI

• Use Semgrep rules to detect string concatenation into query APIs, patterns like + with SQL keywords, or use SQLi-specific rules in SonarQube, CodeQL, Bandit. Create custom rules for your language/framework (e.g., Java PreparedStatement detection).
• Fail CI on findings or require triage.

5. Edge test cases to bypass naive sanitizers

• Inputs with encoded/Unicode chars, comment injection ("name'; --"), stacked queries ("; DROP TABLE..."), multi-byte encodings, long inputs, null bytes, percent-encoded values, parameter marker confusion ("%s vs ?"), and inputs that look sanitized but include payload after benign token.
• Example cases list:
  • "Robert'); DROP TABLE users;--"
  • "admin' OR '1'='1"
  • "name%0A;UPDATE users SET role='admin'"
  • Unicode homoglyphs, NUL bytes

6. Monitoring & runtime checks

• Add DB-side alerts for DDL executed from app accounts in prod, query plan anomalies, abnormal frequency of statements.
• Use read-only credentials for app services where possible.

Trade-offs and rationale:

• Unit tests catch logic bugs early; integration tests catch framework misuse; fuzzing finds unexpected parser/encoding gaps; static analysis scales across repo. Combined they minimize false confidence from naïve sanitizers. Automate all in CI and periodically run extended fuzzing offline.

Requirements and constraints:

• Reduce CI time for full suite (hours) and PR feedback latency (target < 10–30 min).
• Preserve quality (low flakiness, high coverage for changed code).
• Cost-aware (compute/capacity limits), incremental rollout.

High-level design:

• Change-based test selection + dependency-aware impact analysis → determine minimal test set per commit/PR.
• Parallel test execution across a worker pool with sharding and dynamic scheduling.
• Cache test results/artifacts (build outputs, test binaries, docker layers).
• Risk-split test tiers: Fast unit/smoke for PRs, expanded integration/regression on main branch or nightly.
• Incremental runs for PRs: quick pre-submit, fuller run on merge or periodic.

Core components:

1. Change Analyzer

• Inputs: git diff, language/module dependency graph, test-to-source mapping (via static analysis + historical coverage).
• Output: impacted modules and candidate tests.

2. Test Impact Service

• Maintains test dependency graph, historical trace (which tests touched which files), and flakiness scores.
• Ranks tests by likelihood-of-failure and runtime; selects required tests using thresholds.

3. Scheduler & Sharder

• Assigns tests to workers, uses runtime-aware bin-packing and speculative execution for long tests.

4. Cache Layer

• Artifact cache keyed by commit hash + environment; stores build artifacts and test results (for deterministic tests).

5. Risk & Policy Engine

• Implements rules: PR quick-pass (units + smoke), nightly full suite, gate on main requires full or high-risk subset.

6. Observability

• Collects test durations, failure rates, cache hit rate, selection recall (did selection miss failures?) and cost metrics.

Selection strategies:

• Conservative mode: include all tests touching changed files + their transitive dependents.
• Probabilistic/prioritized: include top-K tests by historical failure probability and coverage until cumulative recall threshold reached.
• Fallback: if change impacts core libs, run expanded set.

Parallelization:

• Use cloud/on-prem worker autoscaling, run short tests first (shortest-job-first), shard long-running tests across workers, support test-level and suite-level parallelism.
• Dynamic retry and selective re-run of flaky tests on separate workers.

Caching:

• Build artifacts cached per commit or per dependency version; test outputs cached only for idempotent tests and used for downstream dependent steps.
• Validate cache safety by hashing inputs (source, deps, env). Invalidate on env changes.

Risk splitting & incremental runs:

• Define tiers: Tier 0 (unit, lint, fast smoke) required on PR; Tier 1 (integration) on merge/CI pipeline; Tier 2 (e2e, full regression) nightly or on release candidate.
• Optionally run targeted Tier1 tests in PR for high-risk files.

Trade-offs:

• Precision vs safety: aggressive selection reduces time but risks missing regressions. Mitigate with conservative thresholds, periodic full-suite runs, and monitoring selection recall.
• Complexity vs ROI: building dependency analysis and caching increases engineering cost; prioritize quick wins (parallelization, test sharding, caching) first.
• Flakiness: caching and parallel runs may hide flaky behavior; invest in flakiness detection and quarantine unstable tests.
• Cost: more parallel workers lowers latency but increases CI bill; autoscale and spot instances to balance.

Rollout strategy:

1. Measure baseline metrics (full runtime, per-test durations, failure history).
2. Quick wins (weeks): enable parallel workers, test sharding, and basic artifact caching.
3. Instrumentation (simultaneous): collect coverage and per-test source mapping.
4. Phase in change-based selection in shadow mode (select tests but still run full suite) for 2–4 weeks and track missed failures.
5. Enable selection for PRs with conservative thresholds; run full suite on main/merge.
6. Gradually lower conservatism as recall/precision improve; add probabilistic prioritization for long-running suites.
7. Enforce flaky-test remediation plan and continuous monitoring; iterate policies monthly.

Monitoring & KPIs:

• PR feedback time, full-suite runtime, selection recall (missed regressions), cache hit rate, CI cost/hour, flaky test count.

This plan balances fast feedback for developers with safety and cost control via staged rollout, measurable telemetry, and iterative relaxation of selection conservatism.

Situation: An intermittent CI test failure that never reproduces locally can come from flakiness, infra instability, race conditions, or timing/timeout sensitivity. Here's a systematic debugging plan to identify and fix the root cause.

1. Triage & pattern analysis

• Collect CI failure history: frequency, time-of-day, node/agent IDs, commit hashes, PR vs master, test shard.
• Capture failing job artifacts (logs, test output, screenshots, core dumps, coverage).
• Note whether failures correlate with load spikes, high CPU/memory, or network issues.

2. Telemetry to enable/collect

• Test-level: full stdout/stderr, stack traces, assertion locations, test durations, timestamps with millisecond precision.
• Infra: agent CPU, memory, disk I/O, network latency, process counts, container cgroup metrics.
• Tracing/profiling: distributed traces (span timings), thread dumps, GC logs, perf or pprof snapshots for native code.
• Environment: exact binary, dependency versions, environment vars, JVM/Node flags.
• Persist a reproducible VM/container image of the failing CI node if possible.

3. Reproduce locally under load

• Re-run test in same OS/container image (use the CI image locally or in Docker).
• Mimic CI resources: limit CPU and memory, run with similar concurrency, use stress tools (stress-ng), network throttling (tc/netem).
• Run tests in loop and parallel to increase chance of exposing race/timing.
• Use CI-scale test runner (same test shard/scheduler).
• Increase log verbosity and enable timestamps.

4. Narrow down cause

• Flaky test (test code): add deterministic seeds, assert pre/post conditions, isolate external dependencies by mocking, add extra logging around setup/teardown.
• Infrastructure instability: look for correlated infra metrics (OOM, disk full, flaky network). Re-run on different agents and compare.
• Race condition: run under race detectors/sanitizers (ThreadSanitizer, Go race, Java concurrency tools), enable heavy instrumentation (LockProfilers), capture thread dumps at failure.
• Timing-sensitive: replace real timers with controllable clocks (fake clocks), reduce test timeouts locally, introduce artificial delays to see sensitivity.

5. Isolation strategy (binary search)

• Bisect commits to see when flakiness began.
• Minimize test: remove unrelated assertions, split test into smaller units to identify the failing section.
• Reproduce single-case under single-threaded mode to check concurrency dependency.
• Swap real dependencies with deterministic fakes to see if external systems cause nondeterminism.

6. Fix approaches

• For race: add proper synchronization, atomic operations, or redesign to avoid shared mutable state in tests.
• For timing: increase conservative timeouts, use condition-based waits (poll with timeout) instead of fixed sleeps, use fake clocks.
• For infra: add retries with exponential backoff for known transient failures, pin CI workers or improve resource isolation, or tweak resource requests/limits.
• For flaky test logic: rewrite to be deterministic, ensure clean setup/teardown, avoid shared state between tests, and add idempotency.

7. Validation & prevention

• Re-run tens/hundreds of attempts in CI and locally under stress to verify fix.
• Add telemetry and assertion checks to ensure invariant holds going forward.
• Add test-level flakiness gating (failing only after N consecutive failures) and alerting for infra anomalies.
• Add post-merge monitoring and a blame-minimizing bisect process for future regressions.

Example quick commands:

• Run with limited CPUs: docker run --cpus="0.5" ...
• Network jitter: tc qdisc add dev eth0 root netem delay 100ms 20ms
• Java thread dumps: jstack -l <pid> > dump.txt
• Go race: go test -race ./...

This plan identifies, reproduces, and isolates the root cause through telemetry, controlled experiments, and targeted fixes, then validates the fix under load before rolling back into CI.

Approach: cover memory safety and UB at multiple levels — unit tests compiled with AddressSanitizer (ASan) and UndefinedBehaviorSanitizer (UBSan), fuzz public APIs with libFuzzer/afl, and enforce sanitizer runs in CI on PRs and nightly builds. Use targeted unit tests that intentionally trigger UB to verify sanitizers are active.

Example unit test (out-of-bounds + signed overflow) using GoogleTest and compiled with ASan/UBSan/libFuzzer where appropriate:

// test/ub_sanity_test.cpp
#include <gtest/gtest.h>
#include <cstdint>

// Simple function under test
int32_t add_saturating(int32_t a, int32_t b) {
    // intentionally incorrect: plain add can overflow
    return a + b;
}

TEST(UBSan_ASan, SignedOverflow) {
    int32_t a = INT32_MAX;
    int32_t b = 1;
    // This triggers signed overflow UB; UBSan should report it.
    int32_t r = add_saturating(a, b);
    (void)r;
}

TEST(ASan, OutOfBoundsAccess) {
    int arr[3] = {1,2,3};
    // OOB read triggers ASan
    volatile int x = arr[5];
    (void)x;
}

Build flags (example):

• g++ -fsanitize=address,undefined,address-use-after-scope -fno-omit-frame-pointer -g

Fuzzing harness (libFuzzer) for public API:

// fuzz/harness.cpp
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    // parse input and call public API to exercise edge cases
    MyPublicAPI::Process(data, size);
    return 0;
}

CI integration:

• Run sanitizers on PRs with a fast subset of tests (ASan+UBSan).
• Nightly/buildkite job that runs full sanitizer suite and long fuzzing jobs.
• Example GitHub Actions step:
  • uses: actions/setup-node@vX
  • run: cmake -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g" && ninja test

Why: ASan finds heap/stack/Global OOBs and use-after-free; UBSan catches signed overflow, misaligned accesses, shift UB, etc. Fuzzing exercises many input paths and finds complex memory/logic bugs. Include repros from failing builds and require fixes before merge.