Security Testing Questions

Start from principles: test behavior and security contracts, not internal code paths. Aim for layered tests that exercise inputs at different boundaries and execution contexts.

Test types & levels

• Unit tests: validate input validation logic, canonicalization, and explicit rejection rules. Use table-driven tests for edge inputs (nulls, very long strings, unicode, control chars). Assert on public API errors/messages, not internal function calls.
• Integration tests: exercise whole stacks (web layer → auth → business logic → DB). Verify auth/authorization decisions, parameterized endpoints, role-based access, and that sanitized inputs never reach sinks. Use test harnesses that mimic real middleware.
• Fuzzing / property-based: run randomized and grammar-based fuzzers against parsers, query builders, and serialization code to find injection and crash scenarios. Use property-based tests (Hypothesis/QuickCheck) to assert invariants (e.g., no DB modifications for read-only endpoints).
• Security-specific: SAST/DAST, dependency scanning, and dedicated penetration tests for auth bypass, CSRF, SQL/OS/LDAP injection and deserialization flaws.
• Load/resource tests: supply large or expensive inputs and simulate concurrent requests to validate rate-limits, quotas, and graceful degradation.

Examples (concise)

• Injection: send payloads containing quotes, encoded sequences, nested JSON; assert requests are rejected or parameterized queries executed (no change in DB state).
• Auth bypass: try accessing endpoints with different token shapes (expired, signed-with-other-key), missing claims; expect 401/403 consistently.
• Excessive resource: upload oversized files, deeply nested JSON, or exponential regex payloads; assert 4xx or throttling, not OOM.

Maintain resilience as implementation evolves

• Test behavior/contracts: assert on outputs, HTTP status codes, or side-effect invariants (DB rows created/deleted), not on internal logs or exception types.
• Use stable fixtures and builders: create inputs through factories rather than hardcoding internal object shapes.
• Mock only external dependencies (third-party services) at integration boundaries; run a smaller set of end-to-end tests against realistic environments.
• Parameterize expected tolerances (latency, size limits) and centralize security policy in configuration so tests reference policies, not code.
• Automate fuzzing and security scans in CI, run heavier suites nightly, and triage failures quickly to avoid bit rot.
• Keep tests small and focused; mark flaky or implementation-dependent tests as such and refactor when they block CI.

This layered, contract-first approach catches injection, auth bypasses, and resource abuse while keeping tests robust to internal refactors.

Requirements & goals:

• Find memory/UB bugs in native PNG parsing code with high signal-to-noise, fast triage, and automated daily regression checks.
• Target: crashes (ASan), UB (UBSan), leaks (LSan), undefined reads, and behavior mismatches.

Strategy overview:

1. Corpus generation

• Start with a curated seed corpus: real PNGs (tiny, valid, malformed samples), plus edge-case images (max chunks, empty IDAT, uncommon color types).
• Augment with synthetically generated samples using a grammar-based generator (Grammar-based tools or grammarinator) describing PNG chunk structure (IHDR, PLTE, IDAT, IEND, ancillary chunks).
• Keep metadata (source, generation method) and label valid vs invalid.

2. Mutation vs generation

• Use generation-based fuzzers for structural coverage (produce valid-but-unusual PNG structures).
• Use mutation-based, coverage-guided fuzzers (libFuzzer, AFL++, honggfuzz) to explore subtle parser states starting from seed corpus.
• Combine both: periodically regenerate seeds from generator and feed into mutation fuzzer.

3. Sanitizers & build

• Build instrumented binaries with AddressSanitizer (ASan) + UBSan (+ undefined behavior ASan flags), and optionally LeakSanitizer/MemorySanitizer where feasible (MSAN requires fully instrumented toolchain).
• Use -fsanitize=address,undefined,integer,bounds; compile with -g -O1 or -O2 (libFuzzer prefers -O1/-g).
• Enable ASan options: detect_stack_use_after_return=1, allocator_may_return_null=1 for robustness.

4. Minimization / shrinking

• Use built-in fuzzer minimizers (libFuzzer -minimize_crash=1) and afl-cmin/afl-tmin to shrink testcases.
• Keep minimized testcase + original in corpus. Record sanitizer stack trace and fuzzer coverage map for dedup.

5. Crash triage

• Automated triage pipeline:
  • Deduplicate by sanitizer signature + top stack frames (libFuzzer / afl unique crash dedupe).
  • Reproduce deterministically (fixed seed, deterministic build) under same sanitizer.
  • Classify: ASan heap overflow, use-after-free, UBSan integer overflow, parser panic, etc.
  • Prioritize by exploitability and frequency.
  • Attach minimized testcase, sanitizer log, repro instructions, and git blame to ticket.
• For complex crashes, run Valgrind/Dr. Memory or run under AddressSanitizer with symbolized backtrace. Use gdb to get core if needed.

6. CI integration & daily runs

• Add a scheduled CI job (daily) that:
  • Runs multiple short fuzzing sessions (e.g., 4-8 instances x 1 hour) across sanitizers and seeds.
  • Uses coverage-guided run for a longer nightly job (e.g., 8-24h) if resources allow.
  • Stores artifacts: crashes (minimized), coverage reports (llvm-cov), and corpus deltas to object storage (S3).
  • Auto-open issues for new high-priority unique crashes with logs and repro steps.
  • Fail builds only on reproducible, high-severity crashes; otherwise mark warnings.
• Maintain a regression test suite: every confirmed bug adds a non-flaky testcase into the corpus/regression tests and is run in CI.

Best practices & trade-offs:

• Balance sanitizer overhead vs speed: run fast mutation fuzzing with ASan in CI; run heavier MSAN/UBSan on periodic deeper runs.
• Keep deterministic seeds and build flags for reproducibility.
• Monitor coverage and corpus growth; prune redundant files using coverage-guided corpus minimization.
• Consider OSS-Fuzz for continuous large-scale fuzzing if open-source.

This approach gives structured discovery, automated triage, and continuous protection against regressions.

Approach summary:

• Use known-good test vectors (RFCs, NIST ACVP) as ground truth.
• Make every test deterministic: inject non-randomness (nonce/IV, RNG) via dependency injection or test hooks.
• Cover correctness, invalid inputs, boundary key sizes, and parsing robustness with both unit tests and fuzzing.

Unit-test plan (structure + examples):

1. Standard vectors
   • Feed input/key/nonce from RFC into primitive; assert ciphertext/plaintext/mac matches expected.
2. Deterministic randomness
   • Replace RNG with a stub that returns fixed bytes.
   • Example (pytest-style for AES-GCM wrapper):

def test_aes_gcm_vector():
    key = bytes.fromhex("feffe...")  # example vector
    iv = bytes.fromhex("cafebabefacedbaddecaf888")
    rng_stub = lambda n: iv  # deterministic
    cipher = AESGCMWrapper(key, rng=rng_stub)
    ct, tag = cipher.encrypt(b"plaintext", aad=b"")
    assert ct.hex() == "expectedcthex"
    assert tag.hex() == "expectedtaghex"

3. Invalid inputs
   • Bad key sizes (e.g., 7, 15, 33 bytes) -> expect specific exceptions.
   • Truncated ciphertexts, wrong tag -> decryption failure, no undefined behavior.
   • Invalid padding -> controlled exception, no data leakage.
4. Boundary tests
   • Min/max key sizes supported (e.g., 128/256 bits), zero-length plaintext, extremely large inputs (streaming path).
5. Property tests
   • Encrypt-then-decrypt round trips with deterministic nonces.
   • Idempotence where applicable (e.g., MAC verification deterministic).

Fuzzing for parsing layers:

• Goal: find crashes, memory errors, panics, and logic errors in parsing.
• Seed corpus: include all canonical test vectors and malformed variants (truncated, extra bytes, bad lengths).
• Tools: libFuzzer/AFL for C/C++; python-afl or python's fuzzing tools for python; honggfuzz. Use coverage-guided fuzzing.
• Strategies:
  • Grammar-aware fuzzing: if formats are structured (ASN.1, TLS records), use a grammar model (Peach/Fuzzilli-like) to mutate structure-aware inputs.
  • Mutation + generation: combine random mutations with generator that produces boundary-lengths, off-by-one lengths, invalid tags.
  • Assertions: run instrumented builds (ASAN/UBSAN, MIRI for Rust) and check no memory corruption, no undefined behavior, and that all errors map to documented exceptions.
  • Expect deterministic results: run fuzzing harness with deterministic seeds (fixed seed for reproducibility); save minimal crash corpus.
• Harness design:
  • Isolate parsing layer from crypto ops: feed parsed structures to a mock crypto implementation to avoid expensive cryptographic computation and keep tests deterministic.
  • For stateful protocols, drive FSM with deterministic event sequences.

CI and reproducibility:

• Run unit tests deterministically in CI.
• Periodically run fuzzers on CI nightlies with saved corpora; triage new crashes.
• Store seeds and failing inputs in repo/bug tracker for reproducibility.

Why this works:

• Standard vectors verify correctness; injected deterministic randomness removes flaky tests; exhaustive invalid/boundary tests plus coverage-guided fuzzing exercise parsing and error paths to catch security-critical bugs early.

Requirements & goals:

• Prevent secrets (API keys, tokens, passwords, PII) from appearing in error messages, logs, alerts, telemetry, crash dumps.
• Detect at dev-time and runtime, block exfiltration, provide remediation workflow.

Testing checklist (manual + automated):

• Code review: ensure logger calls never include secrets; use placeholders (userId=<id>) instead of raw objects.
• Static analysis: run secret-scanning on repo and diffs (pre-commit / CI).
• Unit/integration tests: assert error messages sanitize sensitive fields.
• Fuzz / fault-injection: trigger errors and capture logs to inspect output.
• Runtime monitoring: sample logs, alerts, telemetry for detected secrets.
• Compliance checks: retain-only required fields; PII minimization.

Automated checks (CI + runtime) — regex examples:

• Generic API key patterns:

(?i)(api[_-]?key|access[_-]?token|auth[_-]?token)\s*[:=]\s*(['"]?)[A-Za-z0-9_\-\.=]{16,}\1

• AWS secret / access keys:

(AWS|aws)?_?(ACCESS_KEY|SECRET_KEY|ACCESSKEY|SECRETKEY)|AKIA[0-9A-Z]{16}

• JWTs / long base64 tokens:

eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}

• Credit card (PCI) basic detection:

\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13})\b

• SSN (US):

\b\d{3}-\d{2}-\d{4}\b

Implementations:

• CI pre-merge job: run regex scanner on diffs; fail build on matches (with exception process).
• Log-scrubber middleware: pipeline that masks values matching regexes before emission.
• Runtime detector: sampling agent that scans logs/alerts and raises high-priority incident if secret found.

Remediation process:

1. Triage: automated alert opens ticket with sample, source file, commit/trace.
2. Immediate mitigation: rotate exposed secret, revoke tokens, or disable leaking feature.
3. Fix: update code to remove secret from messages, add masking in logger, add unit test.
4. Post-mortem: root cause, timeline, mitigation verification, follow-up policy changes.
5. Audit: re-scan history and dependent systems for reuse of leaked keys.

Validating third-party libraries:

• Inventory third-party components and their logging surface (SCA tool).
• Run integration tests that exercise library error conditions and capture logs; scan captured output with CI regexes.
• Use library wrappers: intercept library logs (loggers, handlers) and apply scrubbing before sink.
• For closed-source: run sandboxed fuzzing and error injection (simulate bad inputs, network failures) and scan outputs.
• Require vendor attestation/security policy for logging and secret handling in procurement.

Key trade-offs and best practices:

• Balance false positives by tuning regexes and allow-listing non-secrets (e.g., short tokens used internally).
• Prefer structured logging (JSON) so scrubbing/masking targets fields, not free text.
• Rotate keys proactively; treat any detected secret as compromised.
• Make scanner results actionable: link to code location, provide remediation template.

This approach combines prevention (dev rules, structured logging), detection (CI + runtime scanning), and incident response to ensure secrets never leak in errors, logs, or alerts.

Fuzz testing complements unit and property-based testing by exploring the input space in ways those approaches don’t. Unit tests assert expected behavior for known inputs; property-based tests generate inputs constrained by properties and check invariants. Fuzzers, especially coverage-guided ones, mutate real inputs to discover unexpected edge-cases, parser state corruption, memory bugs, and security issues that are hard to encode as properties or examples.

Concrete target: a JSON parser in C/C++ used by a backend service. Unit tests validate valid/invalid JSON examples; property tests might assert that parsing(serializing(x)) == x for generated trees. A coverage-guided fuzzer (libFuzzer or AFL++) will mutate seeds (real JSON payloads) and likely find crashes from deeply nested structures, integer overflows, invalid UTF sequences, or logic that assumes well-formed input.

Integrating into CI:

• Add a nightly or per-merge job that runs the fuzzer in short-time mode (e.g., 10–30 minutes) with sanitizers enabled (ASan/UBSan/LSan) and coverage reporting.
• For critical releases, run longer fuzzing (hours) and/or use OSS-Fuzz for continuous cloud fuzzing.
• Store artifacts (crash inputs, coverage) in artifact storage and link runs to the PR/build.

Triage crashes:

1. Reproduce deterministically using minimized seed and same sanitizer flags.
2. Run minimizer (e.g., llvm-cov/afl-cmin, libFuzzer's -minimize_crash) to get a small test case.
3. Collect stack traces, sanitizer reports, and source location; run under debug build to get line-level trace.
4. Run ASan/UBsan to classify: memory safety, use-after-free, integer overflow, assertion failure, etc.
5. Prioritize by exploitability (memory corruption > crash-only), frequency, and component criticality. Create reproducible bug with minimized input and tests.

Managing corpus/seed growth:

• Start with a small curated seed corpus of real inputs; use coverage-guided corpus culling (libFuzzer’s corpus merge/minimization) to remove redundant inputs.
• Periodically run "corpus minimization" and "coverage trimming" to keep CI runtime bounded.
• Store canonical corpus per target in repo or artifact storage; accept new seeds from fuzzer if they increase coverage or trigger new behavior.
• Archive and label seeds that triggered high-priority bugs for regression tests.
• Use deterministic seeds and CI thresholds (e.g., run until coverage plateau or fixed time) to keep CI stable.

Why this works: fuzzing finds unanticipated interactions and malformed inputs that are expensive to anticipate in unit/property specs. Together they form a layered test strategy: unit tests document expected behavior, property tests assert invariants across broad input classes, and fuzzers probe the unknown boundaries and security-sensitive failures.