Requirements & goals:
- Find memory/UB errors and protocol logic crashes in a C network service that consumes structured messages.
- High signal (low false positives), low flakiness, reasonable CPU cost, CI-safe, automated triage.
Harness architecture:
- In-process harness: link the service library into a small harness binary exposing a deterministic entry point that consumes a byte buffer and drives parser + handlers. Advantages: fast, deterministic, easy to attach sanitizers.
- Out-of-process harness (secondary): a lightweight network proxy that accepts fuzzed packets and forwards them to a containerized service instance for end-to-end checks (covers networking code, handshake state).
- Use a corpus-format shim that translates fuzzer byte buffers to structured messages (field encoding helpers) so fuzzers explore both raw bytes and valid/near-valid structured messages.
Seed corpus management:
- Start with: real traffic anonymized and sanitized, protocol-valid examples, and hand-crafted edge cases (max-length, empty fields, boundary values).
- Maintain corpus on object storage (S3/GCS). Use periodic corpus minimization (afl-cmin/llvm-cov-based minimizer) to remove duplicates and keep seeds small.
- Version-seed sets per-release and tag corpora with coverage metrics. Feed new unique crashes back into corpus after triage.
Sanitizers and runtime config:
- Build harness with ASAN+UBSAN+MSAN builds where appropriate. Prefer ASAN+UBSAN for memory/undefined behavior; MSAN only for uninitialized memory testing (higher overhead).
- Use ASAN options: symbolize, set allocator options (fast_unwind_on_malloc=0), ASAN_OPTIONS=detect_leaks=0,halt_on_error=1 for CI runs. For target fuzzing runs long-term, enable leak detection during offline jobs.
- Enable coverage instrumentation via -fsanitize=fuzzer, or use libFuzzer/clang's sanitizers with AFL++/honggfuzz as appropriate.
CI integration strategy:
- Fast, low-CPU smoke-fuzz stage in PRs: run short libFuzzer (e.g., 30s-2min) with small corpus and ASAN; fail PR only on deterministic crashes (triaged automatically).
- Nightly/weekly longer fuzzing on autoscaled runners (K8s/Jenkins): scale CPU via spot/preemptible instances; maintain multiple fuzzers in parallel targeting different harnesses.
- Protect main CI: run fuzzers in isolated containers with resource limits, timeouts, and artifact upload on crash.
Crash triage automation:
- On crash, automated pipeline:
- Reproduce: rerun failing input 5× on both in-process and out-of-process harnesses under ASAN/UBSAN to confirm deterministic.
- Minimize corpus input (llvm-cov-based) to smallest repro.
- Run sanitizer stack traces + symbolization to produce human-readable report.
- Attach ASAN/UBSAN report, sanitized input, coverage diff, and repro steps to an auto-created issue/bug ticket with priority scoring (based on sanitizer type, impact, hit counts).
- If non-deterministic, run stress-repro (hundreds of runs), collect log, save VM snapshot.
Reducing false positives & flakiness:
- Determinism: prefer in-process harnesses for determinism; if network needed, mock nondeterministic external inputs (time, RNG, sockets).
- Isolate global state: reset global/static state between fuzz iterations; use fork-mode fuzzing (libFuzzer’s persistent mode) carefully.
- Flakiness handling: require N reproducible repeats before creating high-priority bug. Use differential builds (with/without sanitizer) to filter sanitizer-only noise.
- Use sanitizers' suppressions file for known third-party false positives; maintain suppression review process.
Cost control:
- Prioritize coverage-guided fuzzers on critical code paths; run lightweight PR fuzzers (low time per PR).
- Use autoscaling worker pools with spot instances for high-volume long fuzz runs; cap per-fuzzer CPU.
- Pool corpus and use smart seeding to avoid wasted work.
- Use hybrid strategy: frequent short runs (low cost) + fewer long-running dedicated fuzzers for deep discovery.
Metrics & monitoring:
- Track unique crashes/day, unique coverage edges, flake rate, repro rate, time-to-triage, CPU-hours per valid crash.
- Alert if crash rate spikes or flakiness rises.
Trade-offs:
- In-process is faster/deterministic but may miss network stack bugs; out-of-process covers full system with higher cost and flakiness.
- MSAN adds cost; enable selectively.
This pipeline gives fast feedback in CI, deep continuous fuzzing overnight, automated triage to keep developer load low, and measures to reduce false positives and CPU cost.
