Situation: We were rolling out a generative-AI microservice that authors customer-facing summaries; leadership asked for a measurable reliability program so product and SRE could trust continuous delivery.
Concrete, role-appropriate measures I’d put in place:
Static checks (pre-commit / CI)
- Code quality: eslint/flake8 + mypy/type checks, cyclomatic complexity thresholds, 80%+ unit test coverage enforced.
- Model artifacts: model-card linting (metadata, provenance, training data stats), checksum signing and schema validation for model bundle.
- Security: SCA (dependency vuln scan), secret scanning, and a minimal adversarial prompt test suite to catch obvious injection vectors.
Runtime checks (observability / production)
- SLIs/SLOs: latency p95 < 300ms, success rate > 99.5, hallucination-rate < X% (see below), throughput targets.
- Health probes: readiness/liveness, GPU memory & OOM alerts, per-replica queue depth.
- Telemetry: request/response size, token usage, top-k prompt patterns, model confidence/entropy distributions.
Contract testing & deployment strategies
- API contract tests: JSON schema, type checks, and behavioral contract tests using prerecorded prompts → expected structure and safety tags.
- Model contract: golden prompt set (semantics + safety), with allowed variance bounds (BLEU/ROUGE/embedding cosine thresholds).
- Consumer-driven contract tests: integrations run in CI using mock/stub clients to verify backward compatibility.
- Canary & shadowing: route 1–5% of traffic to new model with mirrored inputs; compare outputs with divergence metrics before full rollout.
Chaos & fault-injection strategies
- Inject latency and packet loss in network paths (tc/netem) to validate retries and circuit breakers.
- Simulate GPU OOM / node termination during inference to test autoscaling, request draining, and graceful degradation (fallback model or cached responses).
- Token-limit / truncation tests to see partial-generation handling and user-facing recovery.
- Introduce corrupted model artifact and verify signature validation rejects it.
- Adversarial prompt bombardment during load tests to measure safety filter saturation.
Incident & reliability metrics (operational + quality)
- MTTD (mean time to detect), MTTR (mean time to recover), and change-failure-rate.
- Error budget and burn rate for SLO-driven releases.
- Hallucination rate: percentage of outputs failing human / automated factuality checks (e.g., fact-checker model or retrieval verification).
- Safety/abuse rate: percent of requests flagged by safety filters and effective false-positive/negative rates for filters.
- Regression delta metrics: embedding-distance drift, ROUGE/BLEU shifts on golden set, and performance per model version.
- Business metrics: user satisfaction (NPS/CSAT for generated content), downstream task success (e.g., edit rate, conversion).
Why this combination:
- Static checks prevent regressions early; contract tests and canarying protect consumers from breaking changes; runtime telemetry and chaos exercises expose brittle failure modes; and incident metrics tie engineering health to business impact and SLOs. Together they give a defensible, measurable reliability posture for a generative-AI microservice.
