Testability and Shift-Left Testing Questions

Organizing tests in an SRE-maintained monorepo should make intent clear, enable CI selection, speed triage, and surface flaky/slow tests.

Structure:

• src/
  • service-a/
    • pkg/
    • tests/
      • unit/
      • integration/
      • smoke/
• infra/
  • terraform/
  • tests/
    • unit/
    • integration/
    • smoke/

Principles and naming conventions:

• Directory-level separation: unit (fast, pure), integration (external dependencies, slower), smoke (critical end-to-end sanity).
• File/name prefixes for CI selection and triage:
  • unit tests: test_unit_<module>_*.py or <module>.unit_test.go
  • integration: test_int_<service>_*.py or <service>.int_test.go
  • smoke: smoke_<service>_*.py
• Add metadata tags in test code (pytest markers, JUnit properties):
  • @pytest.mark.integration
  • @pytest.mark.flaky(retries=2)
  • @pytest.mark.slow

CI usage:

• CI job filters by path and tag: fast pipeline runs only unit & smoke for PRs; nightly runs integration and full e2e.
• Use filename patterns and test framework tags to parallelize and route to specific runners (e.g., integration on infra lab).

Surfacing flaky/slow tests:

• Emit timing and retry metadata to test reports (JUnit XML).
• Fail CI on new flake regressions; aggregate historical runtimes and mark tests > p95 as slow.
• Automatically create issues for tests with >N flake incidents and annotate owners via test metadata.

Triage tips:

• Ensure each test file has a clear owner tag and SLO-impact flag (e.g., @owner("team-a") @slo("P99-latency")).
• Make test names descriptive: smoke_auth_login_should_return_200 — immediate signal of intent and failure impact.

This convention helps CI pick the minimal fast surface for PRs, schedules expensive tests appropriately, and makes diagnosis/ownership immediate when tests fail.

1. Triage / immediate actions (first 30–60 minutes)

• Reproduce locally and in CI: run the failing job with same image, env, network mode to confirm intermittent 'connection refused'.
• Quarantine test: mark flaky in test runner / move to flaky suite to stop pipeline noise while investigating.
• Increase CI job retries (temporary) or add short backoff+retry in test to reduce transient failures causing alerts.
• Capture artifacts: collect CI logs, container/network/DNS logs, timestamps, and full stack traces for failed runs.

2. Diagnosis checklist (next 1–4 hours)

• Correlate failures with CI host metrics: CPU, memory, disk, open ports, ephemeral port exhaustion, docker daemon restarts.
• Check service start order & readiness: ensure services under test are listening before client attempts connection (inspect health-check logs).
• Network/DNS investigation: validate service advertised host/port, DNS resolution, /etc/hosts, and container networking (bridge vs host).
• Resource contention: inspect parallel job contention, ephemeral port reuse, ulimit, and kernel logs (dmesg).
• Dependency flakiness: test if upstream service occasionally rejects connections; run dedicated uptime probes.
• Rate/time correlation: look for pattern (time of day, high concurrency, deployment windows).

3. Short-term fixes (hours–days)

• Add robust readiness/wait logic to test harness (wait-for with timeout + health-check polling).
• Use exponential backoff + limited retries in client attempts.
• Increase CI resources or serialize problematic jobs to avoid contention.
• Pin deterministic ports or use ephemeral port allocation helpers to avoid collisions.
• Improve logging and attach pcap/tcpdump selectively for deeper failures.

4. Long-term fixes (weeks)

• Replace brittle integration points with contract tests or service virtualization/mocks for CI runs; reserve full end-to-end tests to scheduled pipelines.
• Harden services: idempotent startup, graceful backoff, health endpoints, and readiness probes.
• CI infra improvements: autoscaling runners, network QoS, sandbox isolation (separate VPCs), and stronger resource quotas.
• Observability: add synthetic probes, add metrics for connection failures, failure budgets and alerting thresholds tied to SLOs.
• Test hygiene: implement flaky-test detection, track flakiness metrics, require owner-level remediation within SLAs.
• Postmortem & prevention: document root cause, update runbooks, and add regression tests that would catch the issue.

5. Verification

• Run multiple CI runs after fixes, monitor failure rate drop to near-zero, and remove quarantine only after sustained stability (e.g., 50+ runs).

Priority guidance: quarantine & retries immediately to stop noise; then focus on readiness/wait logic and resource contention; finally invest in mocks, CI scaling, and observability to prevent recurrence.

Situation: On a mobile team I joined, we had a lot of regressions in one feature area because the view models were doing too much and our tests were mostly high-level UI tests.

Task: I wanted to improve both code quality and test quality so bugs were caught earlier, without slowing delivery.

Action: I extracted network and persistence logic behind small protocols, moved parsing into a pure mapper layer, and added focused unit tests around the business rules. I also replaced several brittle UI assertions with a smaller number of end-to-end checks that verified the full flow. To keep the team aligned, I added test-review guidance in PRs: if logic could be tested without UIKit/Android UI, it should be.

Result: The feature became much easier to change, and we started catching bad payload handling and state bugs in unit tests instead of in QA or CI. I knew it helped because flaky UI failures went down, PR reviews got simpler, and the same area produced fewer escaped defects after the refactor.

The main lesson for me was that quality improves fastest when tests match the right layer of the architecture.

Situation: A production outage occurred when a deployment introduced a change requiring an authentication header. Tests passed because a mocked downstream service omitted the new header validation; therefore the integration broke in production.

Root cause analysis:

• Primary: Incomplete mock/stub used in tests did not validate the new auth header, so integration tests gave false positives.
• Secondary pipeline gaps: CI ran unit and mocked integration tests but lacked end-to-end (E2E) or contract verification against the real service contract. No gating that validated API contract changes.
• Ownership: No explicit owner for downstream contract changes; teams relied on implicit coordination.
• Observability: Alerts surfaced errors but too late — no pre-deploy contract mismatch detector.

Impact: 45 minutes downtime, failed requests increased by 300%, customer-facing errors and incident response costs.

Action plan (prioritized, owner, ETA, success metric):

1. Immediate (24–72h) — Rollback + hotfix; add temporary runtime fallback to accept missing header where safe. Owner: Oncall SRE. Metric: Recovery <1h for similar regressions.
2. High (1–2 weeks) — Adopt contract testing (Pact or similar) for all inter-service APIs; enforce CI gating: consumer-driven contract verification and provider verification in pipeline. Owner: Dev team + SRE. Metric: 100% of services with contract tests; zero contract mismatches in CI.
3. High (1–2 weeks) — Replace brittle mocks in integration tests with lightweight provider emulators that enforce header/validation rules or run tests against a staging instance of the real service. Owner: Dev teams. Metric: Integration test failures for contract violations >0 in CI.
4. Medium (2–4 weeks) — Add a CI job that runs schema/contract diffs and flags breaking changes; require explicit approval for breaking contract changes. Owner: Platform/CI team. Metric: All PRs with contract diffs require sign-off.
5. Medium (2–4 weeks) — Define and assign contract owners; include change process in API lifecycle docs. Owner: Engineering leadership. Metric: 100% contracts with an owner listed in repo/registry.
6. Medium (2–6 weeks) — Enhance observability: add synthetic canaries and pre-production smoke tests that validate auth flows. Owner: SRE. Metric: Canary failures detected pre-prod >0 and alerts created.
7. Low (4–8 weeks) — Run chaos tests that simulate header omission to validate system resilience. Owner: SRE/QA. Metric: Successful test runs and documented mitigations.
8. Ongoing — Train teams on contract-first development and add contract checks into PR templates. Owner: Eng Enablement. Metric: Training completion rate; reduced post-deploy contract incidents.

Prevention controls and trade-offs:

• Contract testing prevents class of regressions but increases CI time; mitigate via parallelization and selective gating.
• Provider emulators increase fidelity but require maintenance—start with high-risk APIs first.

Follow-up: 30-day review to validate metrics, enforce owners, and add the contract tests into the SLO definition for inter-service API reliability.

Fakes, stubs, and mocks are different test doubles with distinct purposes:

• Stub: A minimal implementation that returns canned data for a call (e.g., a function that returns predefined rows). Use when you just need predictable responses and don’t assert interactions.
• Mock: A test double that both simulates behavior and asserts how it was used (calls, order, parameters). Useful to verify that your code issues the correct queries or transaction sequence.
• Fake: A lightweight working implementation (often in-memory) that behaves like the real system but simplified (e.g., an in-memory key-value store or SQLite used as a fake DB). Fakes exercise real logic without full production complexity.

SRE trade-offs for DB replacement in tests:

• Prefer lightweight in-memory fake when:

  • Fast feedback and CI speed matter (unit tests, pre-commit hooks).
  • You need to validate application logic, schema-less operations, or simple query behavior.
  • Isolation from environment variability is required; lower flakiness and cost.
    Example: use SQLite in memory to test query formatting, caching logic, and basic transactions.

• Prefer isolated real DB instance for integration / e2e tests when:

  • You must validate production-specific features: SQL dialect differences, replication, stored procedures, indexing, connection pooling, auth, backups, or performance characteristics.
  • Ensuring operational behavior under concurrency, failover, and recovery scenarios.
    Example: spin up a containerized Postgres with the same config to test migrations, locks, and SLO-affecting interactions.

Why: fakes give speed and determinism; real instances give fidelity and uncover ops/compatibility issues. As an SRE, combine both: fast unit tests with fakes + a smaller suite of integration tests against isolated real DBs run in CI or nightly pipelines to catch environment-specific failure modes.