System Reliability and Availability Questions

Approach: use psql to run a simple SELECT 1, measure round-trip time in milliseconds, and fail if >200 ms. Keep credentials out of the script by reading from environment variables (CI secrets) or .pgpass.

#!/usr/bin/env bash
set -euo pipefail

# Required env vars: PGHOST, PGPORT, PGUSER, PGPASSWORD, PGDATABASE
TIMEOUT_SEC=${PG_TIMEOUT:-5}
LATENCY_THRESHOLD_MS=200

start_ms() { date +%s%3N; }
end_ms()   { date +%s%3N; }

start=$(start_ms)
# run psql with a short statement timeout; -t -A gives plain output
# use timeout to avoid hanging
if ! timeout "${TIMEOUT_SEC}s" psql "host=${PGHOST} port=${PGPORT:-5432} user=${PGUSER} dbname=${PGDATABASE}" \
    -c "SET statement_timeout = $((TIMEOUT_SEC*1000)); SELECT 1;" -t -A >/dev/null 2>&1; then
  echo "ERROR: psql failed or timed out"
  exit 2
fi
end=$(end_ms)

latency=$((end - start))
echo "Postgres SELECT 1 latency: ${latency} ms"

if (( latency > LATENCY_THRESHOLD_MS )); then
  echo "FAIL: latency ${latency}ms exceeds threshold ${LATENCY_THRESHOLD_MS}ms"
  exit 3
fi

echo "OK"
exit 0

Key points:

• Uses environment variables for credentials (do not hardcode).
• Uses timeout and statement_timeout to bound waits.
• Returns distinct nonzero codes for failure types.

Running as pre-merge CI check without exposing credentials:

• Store PGHOST/PGPORT/PGUSER/PGPASSWORD/PGDATABASE as encrypted CI secrets (GitHub Actions Secrets, GitLab CI Variables, CircleCI env vars).
• In the CI job, inject them into the environment; do not echo them or write to logs.
• Optionally use .pgpass file created at runtime with strict permissions (600) using the secret, then run psql without PGPASSWORD env var.
• Run this script as a lightweight pipeline job (fast runner), mark it required for merge; keep threshold conservative to avoid false positives.

Situation: Multiple teams deploying microservices together cause late integration failures that impact uptime.

Strategy (overall goal: catch cross-team issues earlier while keeping CI time reasonable):

1. Environments

• Developer sandbox: lightweight k8s namespace per branch using ephemeral infra (Tilt / Okteto) so devs can reproduce multi-service interactions locally.
• Integration (staging): production-like cluster with representative config and synthetic traffic for nightly/CI gated tests.
• Canary/prod-like: small percentage of real traffic for final validation before full rollout.

2. Selective integration tests

• Contract testing (consumer-driven): use Pact or similar so each service validates provider contracts during PRs — fast and isolates cross-team API breaks.
• Component integration smoke: for changed services, run small targeted suites exercising critical flows (auth, billing) against mocked or real collaborators.
• End-to-end smoke in staging: minimal scenario set (10–20 key flows) run in parallel nightly and on trunk merges.
• Chaos & resilience: scheduled fault-injection in staging/canary for SLO-sensitive paths.

3. Trunk-based testing & CI optimization

• Pre-merge unit + contract tests run on PRs; full integration only on trunk/merge to avoid combinatorial PR runs.
• Feature flags + dark launches so trunk always releasable; integration failures can be rolled back via flags.
• Test selection: use change-impact analysis to run only affected integration tests (trace dependencies via service registry / call graph).
• Parallelism & caching: split suites across runners, cache build artifacts and docker layers.
• Time budgets: enforce CI job SLAs; prioritize fast feedback (PR < 10m for unit/contract, merge pipeline < 30m for targeted integration).

4. Supporting practices

• Service dependency map + CI-aware test matrix to know what to run when service A changes.
• Consumer teams publish stable contracts and versioning policy; SRE owns orchestration and monitoring for test environments.
• Observability: identical metrics/logging in test envs and automated verification against SLOs.

Example flow:

• Dev opens PR → runs unit + contract tests (local + CI) → on merge, CI runs targeted integration smoke against ephemeral staging with the last-known-good versions of dependent services → if green, canary deploy with 5% traffic and automated SLO checks → promote.

Why this works:

• Shifts integration failure detection left via contract tests and sandboxes.
• Reduces CI time by running expensive integration suites only when necessary, selecting tests by impact, and parallelizing.
• Keeps trunk safe via feature flags and automated canaries, letting teams iterate rapidly without false positives.

Synthetic monitoring uses automated, scripted requests from controlled locations to emulate user interactions and verify service behavior end-to-end, independent of real-user traffic. It's ideal after infrastructure changes to catch regressions before customers do.

Three synthetic checks:

1. Basic health and auth check

• What it validates: GET /health or a lightweight authenticated endpoint to confirm service up, DB/cache connectivity, and that auth tokens are issued/accepted.
• Regression detection: Run every 30s; alert if consecutive failures >3 or status != 200 or auth flow error. Use uptime SLIs and notify when error rate exceeds SLA (e.g., >0.5% over 5m).

2. Critical-path end-to-end flow

• What it validates: A realistic sequence (obtain token, POST /items create, GET /items/{id} read, DELETE) to validate request routing, persistence, background workers, and data consistency.
• Regression detection: Measure success rate and end-to-end latency (p50/p95/p99). Alert on functional failures (any step failing) or latency breaches (p95 > threshold) sustained over N runs. Store per-deploy baselines and compare post-deploy deviations using percentage-change or statistical tests.

3. Schema and correctness check

• What it validates: Call a key read endpoint and validate response schema, required fields, status codes, and business rules (e.g., price >= 0).
• Regression detection: Use JSON schema validator and diff recent responses; alert on unexpected field removals/changes or content anomalies (missing fields, types). Integrate with anomaly detection to flag deviations from historical response distributions.

Implementation notes:

• Run checks from multiple regions, use retries with backoff, tag runs by deployment ID. Automate regression detection by comparing current metrics to pre-deploy baselines, triggering alerts and automatic rollback/runbook if thresholds exceeded.

Test data management for infrastructure testing is the practice of provisioning, protecting, and maintaining the data and credentials used by CI/CD pipelines, integration tests, chaos experiments and staging environments so tests are reliable, repeatable, and safe. For an SRE this means ensuring tests exercise realistic conditions without leaking sensitive production secrets or creating long-lived attack surfaces.

Two safe strategies

1. Ephemeral credentials (short-lived, scoped tokens)

• What: Issue credentials dynamically (e.g., via Vault/OIDC/STSEndpoint) with narrow scopes and TTLs; tests request, use, then revoke them.
• Benefits: Minimizes blast radius, reduces risk from leaked tokens, supports auditability, and aligns with least privilege.
• Trade-offs: Adds system complexity (credential broker, rotation logic), potential flakiness if issuance fails or TTLs expire mid-test; needs reliable network/PKI and retry logic.

2. Secrets redaction and token fencing

• What: Store sensitive values in secrets manager and never include them in logs/artifacts; redact or mask outputs, and replace real secrets with synthetic or hashed values where full fidelity isn't required.
• Benefits: Simple to adopt, reduces accidental exfiltration via logs or test artifacts, and allows post-test retention of traces without secrets.
• Trade-offs: Redaction can hide useful debug info; synthetic data may not exercise full production paths (e.g., encryption-specific behaviors). Requires discipline in tooling to enforce redaction and secure access controls.

Best practice: Combine both—use ephemeral, scoped credentials for tests that must hit real services and redact any outputs; automate issuance/revocation, bake retries, and include auditing and monitoring to detect misuse.

Situation: We must make testing (chaos experiments, risky infra changes) proportional to an SLO-driven error budget so reliability is protected while enabling change.

Policy (calculations + mapping)

• SLO: 99.95% availability → allowed downtime = 0.05% of time.
  • Per month (30 days = 43,200 minutes): allowed downtime = 0.0005 * 43,200 = 21.6 minutes.
• Reserve runway: split error budget into categories: planned experiments (50%), tooling flakiness/unknown (20%), emergency buffer (30%) → experiments budget = 10.8 minutes/month.
• Map experiment aggressiveness to budget cost:
  • Non-disruptive (read-only, limited traffic): ~0.1 min each
  • Scoped canary (1% traffic, degraded latency): ~1–2 mins
  • Cluster-wide destructive (full restart, failover): estimated mean outage 5–15 mins
• Allowable disruptive experiments per month = experiments budget / estimated cost. Example: if cluster-wide ≈10 min, allowed ≈1 per month; if canaries ≈2 min, allowed ≈5 per month.

Enforcement & automation in CI/CD

1. Policy as code: store SLOs, budget, experiment-cost catalog in repo (YAML).
2. Budget service: small microservice that tracks current month consumption, exposes REST API and tokens.
3. CI/CD gates:
   • Pre-approve experiments by checking Budget API; pipeline step reserves budget token before execution; if insufficient, fail and notify.
   • Automated post-test reconciliation: experiment reports actual impact telemetry (synthetic checks/SLI) back to Budget service which deducts actual consumed minutes.
4. Integration points:
   • Chaos/experiment runner (e.g., Litmus, Chaos Mesh) calls Budget API to reserve and to report.
   • Deployment flows consult Budget API before risky infra rollouts; if budget low, only allow canaries or schedule deferred.
5. Observability + rollback:
   • Instrument SLIs (availability, latency) and real-time alerts; if SLI breach or budget near-zero, CI disables disruptive pipelines via feature-flagged gates and opens a human review workflow.
6. Governance:
   • Owner per service who can approve emergency exceptions; monthly review of budget consumption and blameless postmortems; adjust cost estimates based on measured impact.

Example CI gate (pseudo):

- name: reserve-error-budget
  uses: internal/budget-check@v1
  with:
    service: payments
    requested_minutes: 10

Why this works

• Quantifies risk in minutes tied to SLOs, making trade-offs explicit.
• Automates enforcement to prevent accidental depletion.
• Continuous feedback (actual impact reporting) refines estimates over time.
• Balances velocity and reliability with clear governance and emergency paths.