System Reliability and Availability Questions

Situation: We need an automated safety check that runs immediately before promoting a canary deployment to full production to detect regressions and fail fast.

Preconditions (must be true before checks run):

• Minimum traffic: canary must have >= 5% of production traffic AND >= 1k requests/min over the last 5 minutes (adjustable by service).
• Stable infra health: no active node failures, orchestration errors, or recent scaling events in last 10 minutes.
• Sufficient sample window: canary has been running >= 10 minutes (or >= 3x median request latency).
• Error budget state: service has available error budget above a configured floor.

Concrete metric checks (compare canary vs baseline or last stable):

• Error rate: absolute increase > 0.5% OR relative increase > 2x -> fail. Also check 95% CI using Wilson score for proportions when counts are low.
• Latency:
  • P95 increase > 25ms AND relative increase > 20% -> fail.
  • P99 increase > 50ms OR relative increase > 30% -> fail.
• Traffic success ratio (2xx / total): drop > 1% absolute -> fail.
• Throughput: sustained throughput drop > 10% (indicates throttling/regression).
• Resource usage: CPU or memory per pod increase > 40% or the OOM/killed count > 0 -> fail.
• Dependent service error spike: any downstream service’s error rate increases > 50% relative and absolute >0.5% -> fail.
• SLO breach projection: using current error rate and traffic, project if SLO will be missed in the hour -> fail.

Statistical rigor:

• Use sliding windows: compare last 5 minutes of canary to last 30 minutes baseline.
• Require significance: differences must pass a two-sample test (e.g., Mann-Whitney for latencies, proportion z-test for error rates) at p < 0.01 to avoid noisy failures.
• Minimum sample size guard: if samples < threshold, mark metrics as inconclusive (do not promote) and extend canary time or require manual review.

Fail-fast logic:

• Any single critical check (e.g., error-rate doubling, P99 huge spike, OOMs) triggers immediate rollback.
• For non-critical checks, use weighted scoring: assign severity weights, sum > threshold -> fail and rollback.
• Implement escalated fast paths: if multiple independent metrics degrade simultaneously, immediate rollback.

Alerting and observability:

• Immediate pager for on-call with context: diffs, raw metric time series, statistical test results, last deployment manifest, affected pods.
• Post-failure runbook link with automated links to logs, flamegraphs, and tracing spans (sampled traces highlighting errors).
• Slack/issue automated summary for developers with playbook steps and button to re-run checks after fixes.

Automation flow (simplified):

1. Precondition checks -> pass? else block promotion and notify.
2. Collect metrics from telemetry system (Prometheus/Datadog) for canary & baseline.
3. Run statistical tests and threshold comparisons.
4. If fail -> trigger rollback job, send alerts with evidence.
5. If inconclusive -> extend canary window (auto-scale sampling) or require manual approval.
6. If pass -> promote and continue guarded ramp with ongoing monitoring.

Implementation notes / best practices:

• Keep thresholds configurable per service and environment.
• Run checks as part of the CD pipeline (e.g., Argo Rollouts, Spinnaker) with a pluggable policy engine.
• Store outcomes and decisions for audit and postmortem.
• Continuously tune thresholds based on historical false positive/negative rates.

To validate that a simplified Terraform plan snippet contains an aws_instance with instance_type 't3.micro', we can write a small pytest that traverses the "resource_changes" list and asserts at least one resource of type "aws_instance" has change.after.instance_type == 't3.micro'.

import pytest

sample_plan = {
    'resource_changes': [
        {'type': 'aws_instance', 'change': {'after': {'instance_type': 't3.micro'}}}
    ]
}

def has_aws_instance_with_type(plan, expected_type):
    for rc in plan.get('resource_changes', []):
        if rc.get('type') != 'aws_instance':
            continue
        after = rc.get('change', {}).get('after') or {}
        if after.get('instance_type') == expected_type:
            return True
    return False

def test_plan_contains_t3_micro_instance():
    assert has_aws_instance_with_type(sample_plan, 't3.micro'), "Expected aws_instance with instance_type 't3.micro' not found"

Explanation:

• The helper function defensively navigates the nested dict to avoid KeyError.
• The test asserts that at least one aws_instance in resource_changes has instance_type 't3.micro'.
• This pattern is easy to extend for multiple checks (e.g., tags, AMI) and suitable for SRE automated validations of generated plans.

Situation: We need to rotate TLS certificates across multiple regions with zero downtime. Goal: verify issuance, secure distribution, functional validation (handshakes, chain), monitoring for regressions, and safe rollback procedure.

Test Plan

Prereqs:

• Inventory of endpoints (by region, load balancer, ingress) and current cert fingerprint.
• CI/CD job to request certs from CA (ACME/Private) and store in secure secret store (KMS/HashiVault/Secrets Manager).
• Health/traffic divert capability (weighted LB routing) and canary config.

Issuance (automated):

• Trigger cert issuance for region R staged namespace, validate fingerprint and expiry.
• Verify cert metadata: CN/SANs, EKU, key type/size, validity period.
• Example checks: ACME response, Vault pkcs12 export.

Distribution (staged):

• Canary rollout: deploy new cert to 1 instance per AZ in Region A while keeping old cert on others.
• Gradually increase percentage (1% → 10% → 50% → 100%) per region, waiting for validations.
• Repeat region-by-region (start low-traffic region).

Validation checks:

• Handshake success: use automated probes from multiple client vantage points (external and internal). Example command:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

Assert: successful TLS handshake, server presents new cert fingerprint for canary hosts.

• Certificate chain: verify full chain and OCSP/CRL stapling. Example: parse openssl output for “Verify return code: 0 (ok)” and presence of stapled OCSP response.
• App-level tests: run HTTP(S) GETs, mutual TLS if used, and run integration smoke tests to ensure application behavior unaffected.
• Client matrix: test common client TLS versions and cipher suites used by consumers.

Monitoring and Alerting:

• Track metrics: TLS handshake failures rate, 4xx/5xx error rate, latency, cert expiry/fingerprint mismatches.
• Create Prometheus alerts:
  • handshake_failure_rate > threshold (e.g., >0.5% over 5m) → Pager
  • cert_fingerprint_mismatch detected → Pager
  • increased 5xx or latency correlated to canary hosts → Pager
• Log evidence: collect tcpdump/pcap and server TLS logs for failed handshakes.

Rollback:

• If alerts trigger or validation fails within rollback window:
  • Immediately revert LB weight to old instances (automated rollback script) and stop further rollouts.
  • Re-deploy old certificate from secure store to affected hosts.
  • Verify handshakes return to baseline, clear alerts, and create incident for root cause.
• Post-rollback: revoke problematic certs if compromised and block re-issuance until fix.

Additional:

• Runbook with commands, playbook owner, and “go/no-go” checklist.
• Post-deploy audit: certificate transparency logs, expiry alignment, and update documentation.
• Automation: CI/CD pipeline implements canary, validation tests, monitoring hooks, and automatic rollback on failed gates.

Result: phased, observable rotation with automated checks and safe rollback ensures no downtime and rapid recovery if issues occur.

Requirements & goals:

• Catch regressions fast (developer feedback), prevent risky infra changes, minimize cloud spend, and scale to hundreds of modules.

Testing matrix (tiered by risk & change-surface):

• Tier 1 — Fast unit/static checks (every PR, <2m):

  • Who: simple helper modules, pure locals/variables, data-only modules.
  • Tests: terraform fmt, validate, tflint, checkov/terrascan, unit tests with terraform-compliance or terratest table-driven mocks.
  • Why: cheap, fast feedback.

• Tier 2 — Lightweight integration plan-only (on PRs touching module or consumers, ~5–15m):

  • Who: modules that create IAM roles, security groups, networking primitives (non-destructive).
  • Tests: terraform init + plan against small test workspace using a “dry” backend (local/statefile) or remote with plan-only and no apply; use mocked data or limited resource counts.
  • Why: detect interpolation and dependency errors without provisioning.

• Tier 3 — Scoped ephemeral applies (scheduled on merge or on-demand in PR, 30–60m, limited resources):

  • Who: stateful modules (RDS, storage, VPCs for integration), modules with lifecycle/config that only show on apply.
  • Tests: terratest/integration applies to isolated test account/project with tagging and TTL; validate behavior with smoke tests; destroy after success.
  • Why: verify real provider behavior and provider-specific quirks.

• Tier 4 — Full environment deploys (nightly/weekly, costed, high-risk):

  • Who: core infra (multi-AZ networks, production-level clusters, cross-account infra).
  • Tests: full deploys into staging or canary prod-like environments with end-to-end tests and load smoke. Run off-peak and with approvals.
  • Why: catch integration/scale regressions.

Tagging & selection strategy:

• Use module metadata (file: TESTING.md or module meta attributes) with labels: test:fast, test:plan, test:apply, test:full and tags for cost_level:low/medium/high.
• CI detects changed modules via git diff against main and runs tests based on highest-required tier among changed modules and their transitive consumers (dependency graph).
• PR behavior: run Tier1 always; run Tier2 if module/interface changed; offer manual trigger button for Tier3 in PR; Tier4 runs only on scheduled pipelines or gated merges.

Cost controls:

• Use isolated low-cost test accounts/projects with strict quotas, pre-sized minimal instance types, and resource caps in module test variables.
• Enforce ephemeral TTLs and automated teardown; fail pipelines if destroy fails.
• Apply budget alerts and a CI budget guard that refuses Tier3/4 when monthly test spend exceeds threshold unless approved.
• Reuse sandbox resources where possible via namespaces to avoid frequent provisioning of large infra.
• Use provider mocks and local emulators (localstack, moto, kind/minikube) for many integration cases to avoid cloud calls.

Operational details & tooling:

• Store dependency graph and test tiers in a central registry (YAML/JSON); CI (GitHub Actions/Jenkins/Concourse) reads it.
• Use terratest (Go/Python) for apply-destroy and assertions; Harness feature flags for selective runs.
• Caching: cache providers/plugins, remote state where safe, and reuse init artifacts to speed plans.
• Observability: emit test telemetry (duration, cost estimate, failures) into dashboards; run blameless postmortems on infra test failures.

Trade-offs:

• More applies increase confidence but cost/time. The matrix biases fast feedback for most changes and reservates expensive applies for high-risk modules or scheduled windows. This balances speed, cost, and risk while scaling to hundreds of modules.

Approach: periodically compare desired resources in Terraform state to live resources from provider APIs, using batched API calls, caching, and heuristics to suppress noisy diffs. Flag only meaningful divergences; allow policy to mark non-critical drift.

# pseudocode / Python-like
def detect_drift(tfstate, api_client, cache, config):
    drift_report = []
    # group resources by provider+region to batch
    groups = group_by_provider_region(tfstate.resources)
    for group in groups:
        # use cached snapshot if fresh
        cache_key = make_key(group)
        if cache.is_fresh(cache_key, ttl=config.cache_ttl):
            live_index = cache.get(cache_key)
        else:
            live_index = fetch_live_resources(group, api_client, config)
            cache.set(cache_key, live_index)

        for res in group.resources:
            desired = res.attributes
            live = live_index.lookup(res.type, res.id)  # might be None
            diff = compare(desired, live, config.compare_rules)
            if diff:
                severity = classify_diff(diff, config.policies)  # critical / non-critical / ignored
                if not is_false_positive(diff, desired, live, config):
                    drift_report.append({"resource": res.addr, "diff": diff, "severity": severity})
    return drift_report

def fetch_live_resources(group, api_client, config):
    # rate-limit and retry wrapper
    with rate_limiter(api_client, qps=config.qps, burst=config.burst):
        items = api_client.list_resources(group.provider, group.region)
    return index_by_type_and_id(items)

Key points:

• Batching: list resources per provider/region to reduce API calls.
• Rate-limiting: token-bucket per provider client (qps + burst); exponential backoff on 429/5xx; circuit-breaker when provider unhealthy.
• Caching: TTL-based cache of list responses; invalidate on known infra changes (webhook from CI/CD) or short TTL for dynamic resources.
• Avoid false positives:
  • Normalize types (timestamps, tags ordering, whitespace).
  • Use compare_rules: regex ignore (e.g., autoscaling desired capacity), tolerances for floats.
  • Re-check transient diffs: require N consecutive detections OR confirm with second API call after short delay.
• Mark non-critical drift via policy: map attributes -> severity; report but don't alert on low severity; optional auto-correct via terraform apply in low-risk cases.

Complexity: O(N) resource comparisons; API cost dominated by provider list operations. Edge cases: missing permissions, eventual consistency, resources created outside TF but referenced.