Architecture Documentation and Communication Questions

Situation: As the SRE responsible for platform reliability and compliance, I produce a set of documentation artifacts that map system architecture to data flows, storage locations, and access controls so auditors (GDPR/HIPAA) can verify data handling and proof of controls.

Required artifacts (deliverables):

• Data Flow Diagram (DFD) PDF: annotated architecture showing data ingress, processing, storage, and egress points.
• Data Inventory / Mapping Spreadsheet: row-per-field mapping to processor, storage, retention, purpose, legal basis.
• Access Control Matrix: roles → allowed actions → justification and approval owner.
• Retention & Deletion Policy + Automated Workflow: retention triggers, deletion proof, QA checks.
• DPIA summary (if high-risk): risks, mitigations, residual risk.
• Evidence bundle index: where logs, snapshots, approvals live (S3 compliance bucket, SIEM, Confluence page, secure Git repo).
• Audit log & Monitoring spec: retention, integrity (WORM), alerting thresholds, periodic review cadence.

Sample mapping: User Profile Service (selected PII fields)

• Service & Architecture:

  • Frontend (React) → API Gateway (HTTPS) → Auth Service (OAuth2) → User Profile Service (K8s) → Postgres (encrypted) + S3 (profile images)
  • Backups: nightly DB snapshots to encrypted object storage; analytics pipeline receives hashed identifiers.

• Data map (fields):

  1. full_name
     • Category: PII (identifier)
     • Purpose: account display, support
     • Processors: User Profile Service (in-house), Support Dashboard (read-only)
     • Storage: Postgres RDS (AES-256 at rest), cached in Redis (TTL 5m)
     • Retention: until account deletion + 30 days recovery
     • Access policy: Role-based (service accounts, support_read role via IAM), MFA required, access via just-in-time session
     • Evidence storage: DB audit logs (RDS audit exported to SIEM), IAM approval entries (Confluence record + signed ticket)
  2. email
     • Category: PII (contact)
     • Purpose: auth, notifications
     • Processors: Auth Service, Email Provider (third-party: SendGrid)
     • Storage: Postgres (encrypted), masked in analytics (hashed)
     • Retention: until account deletion; transactional email logs retained 90 days
     • Access policy: Only auth and admin roles; no plaintext export without DPO approval
     • Evidence: Email provider contract (DPA) in secure Git, email send logs in SIEM
  3. date_of_birth
     • Category: Sensitive (age verification)
     • Purpose: age gating
     • Processors: User Profile Service
     • Storage: Postgres (encrypted)
     • Retention: hold only if required by service; otherwise remove after verification
     • Access policy: Restricted to compliance team; access requires ticket + manager approval
     • Evidence: Access approval tickets, DB audit trail
  4. ssn (if applicable)
     • Category: Highly sensitive (HIPAA/PII)
     • Purpose: identity verification (explicit legal basis)
     • Processors: External KYC vendor (processor)
     • Storage: Not stored internally; tokenized reference only
     • Retention: vendor-controlled per contract; internal token retained 7 days
     • Access policy: Prohibited internally; only DPO and vendor access documented
     • Evidence: Vendor DPA, tokenization logs, KMS key usage logs
  5. ip_address
     • Category: Personal data (GDPR)
     • Purpose: security logs
     • Processors: Load balancer, logging pipeline
     • Storage: ELK/SIEM (encrypted), retained 90 days (or per incident)
     • Access policy: Security team only; exports require incident ticket
     • Evidence: SIEM audit trail, retention lifecycle policy

Controls & mapping to architecture:

• Encryption:
  • In transit: TLS 1.2+ across all public endpoints.
  • At rest: KMS-managed AES-256 for DB and object store; KMS key rotation every 90 days.
  • Evidence: KMS audit logs in CloudTrail/SIEM.
• Access Controls:
  • IAM with least privilege; roles defined in Access Control Matrix.
  • Service accounts limited to network-scoped VPC and PodIdentity for K8s.
  • Secrets stored in Vault (audit enabled).
  • Just-in-time admin access via PAM with time-limited sessions; all sessions recorded.
  • Evidence: PAM session recordings, Vault audit logs, IAM change logs.
• Network:
  • Private subnets for DB; no public IPs.
  • Egress restricted; third-party vendor endpoints via allowlist only.
  • Evidence: VPC flow logs, firewall rule change records.
• Logging & Integrity:
  • Audit logs sent to immutable storage (WORM-enabled bucket) and SIEM.
  • Log retention per regulation (GDPR/ HIPAA): config-defined, deletions recorded.
  • Evidence: WORM bucket manifests, SIEM search snapshots.
• Data Minimization & De-identification:
  • Analytics receives hashed identifiers; PII stripped or tokenized before transfer.
  • Evidence: ETL transformation scripts in secure Git with signed commits.

Retention & Deletion workflow:

• Automated scheduler triggers soft-delete (account disabled), data retention timer starts.
• After retention window, automated deletion job removes PII and writes deletion proof (hash snapshot) to compliance bucket.
• Manual exception flow: DPO ticket + logged approval; exceptions recorded in DPIA register.

Where artifacts/evidence live:

• Data map & policies: Confluence (restricted to Compliance, Security, SRE, Legal)
• Configuration & IaC: secure Git repo (branch protection, signed commits)
• Audit logs / proofs: WORM S3 compliance bucket (encrypted, restricted), SIEM for searchable logs
• Contracts & DPAs: Secure Document Store (org-specific) with access audit
• Change approvals: Ticketing system (retention aligned to audits)

How this aids auditors:

• Single-pane mapping from field → storage → processors → controls → evidence location.
• Enables quick reproduction of access history, proof of encryption, vendor contracts, deletion proofs and role approvals.

This mapping is maintained as living documents: automated tests verify encryption/config, periodic access reviews (quarterly), and tabletop drills validate retention/deletion workflows.

SLI, SLO, SLA — definitions

• SLI (Service Level Indicator): a quantitative measure of some aspect of service behavior (e.g., request latency, error rate, availability).
• SLO (Service Level Objective): a target for an SLI over a defined measurement window (e.g., 99.9% of requests < 300ms per 30 days).
• SLA (Service Level Agreement): a contractual commitment (often with financial penalties) between provider and customer, usually based on SLOs but legally binding.

Concise format to include in architecture docs/diagrams
Use a compact table or annotation card near each component with:

• SLI name: short descriptive label
• Metric expression: precise metric query (e.g., rate(status>=500)/rate(requests))
• Measurement window: rolling window and evaluation frequency (e.g., 30d rolling, evaluated hourly)
• Threshold (SLO): target and error budget (e.g., 99.9% success → 0.1% error budget)
• Sampling method & granularity: sampling/aggregation (e.g., full sampling with 1m aggregation; or 1:1000 sampling)
• Alerting threshold: when to page (e.g., 99.5% over 1h)
• Owner: team and on-call role (e.g., Payments SRE, @payments-oncall)
• Notes: caveats, test traffic exclusion, synthetic vs. real-user

Example SLI for HTTP checkout endpoint

• SLI name: checkout_success_rate
• Metric expression: (sum(rate(http_requests_total{handler="/checkout", status=~"2.."}[5m])) / sum(rate(http_requests_total{handler="/checkout"}[5m]))) * 100
• Measurement window: 30-day rolling, evaluated hourly
• SLO: ≥ 99.9% success over 30d (0.1% error budget)
• Sampling: server-side full sampling with 1-minute aggregation; synthetic requests excluded via label synthetic="true"
• Alert: fire when success_rate < 99.5% for 30 minutes
• Owner: Payments SRE (payments-sre@company.com), on-call: payments-oncall rotation

This format makes SLI/SLOs discoverable on architecture diagrams and actionable for operations.

Situation: We need an eventually-consistent cache tier in front of a primary DB to reduce latency and load while preserving acceptable correctness for users and meeting SLOs.

Narrative:

• Benefit: caches increase availability and read throughput; on DB outages the cache can continue serving (with stale data).
• Trade-off: stale reads vs availability. Eventual consistency means a window where clients see old values until invalidation or TTL expires. The size of that window depends on write propagation/invalidation strategy and TTL choice.
• SRE focus: bound stale window, measure and alert on freshness, and set SLOs that reflect acceptable staleness (e.g., 99.9% reads <= 100ms and 99% of reads fresher than 2s).

Diagram (ASCII) with annotations:
Client -> Cache (read)
| hit: serve (stale_window)
| miss: -> DB -> Cache (populate)
Client -> DB (write) -> Invalidation / Async update -> Cache

Annotations:

• Stale windows:
  • TTL-only: stale_window = TTL (e.g., TTL=60s → up to 60s stale)
  • Write-through (sync update): stale_window ≈ propagation latency (ms)
  • Invalidate-on-write (async): stale_window = time between DB commit and invalidation delivery + TTL race
• Invalidation flows:
  • Synchronous invalidation: DB transaction publishes invalidation before commit — strong freshness but increases write latency and risk on failure.
  • Async invalidation (eventual): DB emits event after commit → lower write latency, larger stale_window.
  • Write-through/update-on-write: writer updates cache and DB atomically via middleware or transaction — reduces stale reads but adds complexity.
• TTL strategies:
  • Long TTL: high hit rate, large stale windows.
  • Short TTL: fresher reads, higher DB load.
  • Adaptive TTL: increase TTL for cold/immutable objects, reduce for hot/mutable keys.
  • Staggered/refresh-ahead: background refresh before expiry to reduce miss spikes.
• User-visible correctness & SLO impact:
  • Strong freshness SLOs require sync invalidation/write-through, higher write latency and lower availability during partition.
  • High availability SLOs favor TTL/async invalidation; accept bounded staleness and add compensating controls (read-after-write routing, client-side version checks).
  • Measure: freshness (% reads with observed version == latest), error budget consumed by stale incidents, tail latency for writes and reads.
    Recommendations for SRE:
• Define freshness SLOs with measurable metrics.
• Use hybrid: read-after-write routing for writer clients + async invalidation + short TTL for mutable keys.
• Implement monitoring: cache hit rate, invalidation lag histogram, proportion of stale reads (via version checks), and alarms when stale_window or lag exceeds thresholds.

Minimum handoff bundle for a new on‑call SRE (what, why, where):

1. One‑page Service Overview

• What: purpose, owners, SLAs/SLOs, customer impact, normal business hours, paging thresholds.
• Why: quick context to judge severity and priority.
• Where: service page (top of page).

2. Incident Triage & Runbook (step‑by‑step)

• What: exact triage checklist for top alerts (confirm, impact, mitigation, escalation), command snippets, safe checks.
• Why: reduces mean time to acknowledge/mitigate; prevents unsafe actions.
• Where: runbook (linked from service page).

3. Runbook: Mitigation & Rollback Procedures

• What: deploy/rollback commands, feature‑flag toggles, circuit breakers, cache flush steps.
• Why: fastest path to restore service without guessing.
• Where: runbook (clear, copy‑pasteable commands).

4. Escalation & Contacts

• What: primary/secondary on‑call, pager rotation, pager escalation matrix, pager month contacts, team Slack/phone numbers, DRI.
• Why: know who to call when you need help or permission.
• Where: service page + runbook (redundant).

5. Architecture Diagrams (high level + annotated)

• What: components, data flows, dependencies, failover paths, external integrations, stateful stores.
• Why: understand blast radius, single points of failure, where to check.
• Where: diagram repository (link on service page/runbook).

6. Monitoring & Dashboards

• What: links to dashboards, key metrics/thresholds, example queries, logging locations (ELK/Datadog/Stackdriver).
• Why: where to verify health and identify root cause.
• Where: runbook (with direct links), service page summary.

7. Common Alerts & RCA / Known Issues

• What: list of frequent alerts, false positives, quick fixes, and recent postmortems with root causes.
• Why: avoid re‑investigating known problems; speed restoration.
• Where: runbook + postmortem repository (linked).

8. Access & Safety (credentials & permissions)

• What: where to get credentials (vault links), required roles, just‑in‑time access process.
• Why: prevent being blocked by permission issues; maintain security.
• Where: runbook (no secrets in docs) with vault links.

9. Deployment & CI/CD Summary

• What: pipeline status, rollback windows, canary strategy, last successful deploy link.
• Why: to determine if incidents correlate with deployments and how to revert.
• Where: runbook + CI/CD system links.

10. Postmortem Template & Reporting

• What: how to file an incident, timeline expectations, stakeholders to notify.
• Why: ensures consistent learning and fixes.
• Where: service page + postmortem repo.

Practical notes:

• Put only links and short guidance on the service page for quick lookup; keep detailed procedures in the runbook.
• Diagrams live in a versioned diagram repo (Confluence/Draw.io/Diagrams.net/Git repo) so they’re editable and auditable.
• Keep runbooks executable and minimal — copy‑paste commands, exact timestamps, and validation steps. Update after every incident.

Suggested decision record template (language-agnostic, SRE-focused):

• Title: short descriptive name (e.g., "Cache vs Shard for user-session-store")
• ADR ID & Date
• Status: proposed / accepted / rejected / deprecated
• Context: current system, pain points, traffic/latency/scale facts
• Goal / Decision problem: what reliability/perf/cost requirement are we solving (SLOs, target QPS, latency)
• Constraints & Assumptions: budget, team skills, maintenance window, compliance
• Decision: concise chosen approach and rationale
• Alternatives Considered: list options (e.g., caching, sharding, DB replica, read-through cache) with short pros/cons per option
• Trade-offs & Risks: availability, consistency, operational complexity, cost
• Rollout Plan: staged rollout steps, feature flags, canary thresholds
• Rollback Plan: exact steps, automation hooks, health checks to trigger rollback
• Monitoring & Success Criteria: metrics to watch (errors, latency, cache hit rate), alert thresholds
• Testing & Validation: load test plan, chaos scenarios, data-migration verification
• Operational Runbook / Playbook: run commands, escalation path, runbook links
• Dependencies & Migration Steps: upstream/downstream impacts, DB schema changes
• Owners & Stakeholders: decision owner, reviewers, approvers
• Review Date / Sunset Criteria
• Links & Artifacts: design docs, benchmarks, PRs, dashboards

Why: This captures context, alternatives, measurable validation, and explicit rollback/runbook so future engineers can reproduce, evaluate, or reverse the change quickly.