MediumSystem Design

46 practiced

Design a verification and rollout plan to ensure a library vulnerability fix is applied across a heterogeneous fleet of 10,000 nodes with different OS/package managers. Include canary strategy, automation, verification tests, monitoring, and rollback criteria to minimize business impact.

Sample Answer

Requirements & constraints:- Fix a library vulnerability across 10k heterogeneous nodes (various Linux distros, Windows, containers) with minimal business impact, maintain SLAs, ability to rollback.

High-level plan:1. Inventory & grouping- Use CMDB + configuration management to discover package manager, OS, container vs host.- Group nodes by risk, criticality, package manager, and service owner.

2. Build artifacts & automated installers- Create uniform delivery: patched package per package-manager (deb/rpm/choco/winmsi) and container base-images.- Provide idempotent automation playbooks (Ansible/Puppet/Chef/Salt) and scripts for manual windows tasks (PowerShell DSC).

3. Canary strategy- Stage 0 (lab): test on CI pipeline and integration env.- Stage 1 (canary-1): 1% of low-risk nodes (non-prod), validate for 48h.- Stage 2 (canary-2): 5% mixed prod but non-critical services for 72h.- Progressive rollout: 10% → 30% → 60% → 100%, with automated hold gates and human approval.

4. Verification tests (automation)- Pre-check: package hashes, dependency resolution.- Post-install smoke tests: start/health-check services, run unit integration tests, run critical user journeys.- Security verification: vulnerability scanner (SCA) confirms fix.- Performance sanity: lightweight latency/throughput test.

5. Monitoring & alerting- Define KPIs: error rate, latency, CPU/memory, restart rate, package failure rate.- Dashboards and automated alerts (PagerDuty) for threshold breaches during each stage.- Log correlation to detect new exceptions.

6. Rollback criteria & procedure- Automatic rollback if: >X% increase in error rate (e.g., 3x baseline), service unavailability >Y minutes, failed installs >5% in stage, or critical owner veto.- Rollback methods: automated package downgrade playbooks or container redeploy to previous image. Include post-rollback verification.

7. Safeguards & operational controls- Rate limit concurrent upgrades per region; maintenance windows for critical services.- Backup config + snapshot where possible.- Change window, communication plan, runbook, and on-call rotation.

8. Automation & audit- All steps driven by CI/CD pipeline with feature flags, audit logs, and idempotent retries.- Post-rollout audit and postmortem.

Trade-offs:- Slower progressive rollout increases safety but delays remediation. Use high-priority grouping for critical hosts to accelerate safely.

MediumSystem Design

33 practiced

Design a remediation-ticket lifecycle: define states (e.g., new, assigned, in-progress, verification, verified, closed, risk-accepted) and transitions. Explain where to place automated policy gates (e.g., prevent close without verification) and how to detect and correct state drift (tickets stuck in intermediate states).

Sample Answer

Requirements:- Track remediation tickets from discovery to closure with auditability and enforcement of verification/compliance gates- Prevent unsafe transitions (e.g., close without verification) while allowing risk-acceptance where justified- Detect/correct state drift and stuck tickets automatically and with human escalation- Scale to thousands of tickets, integrate with monitoring, issue trackers, CI, policy engines

States & transitions (state machine):- New -> Assigned (auto-assign rules or manual)- Assigned -> In-Progress (assignee starts work)- In-Progress -> Verification (automation or assignee requests verification)- Verification -> Verified (automated tests/policy checks pass) OR -> In-Progress (failed verification)- Verified -> Closed (final sign-off, audit log) OR -> Risk-Accepted (stakeholder approves residual risk)- Risk-Accepted -> Closed (after documentation) OR -> In-Progress (rework requested)- Any state -> Cancelled (false positive) with audit reason

Automated policy gates (placement & enforcement):- Prevent New -> Closed: global gate — require flow through Verification/Verified unless ticket marked Risk-Accepted with signed approval.- On In-Progress -> Verification: trigger CI/CD jobs, integration tests, config scans, and policy-as-code engine (e.g., OPA/Conftest).- On Verification -> Verified: automatically allow only if all checks pass; else fail with detailed diagnostics.- On Risk-Accepted: require recorded approver identity, expiry, and compensating controls; gated by RBAC and automated policy that enforces minimum metadata.- Enforce gates at orchestration layer (ticketing workflow engine) and at system-side (prevent config rollbacks by infrastructure guardrails) to avoid manual circumvention.

Detecting & correcting state drift:- Define SLAs/TTL per state (e.g., Assigned 48h, Verification 24h). Emit metrics: time_in_state, transition_rates, stuck_counts.- Periodic scanner job: - Query tickets exceeding TTL; attempt automated remediation: add reminders, re-run verification jobs, reassign based on on-call rota, or run auto-rollbacks where safe. - If automated steps fail, escalate to owners via pagers and create high-priority incident with context.- Heartbeat pattern: tickets with external async actions (e.g., vendor fixes) register a heartbeat; missing heartbeats trigger re-evaluation.- Reconciliation loop: compare ticket state vs. authoritative system state (e.g., vulnerability fixed in asset inventory). If system shows fixed but ticket not Verified, auto-run verification and close; if ticket shows Verified but system not fixed, reopen and escalate.

Implementation components:- Workflow engine (e.g., Jira workflows or custom state-machine service)- Policy engine (OPA) for gates- CI/CD/test runners integrated via webhooks- Reconciliation service + scheduler for TTL checks- Observability: metrics (Prometheus), dashboards, alerts, and audit logs (immutable)

Scalability & trade-offs:- Use event-driven architecture (Kafka/webhooks) for transitions to scale.- Strong enforcement via centralized policy prevents bypass but needs RBAC/escape hatch with audit to handle urgent ops.- Balance auto-closing vs. false positives by requiring human sign-off only for risk acceptance; automate low-risk fixes fully.

Metrics to monitor:- Mean time in state, % tickets closed without verification (should be 0), number of escalations, reconciliation success rate.

This design enforces safe transitions, places gates where technical verification is possible, and uses automated reconciliation + escalation to detect and correct state drift.

HardSystem Design

26 practiced

You are responsible for compliance mapping: describe how you'd map postmortem findings and remediation actions to controls in two frameworks (e.g., SOC2 and GDPR). Explain how you'd maintain and version the mapping as policies and controls evolve, and how auditors can query this mapping.

Sample Answer

Requirements & scope:- Map each postmortem finding and remediation action to specific control(s) in SOC 2 (e.g., CC6.1 change management) and GDPR (e.g., Art. 32 technical measures).- Maintain history/versioning, evidence links, owner, risk rating, remediation status.- Provide auditors easy query access and exportable reports.

High-level design:1. Canonical data model (stored in a relational DB + Git for authoritative mappings): - postmortem(id, title, incident_date, severity, link) - remediation(id, postmortem_id, description, owner, due_date, status, evidence_links[]) - control(id, framework, control_id, title, description) - mapping(id, remediation_id, control_id, rationale, mapped_by, mapped_at, mapping_version) - mapping_version(version, changelog, effective_date, committed_by, git_commit_hash)

2. Process & tooling: - When a postmortem is created, SRE annotates remediation(s) and proposes mappings via a UI (or PR-to-Git workflow). Mapping proposals stored as JSON/YAML in a repo (e.g., compliance-mappings/) so every change is a Git commit + CI validation. - CI checks: - Validate control IDs exist for referenced framework - Ensure remediation has owner/due_date - Lint evidence links - On merge, CI writes a new mapping_version row (or tags the commit) and triggers a mapping-sync job to update the DB.

3. Versioning & policy evolution: - Treat mappings as code: every change = Git commit + semantic version (major when framework/policy changes, minor for new mappings, patch for metadata). - Control catalog stored in the repo and mirrored in DB; when frameworks update, update control definitions in repo with migration notes. Old mappings keep their mapping_version linking to the previous control definition snapshot. - Provide migration scripts to re-evaluate mappings if a control is split/renamed; keep historical mappings immutable but mark deprecated/updated controls.

4. Auditor access & queries: - Read-only reporting API and SQL views: - Example SQL to list remediations mapped to GDPR controls:

sql

SELECT p.id AS postmortem_id, p.title, r.id AS remediation_id, r.description, c.control_id, c.title, m.mapping_version, r.status
FROM postmortem p
JOIN remediation r ON r.postmortem_id = p.id
JOIN mapping m ON m.remediation_id = r.id
JOIN control c ON c.id = m.control_id
WHERE c.framework = 'GDPR';

- UI export: filtered CSV/PDF of mappings + evidence links + Git commit hash for traceability. - Provide per-audit snapshot: checkout the repo tag for the requested mapping_version and export controls + mappings so auditors can verify historical state.

5. Governance & roles: - SREs propose mappings; Compliance owner reviews and approves via PR. - SLA: mapping proposed within 3 business days of postmortem; compliance review within 5 business days. - Quarterly mapping review to reconcile framework updates.

Trade-offs & rationale:- Storing mappings as code (Git) gives audit-grade immutability, diffs, and review process; mirrored DB optimizes queries for auditors.- Keeping historical snapshots avoids losing context when frameworks evolve.- CI enforces consistency and reduces manual errors.

Edge cases:- Controls that partially apply: allow mapping with weighting/notes.- Control deprecations: keep historical links but mark mappings “to be revalidated”.- Large-scale renames: provide automated suggestions using text similarity but require human approval.

This approach ensures traceability from incident → remediation → control(s), versioned history for audits, and operational workflows aligned with SRE responsibilities.

EasyTechnical

32 practiced

Describe what a security incident postmortem (post-incident review) should include and why it's important for SRE teams. Explain the key sections you would expect (example: timeline, impact, root cause analysis, blast radius, remediation actions, verification steps, owner and due dates), and describe how the postmortem should convert findings into assigned, measurable remediation tasks that prevent recurrence.

EasyBehavioral

33 practiced

Behavioral: Describe a time when you tracked closure of remediation items from a postmortem and ensured fixes were verified. If you don't have direct experience, describe a clear, hypothetical approach you would take to ensure remediation ownership, verification steps, and evidence collection.

Findings Management and Remediation Tracking Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank