Project Delivery and Accountability Questions
How you ensure projects stay on track, handle scope creep, communicate delays to leadership, recover from setbacks, balance technical excellence with delivery, and take ownership of outcomes even when multiple factors are involved.
EasyTechnical
52 practiced
How do you keep project documentation, runbooks, and release notes up to date while development moves quickly? Provide a lightweight checklist and a repeatable process a 3–7 engineer team can follow each sprint to ensure documentation is reliable for on-call responders and new hires. Include triggers for documentation updates (e.g., schema changes, auth flows) and ownership rules.
Sample Answer
I keep docs reliable by treating documentation as part of the sprint “definition of done” and by using small, repeatable steps each sprint so it’s low-friction and predictable.Lightweight checklist (to run each sprint / PR):- Update runbook snippet if change affects startup, healthcheck, failure modes, or recovery steps.- Update release notes entry (what changed, risk, rollout steps, config flags).- Update API/schema/contract docs and add migration notes if breaking.- Add or update auth/permission flows and onboarding steps if user roles changed.- Link new diagrams, design decisions, and test/monitoring gaps.- Mark unchanged docs “verified <date, initials>” if reviewed.Repeatable sprint process (3–7 engs):1. Sprint planning: call out potential doc-impacting items; assign doc owners per story.2. In-dev: small doc edits in the same branch/PR; reviewers check docs as part of code review.3. Pre-merge: checklist item in PR template requires doc updates or “no doc impact” justification.4. Release: release manager publishes release notes (auto-collect PR summaries + manual highlights).5. Post-release: on-call runs quick verification; any gaps generate follow-up tickets.Triggers for updates:- Schema/DB migrations, new API endpoints, auth flows, feature flags, CI/CD or config changes, observability/alert rule changes, major performance optimizations, or changed rollback steps.Ownership rules:- Story owner = doc owner by default; inviteops/oncall for runbook changes.- Maintain a “docs steward” rota (weekly) for cross-checks and tech-writing tweaks.- Critical runbook changes require at least one on-call + one engineer review.- Use Git history and PR linking to hold people accountable.Tools & tips:- Keep runbooks brief: “when/why/how/rollback” + links.- Use templates for runbooks and release notes; automate collecting PR summaries.- Track doc debt as backlog items and allocate 1–2 story points per sprint for cleanup.Outcome: low overhead, timely docs for responders and new hires, and measurable doc coverage tied to sprint work.
HardSystem Design
44 practiced
Design a phased multi-region deployment strategy for a compliance-sensitive feature that requires EU data residency. Address data residency and routing, regional failover, staged canary rollouts per region, rollback procedures, monitoring and alerting per region, and cost-control considerations. Discuss trade-offs between latency, consistency (cross-region reads/writes), and operational complexity.
Sample Answer
Requirements & constraints:- EU data residency for all personal data at rest and for any processing that laws require.- Multi-region availability (EU primary, other regions secondary).- Phased rollout with per-region canaries, automated rollback, regional failover, per-region monitoring, and cost controls.- Low latency for regional users; acceptable eventual consistency for non-critical cross-region reads.Phased plan (high level):1. Design & infra prep (2–4 weeks): deploy EU-region data stores, separate PII schema/tenant flags, routing layer, CI/CD with feature flags supporting region-scoped toggles.2. Internal canary in EU (1 week): enable feature for internal traffic subset in EU only.3. Staged EU customer canaries (2–4 weeks): progressive ramp by customer subset and by percentage per region.4. Expand to other EU regions (if any) then non-EU regions ensuring non-EU traffic never stores PII in EU (read-only or anonymized as required).5. Global safe mode: globally disable writes if cross-region compliance risk detected.Architecture & data residency/routing:- Data plane: EU primary databases (encrypted at rest, HSM/KMS in-region). Non-PII services replicated globally.- Control plane: Global control APIs run anywhere but must not instruct cross-region writes of PII.- Routing: edge (CDN + regional ingress) + regional API gateways that enforce routing rules: writes for EU users → EU DB; reads: prefer local caches for non-PII; for PII, read from EU replica cached regionally with strict TTLs & encryption.- Use per-tenant metadata to enforce residency.Regional failover:- Active-passive for EU data stores with automated failover to a standby in same legal jurisdiction or, if unavailable, a locked-read-only fallback and alerting. Full active-active only if replicas are within EU boundaries and conflict resolution preserves compliance.Staged canary rollout per region:- Feature flags with region scope and percentage ramp.- Start with 1% users → 5% → 25% → 100%, with automated health gates (error rate, latency, business metrics).- Progressive rollouts isolated per region so a rollback in one region doesn’t impact others.Rollback procedures:- Immediate: turn off feature flag for region; traffic reverts to prior code path.- Data cleanup: for writes accepted during canary that must be removed, run reversible operations (soft-delete + audit) and a compensating job; design for idempotency.- DB schema migration patterns: backward-compatible changes, feature-flagged migrations, and ability to run reads against old schema.Monitoring & alerting per region:- Metrics: latency (P95), error rate, CPU/memory, DB write success, data residency violations, business KPIs.- Logs/events tagged by region + tenant; sensitive data redaction.- Alerting: region-specific alerts with runbooks; automated rollback when thresholds breached.- Observability: distributed tracing with region tags, SLA dashboards, audit trails for data residency checks.Cost-control considerations:- Keep EU-only PII stores minimal (right-size instances, use spot/commitment discounts).- Use cross-region read replicas sparingly—cache instead of full replication to reduce egress.- Tiered storage: hot EU storage for recent PII, cold archiving in-region for retention.- Autoscaling with caps per region; feature-flag-based throttles to avoid runaway costs during rollouts.Trade-offs:- Latency vs consistency: local reads are low-latency but may be stale if relying on asynchronous replication. For PII, prefer consistency within EU (single-region writes) and accept eventual consistency for non-PII global reads.- Operational complexity: strict residency + per-region canaries increases operational overhead (routing, monitoring, runbooks). Mitigate with automation (feature flags, infra-as-code, policy engine).- Cost vs redundancy: full active-active across EU zones increases availability but costs more; active-passive reduces cost but increases RTO.Key safeguards:- Policy enforcement layer that prevents non-EU writes for tagged PII.- Audit logs and regular compliance tests (chaos/regulatory drills).- Clear rollback + data remediation plans and automated health gates for each rollout step.
MediumSystem Design
48 practiced
Sketch a 6-week plan to turn an MVP into a production-ready feature. Cover testing strategy (unit, integration, end-to-end, performance), observability (metrics, logging, alerts), security review checkpoints, performance testing milestones, and a staged rollout plan (canary/percentage ramps). State the minimum acceptance criteria you would require to consider the feature production-ready.
Sample Answer
Requirements & constraints (assumed): MVP feature with existing basic tests, target production SLA, <1% error budget, ability to deploy canary and measure metrics.Week-by-week 6-week planWeek 1 — Stabilize & planning- Triage MVP bugs, freeze public API.- Define acceptance criteria, SLOs (latency, error rate), key metrics (traffic, success rate, p95/p99 latency).- Create test plan and observability checklist; schedule security review.Week 2 — Unit + integration coverage- Add/extend unit tests to cover edge cases and business rules (aim 80–90% for critical modules).- Integration tests for upstream/downstream dependencies (DB, auth, third-party).- Run static analysis and dependency vulnerability scan.Week 3 — End-to-end and contract tests- Implement deterministic E2E tests (happy + key failure paths) in CI.- Add consumer-driven contract tests for services.- Begin load test scripting (identify realistic traffic patterns).Week 4 — Performance & resilience- Run performance benchmarks (baseline and stress tests); measure p95/p99 and throughput.- Chaos/resilience tests (timeouts, retries, network partitions).- Fix bottlenecks; iterate.Week 5 — Observability & security checkpoint- Finalize telemetry: structured logs, distributed traces, metrics (business + infra).- Create dashboards and automated alerts (error rate > X, latency > Y, saturation).- Conduct formal security review (threat model, auth, secrets, OWASP) and address findings.Week 6 — Release readiness & rollout plan- Dry-run deployment to staging; run smoke tests and E2E.- Prepare runbook and rollback plan.- Staged rollout: canary 1% for 1-2 hours -> 10% for 24 hours -> 50% for 24–48 hours -> 100% if metrics stable. Automated rollback on alert thresholds.Testing strategy summary- Unit: fast, CI on every PR.- Integration: nightly and pre-merge for critical paths.- E2E: scheduled and pre-release smoke.- Performance: repeated before release and post-release spot checks.Observability- Metrics: traffic, success rate, p95/p99 latency, CPU/mem, queue depth.- Logging: structured logs, correlation IDs, retention policy.- Tracing: entry→downstream traces for latency analysis.- Alerts: actionable, with runbook links and automated paging.Security checkpoints- Dependency scan, SAST, secrets scan in CI (Weeks 1–3).- Threat model + pen-test or focused review (Week 5).- Approval gate before production rollout.Minimum acceptance criteria (must pass all):- Automated tests: unit + integration + E2E passing in CI.- Performance: meets SLOs under expected load (p95 & error budget).- Observability: dashboards and alerts in place and validated.- Security: no critical/high vulnerabilities; security review signed off.- Deployment: successful staging deployment and smoke tests.- Rollout plan and runbook completed with rollback tested.This plan balances speed and risk, with measurable gates before increasing user exposure.
HardTechnical
54 practiced
Propose a set of SLIs, SLOs, and an error budget policy for a high-throughput service that must innovate quickly while maintaining reliability. Provide example numeric thresholds (e.g., availability 99.95% monthly, p99 latency < X ms), describe how you would measure and aggregate these metrics, and explain how error budget consumption would gate releases or trigger mitigation actions.
Sample Answer
Requirements & approach:- Service must innovate quickly (fast deploys) while keeping strong reliability for users. Use a small set of business-aligned SLIs (availability, latency, error rate, and saturation) with clear SLOs and an explicit monthly error budget. Measure at request-level, aggregate with both rolling windows and calendar-month rollups, and use error-budget-driven release gating plus progressive mitigation.Proposed SLIs & numeric SLOs (monthly):- Availability (successful responses 2xx/3xx): SLO = 99.95% (monthly). Error budget = 0.05% of requests.- P99 latency (API response time for user-facing requests): SLO = < 250 ms.- Error rate (5xx responses or RPC failures): SLO = < 0.05% (aligns with availability); separately monitor client-visible errors SLO < 0.02%.- Saturation (CPU or queue length): SLO = < 70% sustained utilization (30-day average) to preserve headroom.Measurement & aggregation:- Instrument at the edge/load‑balancer or service ingress to capture every request with fields: timestamp, status_code, latency_ms, request_type, user_tier.- Use time-series DB (Prometheus + long-term store) and OLAP for request-level rollups.- Availability / error rate: count successful vs total requests per minute, keep per-region/per-endpoint dimensions. Aggregate to monthly by summing counts: monthly_success / monthly_total.- Latency: store histogram buckets (HDR or Prometheus histograms) and compute p99 from aggregated histograms per minute then take the max/median as needed. Use day-over-day and monthly p99.- Saturation: sample host-level metrics, aggregate to cluster percentiles (p95 host CPU).Error budget policy & gating:- Define monthly error budget = 100% - SLO. Example: with 99.95% availability and 10M monthly requests, budget = 5,000 failed requests.- Continuous monitoring: compute error budget consumed = failed_requests_this_month / budget. Track burn rate = (consumed) / (elapsed_fraction_of_month). Example: if 60% of budget consumed in first 10% of month, burn rate = 6x.- Policy tiers: - Green (burn < 1x): Normal deploy cadence allowed (canary, CI/CD). - Yellow (1x ≤ burn < 3x): Restrict risky changes — require extended canary (50% traffic for 30 min), additional approvals, increase observability (tracing sampling up). - Red (burn ≥ 3x or remaining budget < 10%): Freeze feature releases that modify critical paths; run incident review; roll back recent changes; allocate engineering war-room; prioritize reliability fixes.- Automated actions: - Automated rollback of last deployment if 5xx rate exceeds threshold sustained for 5 minutes and coincides with increased latency. - Auto-scale or increase replicas if saturation SLO breached. - Automatic promotion of increased logging/tracing for affected endpoints when burn enters Yellow.- Release gating integration: - CI/CD reads current error budget state via API. If Green -> allow normal canary (progress per metric). If Yellow -> require manual approval and extended canary windows. If Red -> block merges to main for non-critical fixes; only allow hotfixes with on-call approval.- Post-incident: - Blameless postmortem within 72 hours; update SLOs if business tolerance changes; replenish process (e.g., temporarily reduce feature rollout rate) to bring burn under control.Rationale & trade-offs:- Availability + p99 latency + saturation cover user impact, performance, and capacity risk.- Monthly SLO balances fast iteration with strong reliability; burn-rate gating favors automated, objective decisions rather than ad-hoc judgments.- Use histograms and request-level counts for accurate aggregation and minimal bias from outliers.Example numbers recap:- Availability 99.95% monthly, p99 < 250 ms, error rate < 0.05%, saturation < 70%. Error budget-driven gating: Green <1x, Yellow 1–3x (restricted release), Red ≥3x (freeze + mitigation).
HardSystem Design
55 practiced
Design an accountability and approval governance model for scope changes in a safety-critical payment system. Define roles (who can propose, who must approve), approval gates, required evidence (automated tests, security audit, compliance sign-off), rollback authority, and how to ensure auditability and compliance reporting. Discuss trade-offs between speed and safety and how you'd communicate those trade-offs to leadership.
Sample Answer
Requirements:- Safety-critical payment system: no data loss, regulatory compliance (PCI DSS, local regs), low-risk of incorrect funds movement. Support rapid fixes but enforce strong controls and audit trails.High-level model:- Roles: - Proposer: Any engineer or product owner (must include risk assessment). - Technical Reviewer: Senior engineer/architect (validates design, testability). - Security Reviewer: AppSec engineer (security impact, threat model). - Compliance Officer: Legal/Compliance (regulatory fit). - Change Approver (Gatekeeper): Change Advisory Board (CAB) for normal, Emergency CAB (ECAB) for urgent fixes. - Rollback Authority: Release Manager or on-call SRE with documented escalation path.Approval Gates & Evidence:1. Design Gate: Proposed change submitted in RFC with risk matrix, sequence diagrams, and rollback plan.2. Build Gate: CI passing with unit and integration tests; static analysis; dependency checks.3. Pre-Production Gate: Automated end-to-end tests on staging with synthetic payment flows, chaos tests for resilience, and performance benchmarks. Security scan (SAST/DAST) and third‑party library audit.4. Compliance Gate: Compliance Officer signs off, verifying required controls and documentation.5. Production Gate: CAB approves deployment window; deployment must be automated with feature flags/gradual rollout and observability checks.Required Evidence (each gate):- Test coverage reports, automated test run artifacts, security scan reports, vulnerability triage, compliance checklist, deployment playbook, monitoring dashboards, and canary metrics.Rollback Authority & Playbook:- Pre-authorized rollback by Release Manager/SRE on thresholds (error rate, reconciliation mismatch). Playbook includes immediate rollback steps, customer notification template, RCA timeline, and freeze policy.Auditability & Reporting:- Immutable audit log (WORM storage) of change requests, approvals, artifacts, hashes of binaries, CI run metadata, and timestamps. Daily/weekly compliance reports (KPIs: mean time to approve, failed deploy rate, number of emergency changes). Integrate with SIEM and retention per regulation.Trade-offs (speed vs safety):- Faster changes reduce time-to-market but increase risk. Use mitigations: feature flags, canary rollouts, automated gating to keep velocity while containing blast radius. For high-risk scope changes (payments logic), prefer slower path with full CAB; for low-risk non-critical changes, use streamlined approvals.- Communicating to leadership: present quantified risk (expected loss, outage probability), time-to-ship impact, and proposed mitigations. Offer SLAs: “X% of low-risk changes deploy in Y hours; high-risk changes follow Z-day approval” and show historical metrics and cost of incidents to justify balance.This model preserves safety with clear accountability while enabling controlled speed through automation, staged rollouts, and risk-based gating.
Unlock Full Question Bank
Get access to hundreds of Project Delivery and Accountability interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.