Role and Team Specific Responsibilities Questions
Focus on how a specific role fits within a particular team or domain, including domain ownership, systems and services owned, on call and incident responsibilities, autonomy and decision making, and how the role contributes to team outcomes. This topic also covers clarifying reporting relationships, typical stakeholders, and how the team is structured. Candidates should ask and answer concrete questions about team charters, ownership boundaries, and expectations for collaboration.
HardSystem Design
67 practiced
Design a platform and governance process to enforce SLOs across 200 services. Include SLO authoring workflow, telemetry collection architecture, centralized SLO dashboard, mechanisms to automatically enforce error budget policies (for example release blocking or throttling), and how to balance central enforcement with team autonomy.
Sample Answer
Requirements:- Functional: SLO authoring, standardized telemetry ingestion from 200 services, centralized dashboard, automated error-budget enforcement (release blocking, throttling), audit/history.- Non-functional: multi-tenant, low-latency alerts, scalable to 200+ services, RBAC, extensible policy engine.High-level architecture:Service teams → SDK/agent + exporter → Ingestion (Kafka) → Metrics/Tracing TSDB (Prometheus/Thanos + OpenTelemetry/Tempo) → SLO Engine & Policy Evaluator → Central SLO Dashboard & API → CI/CD integrations + Control plane (feature-flag/throttle) → Audit store (Elasticsearch).Key components:1. SLO Authoring Workflow- UI + YAML editor + templates (latency, availability, saturation). Authoring supports objective, service selector, window (rolling/periodic), SLI definition (metric query or trace-based). Git-backed SLO repo with PR workflow and automatic validation tests (syntactic + sample-data sanity checks).- Approval: Teams own SLOs; platform requires governance sign-off for critical infra services via automated checklist.2. Telemetry Collection- Standard SDK instrumentations and OpenTelemetry collectors at platform level. Export metrics to Prometheus/Thanos for long-term, traces to Tempo. Use Kafka to buffer spikes; apply enrichment & canonical labels (team, env, service).3. SLO Evaluation & Dashboard- SLO Engine computes burn rate, error budget windows using precompiled queries against TSDB; store results in a time-series and event DB. Dashboard shows per-service SLOs, error-budget remaining, burn-rate, alerts, and historical changes. Expose APIs for programmatic queries.4. Automatic Enforcement- Policy Evaluator evaluates rules (e.g., if error budget < 10% for 24h: block release) and invokes actions: - CI/CD integration: returns deny/soft-approve to pipelines (a webhook). - Feature flag throttling / rate limits via control plane (Flagsmith/LaunchDarkly-like or envoy rate-limiter). - Progressive enforcement: warnings → soft-block (human override) → hard-block.- All actions are auditable and include runbooks and runbook links.5. Balancing Central Enforcement & Team Autonomy- Default: team-owned SLOs and remediation playbooks. Central platform provides guardrails and templates. Enforcement is policy-driven and tiered: - Low-criticality: advisory alerts, recommended mitigations. - High-criticality infra: automatic hard-blocks after two-tier escalation.- Allow team exemptions via documented SLA exception process with temporary approvals and compensating controls (e.g., increased alerting, extra quotas).- Provide “self-serve sandbox” for teams to test SLO rules & enforcement in staging.Scalability & reliability:- Use partitioned Kafka topics, horizontally scalable TSDB (Thanos), and stateless SLO Engine instances behind leader-election for heavy windows. Cache evaluations for common queries.Trade-offs:- Strong central enforcement reduces outages but can slow teams — mitigated by graduated enforcement and fast exemption flow.- Exact SLI definitions can vary; encourage standard SLIs for common service types but allow custom SLIs with validation.Operational practices:- Quarterly reviews of SLOs with product & SRE, automated drift detection, and monthly error-budget retrospectives. Include SLIs in incident reviews to close loop between incidents and SLO adjustments.
HardSystem Design
79 practiced
Create a detailed onboarding checklist that a service owner must complete before their service is accepted into SRE ownership. Include required artifacts (SLIs with instrumentation, runbooks, automated deploys, chaos/testing results), validation steps, risk assessment, and the gating process to declare the service 'SRE-ready'.
Sample Answer
Requirements & scope- Define functional scope, traffic patterns, availability targets, SLOs/SLA, owner contacts, compliance constraints, expected peak QPS, and data sensitivity.Required artifacts (must be submitted)1. SLIs & SLOs - At least 3 SLIs (availability, latency P99/P95 for key API, error rate). - Instrumentation shown (traces, metrics, logs) with dashboard links.2. Runbook - On-call playbook: detection -> triage -> mitigation -> rollback steps with run commands and runbook IDs. - RCA template and escalation matrix.3. Deploy pipeline - CI/CD manifests, automated canary/blue-green strategy, automated rollback, and 1-click rollback documented.4. Tests & resilience - Unit/integration/e2e test coverage report. - Load test summary (documented test harness, traffic profile, thresholds). - Chaos experiments results (failure modes tested, blast radius, recovery time).5. Security & compliance - Threat model, vulnerability scan, secrets handling, RBAC.6. Observability - Dashboards, alerts (with runbook links), tracing spans tagged by request id.7. Ops automation - Health checks, self-healing hooks, capacity autoscaling config.Validation steps (SRE-led)- Instrumentation sanity: verify metric cardinality, missing tags, correct units.- SLI signal test: inject synthetic errors/latency; confirm SLI changes reflected.- Alert fidelity: fire simulated incidents and confirm alerts route and runbook resolves.- CI/CD run: perform staged deploy in sandbox, validate automated rollback.- Load & chaos re-run: meet SLOs under defined load and after simulated failures.- Security check: pass scans and access policy review.Risk assessment- Failure modes list, blast radius, mean time to recover (MTTR) targets, dependency map, mitigation controls.- Assign residual risk: Low/Medium/High and required mitigations before acceptance.Gating & acceptance (SRE-ready)- Mandatory gates: 1. All artifacts present and reviewed. 2. SLIs defined + instrumentation verified. 3. Runbook covers top-10 failure modes. 4. CI/CD supports automated canary and rollback. 5. Load and chaos tests pass within SLO margins. 6. Security/compliance sign-off. 7. Risk rated ≤ Medium or plan for mitigation.- Acceptance process: - Checklist review meeting with service owner, dev lead, product, and SRE reviewers. - Sign-off matrix: SRE engineering lead approves; security and product countersign. - Declare SRE ownership date, add to on-call rota, schedule 30/90-day SLO review.Post-acceptance- 30/90-day monitoring: validate error budget burn, adjust alerts, iterate runbook.- Quarterly review for capacity and new failure modes.
EasyTechnical
82 practiced
Explain the expectations for SRE on-call rotations within a product team: describe typical shift lengths, expected first-response time, minimum documentation each on-call should have at hand, what actions an on-call SRE can make without manager approval, and standard escalation paths and communication expectations during incidents.
Sample Answer
Typical expectations for SRE on-call in a product team:Shift length & cadence- Commonly 8–12 hour shifts (day/night) or 24–hour rotations with handoff; many teams use weekly rotations for engineers and longer (e.g., monthly) for escalation leads.- Ensure reasonable on-call load: no more than 1–2 incident-heavy shifts back-to-back and protected recovery time.First-response time- Target: acknowledge alerts within 5–15 minutes (depends on alert severity); begin active mitigation within 15–30 minutes for P1s.- SLAs for response should map to alert severity tiers defined by SLOs/error budgets.Minimum documentation every on-call should have- Runbook for common alerts (steps to triage, key dashboards, commands)- Ownership and escalation matrix (who to call for DB, networking, product)- Access list and credentials (vault links), service health dashboards, recent deploys, and rollback procedures- Contact list for on-call peers, managers, and on-call rotationsActions allowed without manager approval- Immediate mitigations: restart services, apply proven hotfixes, roll back recent deployments, scale replicas, toggle feature flags- Short-term config tweaks and traffic routing changes documented in the runbook- Anything that’s reversible and low-risk; escalate before any destructive or long-lived change.Escalation path & communication during incidents- Follow severity-based path: on-call → primary SRE lead → engineering/product owner → incident commander for P1s- Use a single source of truth (incident tracker) and a dedicated channel (e.g., #inc-<id>) for updates every 10–15 minutes initially- Appoint/confirm Incident Commander, Scribe, and Communicator for stakeholder updates- Communicate triage, impact, mitigation steps, ETA, and postmortem plan; notify stakeholders on major customer impact and when service is restored.Principle: act fast to stabilize, prefer reversible actions, keep clear, time-boxed communication, and record every step for post-incident analysis.
MediumTechnical
90 practiced
Design an automated Kubernetes deployment rollback strategy for microservices. Include the rollout strategy (canary/blue-green), health checks required to trigger automatic rollback, how to record and notify the team, and how to test rollback as part of CI pipelines.
Sample Answer
Situation & goal: Provide an automated, safe rollback strategy for Kubernetes microservices that minimizes blast radius, detects regressions quickly, records what happened, notifies the team, and is testable in CI.Rollout strategy (recommended): Canary with automated progressive traffic shifts (or blue-green when DB migrations require it).- Use Argo Rollouts or Flagger + a service mesh (Istio/Linkerd) or NGINX+TrafficSplit for traffic shifting.- Start with 1-5% traffic for new revision, increase to 25%, 50%, then 100% in monitored steps with pause windows.Health checks & rollback triggers:- Kubernetes probes: liveness + readiness must pass before any traffic shift.- Business/observability metrics (via Prometheus): error-rate increase (e.g., >2x baseline or >1% absolute), latency p50/p95 above SLA threshold, increased 5xx rate, saturation signals (CPU/memory >85%).- Synthetic checks / end-to-end user journeys: critical path failures.- Composite rule: trigger rollback when any of the above breach sustained for N minutes (e.g., 3 consecutive 1-min buckets).- Configure cooldown windows so flapping changes aren’t oscillated.Automation implementation:- Argo Rollouts example flow: install canary strategy, define analysis templates that query Prometheus and run webhooks; if analysis fails, Rollout executes automatic rollback to previous ReplicaSet.- Flagger integrates with Prometheus and Istio to automate canary promotion/rollback.Recording & notification:- Record events: annotate Deployment/Rollout with Git commit, build ID, image tag; emit Kubernetes events; log to centralized ELK/CloudWatch.- Post-mortem data: capture Prometheus timeseries window, traces (Jaeger), and request logs for the canary timeframe.- Notifications: on rollback trigger, send: - PagerDuty/SMS for on-call - Slack channel with structured message (service, image, commit, reason, metrics snapshot, rollback link) - Create a ticket in SRE/incident tracker (Jira) via automation- Persist audit: CI/CD pipeline should write a deploy record to a deployment DB or GitOps commit history.Testing rollback in CI:- Unit/build stage produces immutable image tag and pushes metadata.- Integration stage: run deploy to ephemeral cluster (kind/Kind + KinD, or real staging namespace) using same Argo/Flagger config; inject failures (mock upstream errors, increase latency, return 500s) using test harness or chaos tool to validate canary analysis detects and triggers rollback.- Add end-to-end test that asserts rollback occurred and previous revision regained 100% traffic and readiness.- Include scheduled chaos tests in CI/CD (canary-failure sim) to ensure alerts and notification playbooks fire.Operational best practices:- Keep rollout analysis thresholds conservative; tune per-service.- Use feature flags to decouple release from deploy when possible.- Add manual approval step for major releases (DB schema or high-risk).- Maintain runbooks linked in Slack messages with next steps for on-call.- Periodically rehearse rollback playbook and test pager flows.This approach gives fast automatic protection via canary analysis, full observability for post-incident work, and automated CI tests to ensure rollback logic stays reliable.
EasyTechnical
76 practiced
Implement a Python script to delete rotated log files older than 7 days in a given directory. Requirements: run on Linux, skip symlinks, handle large directories efficiently, log its actions, and support a dry-run mode. Describe edge cases you would handle in production.
Sample Answer
Approach: iterate directory with os.scandir (efficient for large dirs), skip symlinks, check file mtime older than 7 days, support --dry-run, log actions to stdout/file, handle permission/race conditions.Key points / complexity:- Time: O(n) scanning entries; Space: O(1) streaming via scandir.- Uses mtime (server local time/clock dependent).Production edge cases to handle:- Clock skew / DST / NTP drift — prefer comparing against curated timestamps or use UTC monotonic reference.- Files being written to while scanning (race): check size/mtime twice or rely on rotation scheme (e.g., .gz rotated).- Hard links / multiple references: deletion affects all links.- Permission/ownership differences — run with appropriate user or use sudo carefully.- Very large directories: consider batching, limiting per-run deletions, or using find + xargs for kernel-level performance.- Logging to central store and alerting on errors or unexpectedly high deletion counts.
python
#!/usr/bin/env python3
import os, sys, argparse, logging, time
from datetime import datetime, timedelta
# Set up logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s: %(message)s")
def delete_old_logs(path, days=7, dry_run=True):
cutoff = time.time() - days * 86400
removed = 0
scanned = 0
# Use os.scandir for performance on large directories
with os.scandir(path) as it:
for entry in it:
scanned += 1
try:
# Skip directories and symlinks explicitly
if entry.is_symlink():
logging.debug("Skipping symlink: %s", entry.path)
continue
if entry.is_dir():
logging.debug("Skipping directory: %s", entry.path)
continue
# Use stat without following symlinks
st = entry.stat(follow_symlinks=False)
if st.st_mtime < cutoff:
logging.info("%s: older than %d days (mtime=%s)", entry.name, days,
datetime.fromtimestamp(st.st_mtime).isoformat())
if not dry_run:
try:
os.remove(entry.path)
removed += 1
logging.info("Deleted: %s", entry.path)
except PermissionError:
logging.error("Permission denied: %s", entry.path)
except FileNotFoundError:
logging.warning("File disappeared before deletion: %s", entry.path)
except Exception as e:
logging.exception("Failed to delete %s: %s", entry.path, e)
else:
logging.debug("Kept (recent): %s", entry.name)
except PermissionError:
logging.error("Cannot access entry (permission): %s", getattr(entry, 'path', entry))
except FileNotFoundError:
logging.warning("Entry disappeared during scan: %s", getattr(entry, 'path', entry))
logging.info("Scan complete. Scanned=%d, Removed=%d (dry_run=%s)", scanned, removed, dry_run)
if __name__ == "__main__":
p = argparse.ArgumentParser(description="Delete rotated log files older than N days")
p.add_argument("dir", help="Directory to scan")
p.add_argument("--days", type=int, default=7, help="Age in days to delete (default 7)")
p.add_argument("--dry-run", action="store_true", default=False, help="Don't actually delete files")
args = p.parse_args()
if not os.path.isdir(args.dir):
logging.error("Not a directory: %s", args.dir); sys.exit(2)
delete_old_logs(args.dir, days=args.days, dry_run=args.dry_run)Unlock Full Question Bank
Get access to hundreds of Role and Team Specific Responsibilities interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.