Continuous Integration and Delivery Pipelines Questions
Design and implement continuous integration and continuous delivery pipelines that reliably build, test, validate, and deploy applications and infrastructure. Topics include pipeline as code practices, defining stages and triggers for builds and tests, automated testing strategies across unit, integration, smoke, and end to end tests, gating and environment promotion, branching and release strategies, artifact management and versioning, and deployment patterns such as rolling updates, blue green deployments, and canary releases. Candidates should be able to design rollback and recovery procedures, integrate infrastructure provisioning into pipelines, select and configure pipeline tooling such as Jenkins, GitHub Actions, GitLab CI, Azure Pipelines, or cloud vendor pipeline services, and reason about observability and reporting for pipeline health and test execution. Practical considerations include environment parity, pipeline security, secrets handling, pipeline as code best practices, and trade offs between speed and safety.
EasyTechnical
60 practiced
Compare rolling updates, blue-green deployments, and canary releases. For each pattern describe how traffic is shifted, rollback speed, capacity and provisioning implications, and typical use-cases. Explain trade-offs for SREs in terms of operational complexity, observability needs, and risk exposure.
Sample Answer
Rolling updates:- Traffic shift: Gradually replace instances in-place (batch-by-batch) behind the existing load balancer; traffic moves as each pod/node is drained and replaced.- Rollback speed: Moderate — can stop further batches and roll back remaining ones, but already-updated instances may need explicit rollback.- Capacity/provisioning: Minimal extra capacity required if you update in-place; can require extra capacity if doing surge/upscale to maintain availability during draining.- Use-cases: Routine patching, safe upgrades when state is compatible.- SRE trade-offs: Low infrastructure cost and simpler provisioning, medium operational complexity. Observability needs: clear per-batch metrics, deployment-stage health checks. Risk: exposure to partial-failures that affect subsets of users.Blue-green deployments:- Traffic shift: Switch all traffic at once from the blue (old) to the green (new) environment via routing change (DNS, load balancer).- Rollback speed: Very fast — flip back to previous environment instantly.- Capacity/provisioning: High — must provision full duplicate environment (100% capacity).- Use-cases: Major releases, schema-safe changes, zero-downtime requirements, compliance-sensitive rollouts.- SRE trade-offs: Simpler rollback and clear separation, but higher cost and orchestration complexity to keep both environments in sync. Observability: compare full-environment metrics and smoke tests before cutover. Risk: big-bang cutover risk if testing missed issues.Canary releases:- Traffic shift: Route a small percentage of traffic to new version, ramp gradually to 100% based on metrics.- Rollback speed: Fast for canary group (stop/reroute quickly); full rollback requires draining and switching remaining traffic.- Capacity/provisioning: Requires extra instances for canary slice (small additional capacity), minimal compared to blue-green.- Use-cases: Risky changes, A/B tests, feature flags, validating behavior in production.- SRE trade-offs: Highest operational complexity — needs sophisticated traffic routing and automated rollbacks. Observability: strong requirement for fine-grained, real-time metrics, automated anomaly detection, user- and request-level tracing. Risk: low blast radius if well-instrumented; high if observability gaps exist.Summary for SREs:- Choose rolling for cost-efficiency and simplicity, blue-green for fastest rollback and isolation at cost of capacity, canary for safest gradual verification but demands robust observability and automation. Prioritize health checks, SLO-driven gates, automated rollbacks, and runbooks to manage risk and complexity.
MediumTechnical
41 practiced
Propose SLOs and an error-budget policy that connect deployment frequency to user-facing reliability. Explain how the pipeline should react when error budget is exhausted (e.g., stricter gates, temporary deployment freeze), and whether these controls should be automated or human-driven.
Sample Answer
Requirements & targets:- SLOs (example, 30‑day window): - Availability: 99.95% (error budget = 0.05% → ~21.6 min downtime/month) - Latency: p95 < 200ms for API requests - User‑facing error rate: <0.5% of requests- Error budget defined on the primary user‑impact metric (availability or error rate) and evaluated on rolling 30‑day and 7‑day windows.Error‑budget policy linking to deployment frequency:- Define burn-rate = (error observed / error budget) per unit time. Set tiers with concrete controls: - Green (burn ≤ 1, ≤25% budget used): normal CI/CD — continuous merges, automated canaries, fast rollouts. - Yellow (1 < burn ≤ 4, 25–50% used): moderate restrictions — reduce canary cohort size, increase canary duration (e.g., from 5m→30m), require automated metric checks before ramp. - Orange (4 < burn ≤ 8, 50–100% used): strict restrictions — require manual approvals for deploys, only releases with low risk (patches, config) allowed, mandatory longer canary + staged rollout (1%→10%→100%) with pause points. - Red (budget exhausted, >100%): deployment freeze for new features; only emergency bugfixes allowed with two‑person oncall approval and automated rollback if health degrades.Pipeline reactions (concrete mechanics):- Automated enforcement where possible: - CI/CD polls SLO service/monitoring API for current budget state before allowing pipeline to proceed. - Canary orchestrator enforces cohort size/duration and runs automated health checks (error rate, latency, saturation, business KPIs). If checks fail, immediate automated rollback. - When entering Yellow/Orange, pipeline adjusts by switching to conservative templates (smaller cohorts, longer waits) automatically.- Human in the loop for high‑risk decisions: - Orange requires a designated approver (oncall SRE + release owner) to validate mitigations before proceeding. - Red triggers an automated freeze and an incident-like process: alert the SRE lead, require blameless review, produce a short-term remediation plan before unfreezing.Rationale & tradeoffs:- Automate routine enforcement to reduce toil and latency and ensure consistent policy application.- Keep humans for judgement where context matters (ambiguous root cause, business tradeoffs).- Use dashboards and automated notifications showing current burn rate, projected time to exhaust, and the pipeline state so devs can proactively pace releases.- Keep the policy transparent and reversible: warn teams early (e.g., at 25% and 50% thresholds) and allow expedited appeals with documented rationale.Instrumentation & ops:- SLO exporter + alerting (Grafana/Prometheus), CI/CD plugin to read policy, canary service to enforce gates, and playbooks for manual approvals and emergency rollbacks.- Regularly review SLOs and thresholds with product to balance innovation vs reliability.
MediumTechnical
42 practiced
Describe a set of pipeline-as-code best practices an SRE organization should enforce to improve reliability and security. Cover testing pipeline changes (linting and dry-run), shared templating or libraries, least-privilege for pipeline accounts, and automated validation of pipeline changes before they reach protected branches.
Sample Answer
Start with goals: make pipeline changes safe, auditable, and least-privilege by default. I’d enforce the following concrete practices.1) Testing pipeline changes — linting + dry-run- Require pipeline-config linting in CI using canonical linters (e.g., YAML/schema validators, TFLint for Terraform, Conftest/OPA for policy rules). Run linters on every PR and fail fast.- Provide “dry-run” execution stage that simulates steps (terraform plan, helm template, ci emulator) and produces an artifacts report so reviewers can see intended side effects before apply.Why: catches syntax, policy and semantics problems early, reduces risky merges.2) Shared templating & libraries- Maintain a centrally versioned pipeline template repo (or shared libraries like Jenkins Shared Lib, GitLab CI includes, GitHub Actions reusable workflows). Expose stable, reviewed primitives (build, test, deploy, notifications).- Enforce semantic versioning and require dependabot-style PRs to consume updates.Why: reduces duplication, centralizes security fixes and best-practices.3) Least-privilege pipeline accounts- Use unique short-lived service identities per pipeline with minimal IAM roles (scoped to project/namespace + action). Issue creds via OIDC/STS so no long-lived secrets are stored.- Apply resource-level deny-first policies and require approval for role expansion.Why: limits blast radius if a pipeline is compromised.4) Automated validation before protected branches- Protect main branches; require PRs pass mandatory checks: lint, unit/integration test, dry-run, policy-as-code (OPA/Conftest), SCA for images.- Gate merges on signed commits and enforce branch protection rules (required reviewers, linear history).- Add a “canary merge” workflow: merge to a staging branch automatically and run end-to-end validation before promoting.Why: prevents misconfig changes reaching production and ensures auditability.Operationalize: document standards, run regular audits, rotate templates, and measure MTTR/failed-deploy rates to validate effectiveness.
HardSystem Design
62 practiced
Architect a scalable multi-tenant CI/CD platform for an enterprise running 1,000 microservices and processing 25,000 builds/day across regions. Describe key components (ingestion/scheduler, runner pools, artifact store, caching layer, logging/telemetry), tenancy/isolation model, quota and fair-share, failure domains, monitoring, and cost-control mechanisms.
Sample Answer
Requirements (clarified):- 1,000 microservices, 25k builds/day globally, multi-region, secure multi-tenant isolation, <99.9% platform availability, cost efficient, observable, quota/fair-share.High-level architecture:- API Gateway + Control Plane (ingestion/scheduler) → Global scheduler + regional schedulers → Runner Pools (regional, node-pools) → Artifact Store + Cache CDN → Logging/Telemetry pipeline → Control plane DB (multi-region, strongly consistent metadata).Key components:- Ingestion / Scheduler: Front-end accepts jobs, validates, enforces policies, tags for affinity. Global scheduler routes to regional schedulers using latency/cost/availability. Regional schedulers do bin-packing, preemption policy, and lease runners.- Runner pools: K8s node pools (spot, ondemand, GPU) with autoscaler. Use containerized ephemeral runners launched per-job in sandboxed namespaces (gVisor / kata) and network policies. Use warm container pools for fast start.- Artifact store & cache: S3-compatible object store for immutable artifacts + regional caching edge (CDN or regional cache tier). Use content-addressed storage and deduplication.- Caching layer: Build cache service (remote cache, ccache/Bazel) with sharded key-value stores (Redis cluster + local on-node cache).- Logging/Telemetry: Structured logs + traces shipped via agents (Fluentd) to centralized pipeline (Kafka → processing → long-term storage like OLAP). Metrics to Prometheus long-term in Cortex/Grafana for SLOs.- Metadata DB: Multi-region CockroachDB for job state, quotas, audit.Tenancy & isolation model:- Tenant = org/team. Logical isolation via namespaces, RBAC, network policies. Stronger isolation via dedicated runner pools or vPC per high-security tenants. Multi-tenant storage uses per-tenant prefixes + encryption keys (KMS).Quota & fair-share:- Hierarchical quotas (org/team/service). Token-bucket rate limits at ingress + weighted fair-share scheduler using dominant-resource fairness. Emergency override and SLO-based error-budget borrowing with chargeback.Failure domains & resilience:- Regional schedulers tolerate region loss; global scheduler routes to healthy regions. Runners ephemeral; store artifacts in multi-region replicated buckets. Design so partial region failure degrades throughput not correctness. Use chaos testing and runbooks.Monitoring & SLOs:- SRE defines SLOs (job success rate, queue latency, provision latency). Alerting on symptom-level metrics and error budgets. Dashboards: job throughput, queue sizes, runner utilization, cache hit rates, cost per build.Cost-control:- Mixed instance types, autoscaling, spot-preference with graceful eviction. Cold-start reduction via warm pools. Cache hit optimization reduces build time. Chargebacks and per-tenant quotas; budget alerts; circuit-breaker to throttle low-priority tenants when cost thresholds hit.Security & compliance:- Signed artifacts, supply-chain scanning, least-privilege secrets via Vault, audit logs.Trade-offs:- Stronger isolation (dedicated VMs) increases cost; shared containers increase efficiency. Choose hybrid: most tenants share, high-risk tenants isolated.This design scales horizontally, isolates tenants, enforces fair share, and gives SREs observability and cost controls to operate 25k builds/day across regions.
EasyTechnical
64 practiced
List and explain key metrics to evaluate CI/CD pipeline health from an SRE perspective. Include build success rate, mean time to restore a failed pipeline, average queue time, runner utilization, flakiness rate, and deployment frequency. Suggest sensible alert thresholds and how to convert some into SLOs.
Sample Answer
Key CI/CD health metrics (SRE perspective), why they matter, sensible alert thresholds, and how to convert to SLOs:1) Build Success Rate- What: % of pipeline runs that finish successfully (per repo/team).- Why: Direct indicator of build quality and blocker frequency.- Alert: drop below 95% over a 1h rolling window (critical if <90%).- SLO: 99% successful builds per week for main branches; track error budget for flaky fixes.2) Mean Time to Restore (MTTR) a Failed Pipeline- What: Median time from pipeline failure detection to successful recovery (fix + pass).- Why: Measures operational responsiveness and automation effectiveness.- Alert: MTTR > 30 minutes for prod pipelines.- SLO: MTTR ≤ 15 minutes for release pipelines (monthly target).3) Average Queue Time- What: Time jobs wait before starting (scheduling delay).- Why: Capacity/backlog indicator; affects developer cycle time.- Alert: average queue time > 5 minutes sustained 15m (for CI); >30m for heavy batch builds.- SLO: 95th percentile queue time < 2 minutes for CI jobs.4) Runner/Agent Utilization- What: % utilization of build runners (CPU, concurrent slots).- Why: Capacity planning and cost optimization.- Alert: sustained utilization > 80% or < 20% (starvation vs overprovision).- Use utilization to size autoscaling policies; no direct SLO but tie to availability SLOs.5) Flakiness Rate- What: % of failing tests that pass on re-run (or tests failing intermittently).- Why: Wastes cycles, hides real regressions.- Alert: flakiness rate > 2% on test suites or if a single test flakes >5 times/week.- SLO: Reduce test-suite flakiness to ≤1% per quarter; use error budget to prioritize fixes.6) Deployment Frequency- What: Number of successful deployments to an environment per time window.- Why: Measures delivery velocity and release automation health.- Alert: unexpected drop (>50% vs baseline week-over-week) for release trains.- SLO: e.g., at least 1 production deployment per day for product teams, or 99% of scheduled deploys succeed.Notes on converting to SLOs- Choose user-impacting metrics first (deploy frequency, MTTR, build success for release branches).- Use percentile-based SLOs (e.g., 95th/99th) to avoid noisy outliers.- Define clear measurement windows (daily/weekly/monthly) and succeeded conditions (which branches, job types).- Back SLOs with SLIs, dashboards, and alerting that include context (failed steps, logs, recent changes).- Tie error budgets to remediation actions: if budget burned, freeze feature work until flakiness/build failures are fixed.Practical tips- Segment metrics by pipeline type (PR CI, nightly, release) and critical services.- Instrument events (start/finish/fail/retry) with metadata (branch, author, commit).- Automate reruns for known transient failures and track rerun rate separately.
Unlock Full Question Bank
Get access to hundreds of Continuous Integration and Delivery Pipelines interview questions and detailed answers.
