Deployment Orchestration and Cloud Platforms Questions

Design deployment architecture and orchestration for applications and infrastructure on cloud platforms. Topics include containerization and container orchestration systems, how orchestration and platform services affect deployment strategy, operational considerations for scaling and reliability, integration of monitoring and logging, and decision criteria for when orchestration adds value. Candidates should be able to describe deployment models for monoliths and microservices, patterns for scaling and resilience, and cloud native considerations such as managed services, auto scaling policies, and environment separation.

EasyTechnical

86 practiced

Describe a minimal logging pipeline for cloud-native applications: how logs travel from application container to long-term storage and query. Include components such as log shippers, aggregators, indexers, alerting integration, and managed service alternatives.

MediumTechnical

81 practiced

How would you implement blue-green or canary deployments using managed services (for example AWS ALB, Cloud Run, Azure App Service) without operating your own orchestrator? Provide concrete steps for traffic shifting and verification.

Sample Answer

Start by clarifying goal: zero-downtime deploy with controlled traffic shift and easy rollback using managed platform features (no self-managed orchestrator). General pattern: provision parallel environments, shift traffic incrementally, verify, then cut over or rollback.

Concrete steps (ALB / Cloud Run / Azure App Service):

1. Provision versions- ALB: Deploy v1 and v2 as separate target groups (or ECS tasks/ASGs for each version).- Cloud Run: Deploy separate services or revisions (service A = v1, service B = v2).- Azure App Service: Use Deployment Slots (production slot = v1, staging slot = v2).

2. Configure routing for gradual shift- ALB: Use weighted target group routing via AWS Application Load Balancer + Listener Rules or Route 53 weighted records in front of ALB to split traffic (e.g., 1%, 5%, 25%, 100%).- Cloud Run: Use Traffic Splitting on service revisions (console/CLI) to assign weights.- Azure App Service: Swap slots or use Traffic Routing (percentage-based) to route X% to staging.

3. Verification at each step- Health checks: ensure platform-level health is green.- Synthetic tests: run smoke tests (endpoint checks, auth flows).- Canary metrics: monitor error rate, latency (p95/p99), saturation, business metrics (conversion, orders) for both versions.- Logging/tracing: ensure distributed traces tag by revision and check for regressions.

4. Shift plan and automation- Start 1% for 10–30 minutes, then 5%, 25%, 50%, 100; or use automated canary tools (AWS CodeDeploy AppSpec with Lambda hooks, or Cloud Run gradual traffic API).- Automate rollback if error rate > threshold (e.g., >3x baseline) or latency increases by X%.

5. Rollback and cutover- If metrics fail: immediately set weights back to 100% v1 and drain connections for v2; in ALB, deregister targets gracefully; in Cloud Run, set traffic back to revision1; in Azure, swap back or set routing to 0% for staging.- If stable: ramp to 100%, decommission v1 or keep for quick rollback.

Trade-offs & considerations- DNS-based splits (Route 53) add DNS TTL delays; prefer ALB/Cloud Run/Azure slot native splitting for instant changes.- Stateful sessions: use sticky sessions carefully—prefer stateless deployments or shared session store.- DB migrations: use backward-compatible migrations or feature flags to avoid coupling traffic shift to schema changes.- Cost: running two versions doubles compute during rollout.

This approach uses native managed capabilities for safe, observable deployment without owning an orchestrator, with clear verification and automated rollback criteria.

MediumSystem Design

73 practiced

Design an artifact promotion model ensuring immutability and traceability as artifacts move from CI to production. Include artifact signing, metadata storage, provenance, and how you would handle emergency hotfixes without compromising traceability.

Sample Answer

Requirements & constraints:- Immutability: once an artifact is produced, its binary must be immutable.- Traceability/provenance: every stage (build, test, promote) must record who/what/when/why.- Signing: artifacts must be cryptographically signed at build and at promotion boundaries.- Emergency hotfix: allow fast fixes while preserving audit trail.

High-level architecture:CI Build System → Artifact Registry (immutable blobs + content-addressable storage) → Metadata & Provenance DB → Promotion Service → Deployment/CD → Audit/Forensics Store

Key components:1. Artifact Registry (CAS): store artifacts by digest (SHA-256). Objects are immutable; uploads are append-only.2. Build Signer: CI agent signs the digest with build key (e.g., PGP or sigstore/fulcio + Rekor). Signature stored alongside artifact.3. Metadata & Provenance DB: immutable event log (append-only ledger, e.g., PostgreSQL with WAL and/or an append-only store like Kafka or blockchain-style hashes). Records build_id, commit, builder, dependencies, SBOM, signature refs, test results, timestamps.4. Promotion Service: verifies signatures, policies (QA passed, vuln scan), issues a promotion record (signed) that references artifact digest and environment (staging, canary, prod). Promotion records are immutable.5. Key Management: KMS + hardware-backed keys; rotate keys with key-versioning; record which key signed which artifact/promotion.6. Audit/Forensics: read-only snapshots, Rekor transparency log or equivalent for public verifiability.

Data flow:- CI builds produce artifact -> compute SHA-256 -> store in CAS -> Build Signer signs digest and writes to Rekor + Metadata DB including SBOM and test artifacts.- Promotion: operator/system requests promotion -> Promotion Service validates build signatures, policies -> writes a signed promotion record to Metadata DB and Rekor -> deployment reads artifact by digest.

Emergency hotfix workflow:- Hotfix branch build follows same build pipeline and signs artifact. To expedite, allow a fast-track policy that lowers gating but still requires signature and creates an "emergency-promotion" record including justification, approver identity, and TTL for post-mortem. All fields are mandatory; Promotion Service enforces logging. After deployment, run a retrospective automation to attach extra evidence (post-deploy tests, code review comments) to the promotion record. Never allow unsigned or re-signed binaries without generating a new digest and signed promotion record.

Security & integrity controls:- Enforce content-addressable immutability; registry rejects any attempt to overwrite digest.- Use transparency log (Rekor) for non-repudiation.- KMS per-environment keys; use ephemeral delegated signing tokens for CI to avoid long-lived keys on agents.- RBAC and approvals recorded in provenance.

Scalability & availability:- CAS backed by S3-compatible storage with lifecycle policies.- Metadata stored in DB sharded by project; write-ahead event log for audit.- Promotion Service horizontally stateless behind load balancer.

Trade-offs:- Storing full artifacts in provenance DB is heavy — keep references (digests, SBOM URI) instead.- Rekor adds complexity but provides public/verifiable audit.- Fast-track hotfixes trade rigor for speed; mitigated by mandatory logging and post-facto evidence.

This design ensures every artifact and promotion is cryptographically linked, immutable, and traceable while enabling controlled emergency fixes that preserve a complete audit trail.

MediumSystem Design

74 practiced

Propose an integrated monitoring and logging architecture for a cloud platform hosting ~200 microservices. Cover metrics, logs, traces, alerting strategy, retention tiers, cost controls, and decisions between managed vs self-hosted components.

Sample Answer

Requirements (assumptions): ~200 microservices across multiple k8s clusters, 99.9% availability target, multi-tenant teams, cost sensitivity, compliance for logs for 90 days.

High-level architecture:- Instrumentation: OpenTelemetry SDKs for traces/metrics, standardized semantic conventions, sidecar or daemonset collectors.- Metrics: Prometheus for pull scraping of k8s/ app metrics → Prometheus Thanos (or Cortex) for long-term, global view and HA. Use PromQL for dashboards/alerts.- Logs: Structured JSON logs shipped by Fluentd/Fluent Bit to a log platform. Options: managed (Datadog/LogDNA/Elastic Cloud) or self-hosted ELK/Opensearch on EKS with S3 cold storage.- Traces: OTLP -> OpenTelemetry Collector -> managed tracing (Lightstep/Datadog/Tempo/Jaeger backend + cortex/tempo for scaling). Correlate trace IDs in logs/metrics.

Data flow: apps -> OTEL collector (local) -> metric backend (Prometheus/Thanos) + log sink + trace backend. UI/dashboard: Grafana with unified datasource plugins.

Alerting & SLO strategy:- Define SLIs (latency, error-rate, availability) and SLOs per service.- Alerting tiers: page (P1/P0) for SLO burn rate and on-call; email/Slack for ops; automated remediation runbooks.- Use alert suppression, grouping, and threshold based on burn rates to avoid alert fatigue.

Retention tiers & cost controls:- Metrics: high-cardinality short-term (15d) in Prometheus; aggregated/rollups in Thanos for 1y. Configure label cardinality limits.- Logs: hot (7–14d) indexed for search; warm (30–90d) less-indexed, stored in compressed S3; cold/archival in cheap object storage for compliance.- Traces: sample 100% in dev/staging; prod 5–20% adaptive sampling + store error/slow traces 100%. Retain full traces for 7–30 days; aggregated trace metrics longer.

Cost control practices:- Use sampling, metric aggregation (histogram buckets → summaries), and cardinality caps.- Enforce logging libraries to avoid unbounded keys; log-level gating (info/error); client-side filtering.- Use multi-tenant quotas, rate limits, and chargeback for teams.- Archive to S3 with lifecycle policies; spot instances for self-hosted components.

Managed vs self-hosted trade-offs:- Managed (Datadog, New Relic, Lightstep, Elastic Cloud): faster time-to-value, less ops, built-in UIs/alerting, predictable scaling — higher recurring cost, vendor lock-in, limited customization.- Self-hosted (Prometheus + Thanos/Cortex, OpenSearch, Tempo/Jaeger): lower long-term cost at scale, full control, customizable retention and storage (S3), but requires SRE ops, upgrades, HA design.- Recommendation: hybrid: Managed SaaS for traces and centralized logs for faster onboarding and powerful UX; self-hosted Prometheus+Thanos for metrics (to control cardinality & long-term retention) OR use managed Cortex. Start small with managed to accelerate proofs and migrate high-volume components to self-hosted as scale demands.

Operational considerations:- Security: TLS, encryption-at-rest, RBAC, network policies.- Observability pipeline resilience: buffering, backpressure, retry policies.- Runbooks, playbooks, and on-call rotations; periodic SLO review and cost audits.- Pilot: onboard 10 services, validate sampling/retention, then roll out.

MediumTechnical

79 practiced

You must recommend an Infrastructure-as-Code strategy and workflow across multiple cloud accounts. Choose tooling (e.g., Terraform, CloudFormation, Pulumi), explain state management, drift detection, plan/apply workflows, and how you would enforce guardrails (policy-as-code) and remediate drift.

Sample Answer

Recommendation: use Terraform as primary IaC for multi-cloud because of provider breadth, mature ecosystem, and strong CI/GitOps integrations. Use CloudFormation for AWS-only teams that prefer native templates, and Pulumi when engineering wants full-language constructs.

State management:- Remote, encrypted state per workspace (one workspace per account/environment/region) using Terraform Cloud/Enterprise or S3 + DynamoDB lock (server-side SSE, KMS). Use workspaces to isolate drift blast radius and RBAC to control who can read/modify state.- Keep sensitive outputs encrypted; enable state versioning and retention for audit.

Drift detection:- Continuous: run scheduled terraform plan across workspaces (CI job or Terraform Cloud run triggers).- Complement with cloud-native tools (AWS Config, Azure Policy) and open-source driftctl to detect resources created/changed outside IaC.- Alert when plan shows changes; classify as safe/non-safe via policy.

Plan/apply workflow:- GitOps: All changes via pull requests to repo per module/environment.- CI steps: 1) fmt/validate/static checks; 2) terraform init + plan (using workspace credentials) and post plan as PR comment; 3) policy checks; 4) manual or automated approval -> terraform apply executed by CI runner or Terraform Cloud with privileged token.- Use ephemeral short-lived credentials for pipelines; enforce least privilege.

Policy-as-code & guardrails:- Enforce policies pre-merge and pre-apply using Rego/OPA (Conftest), Terraform Cloud Sentinel, or Checkov. Examples: deny public S3, enforce tagging, enforce approved AMIs, cost limits.- Enforce at cloud control plane too: AWS SCPs, Azure Management Groups, GCP Organization Policies to prevent circumvention.

Drift remediation:- For low-risk drifts, run automated reconciliation: scheduled terraform plan + auto-apply for non-breaking drifts with audit logging.- For risky drifts, create incident/ticket and require owner approval; optionally run remediation playbooks (Runbooks + automation account) that call IaC apply.- Maintain audit trail: who changed what, when, and why (link PR or incident).

Operational practices:- Modularize modules, version modules in registries, run regression tests (unit + integration with Terratest).- Provide onboarding docs, example pipelines, and a “secure defaults” module.- Monitor cost and resource limits; run periodic policy reviews.

This balances developer velocity (GitOps + Terraform), governance (policy-as-code + cloud policies), and reliable drift detection/remediation (scheduled plans, native config rules, automated safe applies).

Unlock Full Question Bank

Get access to hundreds of Deployment Orchestration and Cloud Platforms interview questions and detailed answers.

Join thousands of developers preparing for their dream job.