Build vs. Buy vs. Cloud vs. On Premise Trade Offs Questions
Understanding key trade-offs in technology decision-making: (1) Build vs. Buy - custom development flexibility vs. packaged software speed/cost, (2) Cloud vs. On-Premise - operational burden, control, scalability, security, cost, (3) SaaS vs. Licensed - flexibility, upgrade frequency, customization options. Understanding implications for cost, time-to-value, flexibility, control, and ongoing support.
MediumTechnical
40 practiced
A global retailer reports poor page load times in APAC. Propose architectural options to address latency (cloud CDN, edge compute/edge functions, multi-region deployments, or on-prem point-of-presence) and compare trade-offs between cloud-managed solutions and on-prem PoPs regarding cost, operational complexity, data sovereignty, and deployment speed. Include monitoring and rollout plan.
Sample Answer
Requirements (clarify): reduce APAC page-load latency to <200ms TTFB for 95% requests, preserve data sovereignty (country X), support dynamic personalization, low deployment time (8 weeks), bounded ops team.High-level options:1) Global Cloud CDN + regional POPs (managed CDN like Cloudflare/Akamai/AWS CloudFront)2) Cloud CDN + Edge Functions (Lambda@Edge/Workers) to run personalization and A/B logic at edge3) Multi-region backend deployment (APAC regions in cloud) + regional DB replicas4) On-prem Point-of-Presence (PoP) in APAC (colocated caching + edge servers under company control)5) Hybrid: Managed CDN for static + on-prem PoP or regional cloud for sensitive dynamic dataComparison (cloud-managed CDN vs on‑prem PoP)- Cost: - Cloud CDN: OpEx, predictable pay-as-you-go, lower upfront. Costs scale with traffic and features (edge compute more expensive per request). - On‑prem PoP: High CapEx (hardware, colo, network), fixed recurring colo/network costs; potentially lower long-term per-request cost at very high scale.- Operational complexity: - Cloud: Low ops (provider handles infra, patching), faster iterations. - On‑prem: High ops burden (hardware, redundancy, failover, maintenance), need local engineering presence or partner.- Data sovereignty & compliance: - Cloud: Many providers offer regional data residency options; verify vendor contracts and in-country presence. Might not satisfy strict residency without local infra. - On‑prem: Full control—best for strict sovereignty, auditability.- Deployment speed: - Cloud: Fast (hours–weeks) to enable POPs and edge functions. - On‑prem: Slow (months) for procurement, deployment, and testing.Recommended solution (balanced):- Phase 1 (0–8 weeks): Enable global managed CDN with APAC POPs + enable edge caching rules for static assets, and deploy edge functions for lightweight personalization. Configure origin failover to nearest APAC cloud region. This gives immediate latency wins with low ops.- Phase 2 (8–20 weeks): Deploy multi-region backends in APAC with read-replicas for session/profile data; move sensitive datasets that must remain in-country to regional cloud tenants or dedicated PoP if required by law.- Phase 3 (optional, 20+ weeks): If strict sovereignty or very high predictable traffic warrants it, deploy on‑prem PoP(s) in-country via trusted colo partner; integrate with CDN as origin or ISP peering.Monitoring and rollout plan:- Monitoring stack: Real User Monitoring (RUM) + synthetic checks (global probes) for TTFB, LCP, error rates; CDN provider metrics (cache hit ratio, edge compute latency); backend metrics (p95 API latency, DB replication lag); logging pipeline (centralized ELK/Datadog) with alerting on SLO breaches.- KPIs: 95th-percentile TTFB, cache hit ratio >85%, API p95 <150ms, error rate <0.5%.- Rollout strategy: Canary by geography and user segment—start with 5% APAC traffic routed via CDN + edge functions; monitor KPIs 24–72h; expand 25% → 50% → 100% if stable. For backend changes, use blue/green deployments with health checks and DB replica promotable pipelines.- Rollback plan: automated routing rollback via DNS/CDN config and feature flags; circuit-breaker for edge functions to fall back to origin.Trade-offs summary: start with cloud-managed CDN + edge compute for fastest impact and lowest ops. Use multi-region backends for dynamic content. Only invest in on‑prem PoPs if required by compliance or when long-term TCO and traffic justify the operational overhead.
HardTechnical
64 practiced
Describe a staged migration strategy from an on-prem monolith to a cloud-native microservices architecture. Cover cutover approaches (strangler pattern vs big-bang vs parallel run), data migration techniques (dual writes, change data capture), testing strategies, rollback mechanisms, and change management risks. Recommend an approach for an e-commerce application with high availability and low-latency requirements.
Sample Answer
Clarify scope/constraints first: acceptable downtime, SLA targets (99.99?), peak traffic profiles, data consistency needs (strong vs eventual), team readiness.Cutover approaches- Big‑bang: switch traffic once everything migrated. Fastest end state but high risk; unacceptable for high‑availability e‑commerce.- Parallel run: run both systems concurrently for a period; safer but expensive and complex for reconciliation.- Strangler pattern: incrementally route functionality to new microservices behind API gateways or feature flags. Lowest risk and allows iterative verification; best for complex, high‑traffic systems.Data migration techniques- Dual writes: app writes to both old DB and new services. Simple but risks divergence and increased latency; requires idempotency and retry logic.- Change Data Capture (CDC): capture DB binlog/transaction stream to propagate changes to new data stores (Debezium, AWS DMS). Enables near real‑time sync with low app impact.- Backfill + rollforward: bulk migrate historical data, then use CDC for deltas.Testing strategies- Contract and consumer‑driven contract testing to avoid regressions.- Canary releases and progressive traffic shifting (5→20→100%) with metrics gating (latency, error rate, business KPIs).- Chaos testing and resiliency tests for downstream failures.- Data validation: reconciliation jobs comparing counts/hash checksums; shadow queries to validate responses.- Performance testing at peak loads including end‑to‑end latency and payment/checkout flows.Rollback mechanisms- Feature flags and traffic routing allow instant rollback by re-pointing to monolith.- Maintain bidirectional compatibility in APIs during migration window.- For data: write-through compensation, tombstones, and compensating transactions; keep CDC consumers idempotent.- Maintain an “audit log” and snapshot before major state changes.Change management risks & mitigations- Data divergence: mitigate with CDC + reconciliation and alerts.- Latency regressions: measure p99/p999; colocate services and use caching/CDNs; optimize network paths.- Operational complexity: adopt SRE runbooks, runbooks for cutover/rollback, run automated health checks.- Organizational: ensure domain ownership, training, and phased team onboarding; prioritize business‑critical paths (checkout, inventory).Recommendation for e‑commerce (HA, low latency)- Use the strangler pattern with domain decomposition around critical flows (checkout, cart, catalog).- Start with read‑heavy services (catalog) using CDC + read replicas and caches (Redis, CDN). Then move payment/checkout last with strict contract tests and staged canaries.- Employ API Gateway, traffic shifting, and feature flags. Use CDC for data sync, dual reads temporarily for validation, and reconciliation jobs. Enforce SLOs with observability (distributed tracing, p99/p999 metrics) and automated rollback gates.This balances risk, preserves availability, and enables iterative validation while meeting low‑latency requirements.
MediumTechnical
48 practiced
For a hybrid service where the control plane runs in cloud and user data is stored on-premise, propose an observability strategy. What should you monitor (metrics, logs, traces), how would you instrument services across environments, what data collection architecture would you choose (centralized vs federated), and how would you set SLOs that reflect both cloud and on-prem availability?
Sample Answer
High-level approach: treat observability as end-to-end (control plane + on‑prem data plane) so SREs and architects can reason about user-visible outcomes and root cause (cloud vs on‑prem). Use standard, portable instrumentation (OpenTelemetry) and a hybrid collection architecture that balances central visibility with on‑prem data residency.What to monitor- Metrics (high-cardinality where needed): - Control plane: API latency/percentiles, 5xx/4xx rates, auth failures, config‑push latency, control‑plane CPU/memory. - Data plane (on‑prem): read/write success rate, IOPS/latency, replication/sync lag, storage capacity, network RTT to cloud, local resource saturation. - Platform: DNS/DNS resolution, TLS handshake times, connection churn, queue lengths.- Logs: - Structured logs for control and data plane (correlate request IDs). Capture warnings/errors, config changes, authentication/audit logs (respecting PII).- Traces: - Distributed traces across control→data calls. Propagate trace/request IDs through control plane calls, agent-to-cloud RPCs, and local storage operations.Instrumentation strategy- Use OpenTelemetry SDK (or vendor equivalent) for metrics/traces/log context propagation.- Lightweight on‑prem agent collector (e.g., OpenTelemetry Collector, Vector, Fluent Bit) that: - Exposes local Prometheus metrics for scraping. - Buffers and batches data, enforces sampling/PII redaction, and ships over secure channel.- Inject correlation IDs at ingress (API gateway) and surface them in logs and traces.- Health-check endpoints and readiness/liveness for both control and on‑prem components.Data collection architecture: hybrid/federated- Federated collection with central aggregation: - On‑prem collectors perform local scraping, pre-aggregation, sampling, and data filtering to meet residency and bandwidth constraints. - Secure relay (TLS, mTLS, auth tokens) to central observability backend (multi-tenant) for global dashboards, SLO calculation, and retrospective analysis. - Use federated Prometheus model (Prometheus + Thanos/Cortex) or push model (Cortex/SignalFX) depending on scale. - Logs: local retention for audit-sensitive logs; centralize operational logs (redacted) to ELK/Tempo/Lightstep for correlation. - Traces: sample high-value traces and send to central tracing (Jaeger/Tempo) with local short-term retention.Why federated: preserves on‑prem control/residency and reduces network/bandwidth, while enabling global correlation and multi-site troubleshooting.SLO strategy (cloud + on‑prem)- Define user‑visible SLOs first (end‑to‑end), then component SLOs: - Example end-to-end SLO: “99.9% of read/write requests succeed within 200ms over 30 days” — measured from client perspective through control plane and on‑prem. - Component SLOs: - Control plane API availability: 99.95% (measured via synthetic checks and real traffic). - On‑prem data availability: 99.9% (measured from local health + telemetry).- Composite SLOs: - Calculate end‑to‑end as function of component SLOs with weighted impact (e.g., if control-plane failure causes full outage, it dominates). - Maintain separate error budgets and runbooks per component; use composite SLOs for customer-facing reporting.- Alerts and escalation: - Use symptom-based alerts on user‑visible SLO burn (e.g., 10% error budget consumed in last 7 days), plus KPI alerts on leading indicators (replication lag > threshold, network RTT increase). - Provide automated degraded-mode guidance (e.g., switch to local-only operations) and circuit-breakers.- Measurement windows: - Use rolling 30-day windows for business SLOs, tighter windows for operational alerts (5m/1h).- Compliance/data residency: - For PII/audit logs keep local-only storage and expose aggregated metrics centrally.Operational practices- Synthetic checks from multiple regions and on‑prem agents to validate connectivity and latency.- Blameless postmortems, playbooks for on‑prem vs cloud failures, and runbook automation to remediate (restart agent, flush buffer, fallback to local-only).- Regular SLO reviews with customers; expose transparent incident metrics.Tradeoffs- Centralized-only gives simpler correlation but can violate residency and increase bandwidth.- Fully federated simplifies compliance but makes global correlation harder; hybrid/federated is the pragmatic compromise.This design gives end‑to‑end visibility, respects on‑prem constraints, and ties technical metrics to customer‑facing SLOs with clear alerting and remediation.
MediumTechnical
41 practiced
Explain vendor lock-in in cloud contexts. Provide six practical indicators that a project is locked into a specific cloud provider (for example heavy use of proprietary managed services) and four mitigation strategies a Solutions Architect can propose during design to reduce lock-in risk while keeping operational efficiency.
Sample Answer
Vendor lock-in in cloud contexts is when an application or organization becomes dependent on a single cloud provider’s proprietary APIs, services, tooling, or operational patterns such that moving away becomes costly, risky, or impractical. As a Solutions Architect I weigh business benefits of managed services against portability risks.Six practical indicators of cloud lock-in- Heavy use of proprietary managed services (e.g., DynamoDB, BigQuery, Aurora) for core functionality.- Reliance on provider-specific identity, networking or security primitives (IAM roles, VPC peering, provider-only WAF features).- Data formats or storage tied to provider tooling (proprietary export formats, opaque backup snapshots).- Use of provider-specific deployment/orchestration (CloudFormation, Azure Resource Manager templates) without abstractions.- Operational runbooks and tooling built around provider consoles, CLIs, or monitoring (CloudWatch-specific alarms, logs).- Significant investment in provider-specific serverless functions, event buses, or proprietary middleware.Four mitigation strategies to reduce lock-in while keeping efficiency- Design with abstraction: use cloud-agnostic interfaces (Kubernetes, Terraform, standard SQL) and an adapter layer so business logic talks to interfaces, not vendor APIs.- Use managed services selectively: prefer managed services for nondifferentiating capabilities but choose portable standards where vendor lock-in is risky (e.g., managed PostgreSQL vs. proprietary DB).- Data portability and exportability: store data in open formats, maintain automated backup/export pipelines, and define SLAs for export time and format.- Infrastructure as Code + CI/CD with provider-agnostic tooling: use Terraform modules, Helm charts, and CI pipelines that can target multiple providers; codify provider-specific differences in small modules.Trade-offs: full avoidance reduces some operational efficiency; propose risk-based approach—use provider-managed services when they deliver clear business value and mitigate portability with the above patterns.
MediumSystem Design
46 practiced
Design a CI/CD pipeline that can deploy to both cloud environments and on-premise targets, including Kubernetes clusters and VMs. Specify recommended tooling choices for orchestration, artifact storage, secrets management, deployment strategies (blue/green, canary), and approaches to detect and manage environment drift between cloud and on-prem deployments.
Sample Answer
Requirements (clarify): deploy CI-built artifacts to cloud and on‑prem; targets include Kubernetes clusters and VMs; secure artifacts & secrets; support progressive rollout patterns; detect/manage config drift; integrate with existing identity and networking.High-level architecture:- CI (build & test) → Artifact Registry → CD orchestrator → Target adapters (K8s clusters, VM agents) → Observability & drift detection → Policy & secrets services.Recommended tooling (practical, vendor-neutral):- CI orchestration: GitHub Actions / GitLab CI / Jenkins X / Tekton (Kubernetes-native). Choose Tekton/GitHub Actions for cloud-native pipelines.- CD orchestration: ArgoCD (GitOps) for Kubernetes + Spinnaker for hybrid deployments (K8s + VMs) OR use ArgoCD + Ansible/HashiCorp Nomad adapters for VMs.- Artifact storage: Harbor / JFrog Artifactory / AWS ECR / GCR for container images; Nexus or S3-backed storage for non-container artifacts.- Secrets management: HashiCorp Vault (on-prem + cloud), integrated with cloud KMS (AWS KMS/Azure Key Vault/GCP KMS) for encryption at rest and auto-unseal. Use CSI secrets driver for K8s and Vault agents or Vault SSH/PKI for VMs.- Infrastructure as Code (IaC): Terraform for cloud and on-prem provisioning; Packer for golden VM images; Ansible for configuration management on VMs.- Progressive rollout: Argo Rollouts or Istio/Linkerd + Envoy for canary; Spinnaker supports blue/green and canary for mixed targets. Feature flags: LaunchDarkly or open-source Unleash for application-level control.- Policy & security: OPA/Gatekeeper for K8s admission policies; Snyk/Trivy for image scanning integrated into CI.- Observability & drift detection: Prometheus + Grafana + Loki for logs; kube-state-metrics and metrics exporters; use Terraform Cloud/Enterprise remote state and drift checks; Cloud Custodian for cloud resource policies; reconcile loops (ArgoCD) for K8s config drift; periodic agent-based auditors (OSConfig, puppet/chef reports) for VMs.Deployment flow (concrete):1. Developer pushes branch → CI builds, runs tests, produces immutable artifacts (container images with semantic tags + checksumed artifacts), signs artifacts.2. CI pushes image to Harbor/ECR and publishes metadata to Git (image tag in manifest).3. GitOps repo updated (automatically or via PR) with new manifest.4. ArgoCD watches repo and applies to K8s clusters; for VMs, Spinnaker or an Ansible job pulls new artifacts and performs deployment using systemd/docker-compose or Kubernetes-in-docker if applicable.5. Use Argo Rollouts/Service Mesh for canary: shift percentage traffic, monitor health and business metrics, automatic promotion or rollback.6. For blue/green on VMs, orchestrate traffic switch at load balancer (HAProxy/NGINX/Cloud LB) with health checks and automated cutover.Secrets & identity:- Centralize secrets in Vault; use short-lived credentials (vault-agent, Kubernetes auth, AppRole for VMs).- Integrate CI with OIDC (GitHub Actions/GKE/IAM) to avoid long-lived CI secrets.- RBAC enforced via cloud IAM + K8s RBAC; require MFA and least privilege.Drift detection & remediation:- K8s: ArgoCD continuous reconciliation; enable resource health checks and alerts; use Gatekeeper to prevent unauthorized changes.- IaC drift: Maintain all infra in Terraform; enable periodic plan runs against remote state; alert on diffs; optionally run automated apply in safe modes or open PRs for operator approval.- VM config drift: Use configuration management (Ansible/Puppet) with periodic "converge" runs and reporting; integrate OS-level agents to report package/config drift.- Cloud resource drift: Cloud Custodian + Terraform state drift checks + AWS Config / Azure Policy for compliance detection.- Process: classify drift into auto-remediate (security patches, self-healing) vs. require human (schema changes). Record all auto-remediations in GitOps commits where possible to keep source-of-truth.Trade-offs and considerations:- GitOps (ArgoCD) is ideal for K8s declarative deployments and drift remediation; for VMs, imperative tools (Ansible/Spinnaker) are stronger—use a hybrid approach that keeps declarations in Git to maintain auditability.- Spinnaker handles multi-target complex deployments well but has operational overhead; consider managed offerings or ArgoCD + Argo Workflows + Ansible for lighter footprint.- Secrets: Vault gives portability but increases operational complexity vs. using cloud KMS (simpler but cloud-bound).- Network: ensure secure connectivity for on‑prem agents (mTLS, private links, VPN) and obey egress policies.Operational best practices:- Enforce immutable artifacts, image signing (cosign), SBOM generation.- Automated security scanning during CI and gating CD.- Canary analysis driven by automated metrics (SLOs) with rollback thresholds.- Keep a single GitOps source-of-truth for deployments; every change must be traceable to commits/PRs.- Regularly run chaos/drift drills and rehearse rollback and remediation.This design gives a repeatable, auditable CI/CD pipeline that supports Kubernetes and VM targets across cloud and on‑prem while providing strong security, progressive rollouts, and automated drift detection/remediation.
Unlock Full Question Bank
Get access to hundreds of Build vs. Buy vs. Cloud vs. On Premise Trade Offs interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.