Cloud and Infrastructure Architecture Questions
Design of cloud and infrastructure solutions from simple cloud applications to end to end cloud systems and large scale infrastructure. Topics include stateless versus stateful design, VPC and networking basics, security groups and firewalls, multi environment management, deployment topology, capacity planning, global distribution, cloud native services, infrastructure as code, and operational concerns for sustaining large scale cloud environments.
HardSystem Design
57 practiced
Design a multi-tenant Kubernetes architecture to host a SaaS platform for 1,000 tenants. Tenants must be isolated, billing must be tracked per tenant, and the platform should remain cost efficient. Discuss trade-offs between per-tenant clusters and shared clusters, namespace strategies, RBAC, network policies, resource quotas, admission controllers, tenant onboarding automation, and telemetry for billing.
Sample Answer
Requirements (clarify): 1,000 tenants, strong isolation, per-tenant billing, cost efficiency, operational simplicity, reasonable blast radius, support onboarding automation, observability.High-level approach: hybrid model — mix shared multi-tenant clusters (most tenants) + per-tenant clusters for high-risk / high-compliance customers. Use namespace-per-tenant within shared clusters, with strict RBAC, NetworkPolicies, ResourceQuotas, and AdmissionControllers to enforce isolation and limits. Reserve a pool of dedicated clusters (or node pools) for tenants requiring dedicated compute or compliance.Core components:- Control plane: managed Kubernetes (EKS/GKE/AKS) to reduce ops overhead.- Cluster topology: N shared clusters (e.g., 10–20) each hosting ~50–100 namespaces to limit blast radius and keep API / etcd healthy; autoscaled node pools by workload class (cpu/memory/spot).- Namespaces: one namespace per tenant (or per-tenant-per-env) with standardized labels/annotations for billing (tenant-id, plan).- RBAC: least-privilege Roles scoped to namespace; use OIDC + IAM integration for platform operators. Use Kubernetes PodSecurity (or OPA Gatekeeper) to deny hostNetwork/hostPath and privilege escalation.- Network isolation: NetworkPolicies default deny for ingress/egress; CNI that supports namespace isolation (Calico/Canal) and egress NAT per-tenant if needed.- ResourceQuotas & LimitRanges: enforce CPU/memory/pod counts per namespace; implement ClusterResourceQuota to enforce aggregated limits per tenant across namespaces if using shared clusters.- Admission controllers: OPA Gatekeeper for policy enforcement (labeling, image registry allowlist, cost-control annotations), ValidatingAdmissionWebhook to inject sidecars (billing metering) and enforce quotas on ephemeral storage.- Billing & telemetry: instrument resource usage via Prometheus metrics + Kube-state-metrics and node exporter, collect per-tenant metrics using labels; use Thanos/ Cortex for long-term storage. For accurate billing include: - Pod/container CPU/Memory usage sampling and conversion to cost with price model (node-type price + overhead). - PersistentVolume usage tracked via CSI metrics. - Network egress tracked at CNI/cloud LB logs. - Export to billing pipeline (Kafka -> processing -> billing DB) with tenant-id tags; support per-minute aggregation.- Onboarding automation: self-service API/portal backed by an operator (CRD Tenant) that: - Creates namespace(s), RBAC bindings, NetworkPolicies, ResourceQuotas, default ConfigMaps/Secrets from templates. - Annotates resources with tenant-id and plan. - Provisions dedicated cluster or upgraded resources if plan requires. - Runs IaC pipelines (ArgoCD) to deploy tenant bootstrap configuration and enforce drift.- Cost optimization: - Use shared clusters for most tenants on burstable node pools and spot/preemptible instances with fallbacks. - Right-size resource quotas and offer plans that encourage efficient usage. - Consolidate low-traffic tenants onto fewer nodes via bin-packing; use Vertical/Horizontal Pod Autoscalers.- Security & compliance: encrypt secrets (KMS), use pod security policies/policies via Gatekeeper, audit logs forwarded to centralized SIEM per-tenant filtering when required.- Operational considerations: - Scale: limit namespaces per cluster to keep etcd/api latency acceptable; scale clusters horizontally. - Upgrades: use rolling upgrades per cluster group; maintain cluster image/version consistency. - Incident isolation: cluster-level boundaries for blast radius and easier tenant eviction for abuse.Trade-offs:- Per-tenant clusters: strongest isolation, easiest billing attribution, but high cost/operational overhead at 1,000 tenants — impractical except for few high-value tenants.- Namespace-per-tenant in shared clusters: cost-efficient, manageable, but requires strong policy enforcement (RBAC, NetworkPolicy, Admission) to avoid noisy neighbors and cross-tenant leakage.- Hybrid approach balances cost and isolation: dedicate clusters only for compliance or heavy tenants.Why this design: it gives secure isolation for most workloads with cost control and automation to scale to 1,000 tenants, while retaining the option to provide dedicated clusters when required by compliance or SLAs.
HardTechnical
50 practiced
Design a hybrid cloud architecture for a client with large datasets that cannot be moved fully to public cloud due to data gravity. They need near real-time analytics on a subset of data in the cloud and low-latency local processing on-prem. Describe connectivity, streaming replication versus batch ETL, edge processing patterns, orchestration, security, and how to centralize control while minimizing data movement.
Sample Answer
Requirements clarification:- Large on-prem datasets remain on-prem (data gravity).- Near real-time analytics on a subset in public cloud.- Low-latency local processing for operational workloads.- Centralized control, strong security, minimal data movement.High-level architecture:- On-prem: primary data lake (HDFS/NAS or object store), local stream processors (Kafka/FS/CDC), edge compute nodes for low-latency processing.- Cloud: analytics cluster (Kubernetes + Spark/Beam/Flink), BI/ML workbench, object storage for ingested subset.- Connectivity: dedicated private link (AWS Direct Connect/Azure ExpressRoute/GCP Interconnect) with VPN fallback; use SD-WAN for multi-site connectivity and path-aware routing.Data movement strategy:- Streaming replication for hot subsets: use CDC (Debezium) or change-streams to publish events to a durable message bus (Kafka MirrorMaker/Confluent Replicator) that forwards a filtered topic to cloud via secure connector. Enables near real-time analytics with minimal footprint.- Batch ETL for cold/large objects: periodic, compressed snapshots (Parquet/ORC) transferred via multipart upload, validated by checksums and manifest files.- Minimize movement by pushing compute to data: run federated queries (Presto/Trino) or pushdown workloads to on-prem where possible; only materialize cloud datasets when needed.Edge processing patterns:- Inline low-latency processing at edge: lightweight containers (k3s) running inference or stream enrichment, outputting metadata/events to local topics.- Aggregation at edge gateway: pre-aggregate, sample, or anonymize before replication.- Canary/feature rollouts mirrored in cloud for model training.Orchestration and control:- Use a central orchestration plane (Airflow/Airbyte/Argo Workflows) deployed in cloud with agents on-prem to schedule, monitor, and trigger pipelines.- Metadata/catalog (Glue/Atlas) federated across environments to maintain data lineage.- Observability: unified logging/metrics (Prometheus + remote-write, ELK with forwarding), SLAs and alerting.Security & compliance:- Zero-trust network model: mTLS between components, per-topic ACLs in Kafka, VPC endpoints for storage, strict IAM roles.- Data classification and policy engine: enforce encryption-at-rest (KMS/HSM) and in-transit (TLS 1.2+), tokenization/field-level encryption for PII before crossing boundary.- Audit logging, SIEM integration, and periodic key rotation. Use DLP to prevent inadvertent transfer.Trade-offs and rationale:- Streaming gives sub-second to seconds freshness but requires operational complexity and network stability. Batch lowers cost and suits large cold data.- Federated query reduces movement at cost of query latency and cross-site bandwidth usage.- Centralized control plane simplifies governance but requires reliable secure connectivity and well-designed agent model.Implementation checklist:- Proof-of-concept: mirror a small hot-topic via CDC → Kafka → cloud analytics.- Define SLOs for latency, throughput, and cost models.- Harden security (pen tests, IAM reviews) before production roll-out.This design balances low-latency local processing, near real-time cloud analytics, centralized control, and minimal bulk data movement by combining CDC/streaming for hot subsets, batch transfer for cold data, edge aggregation, and a federated orchestration and security model.
MediumTechnical
50 practiced
Design network microsegmentation for a multi-tier cloud application using cloud security groups and Kubernetes network policies. Explain how to enforce least privilege between front-end, app, and DB tiers, manage rule lifecycle and auditing, and avoid rule explosion while meeting compliance needs.
Sample Answer
Situation / intent: Design microsegmentation for a 3‑tier cloud app (frontend → app → db) that enforces least privilege using cloud Security Groups (SGs) and Kubernetes NetworkPolicies, while keeping rules manageable and auditable for compliance.Architecture summary:- Layer 1 (cloud): Use cloud SGs / NACLs as coarse perimeter control (internet-facing ALB → frontend SG; app SG; db SG). SGs restrict IP ranges and protocols to only what's needed (ALB → frontend: 443; frontend→app: app-range ports; app→db: DB port).- Layer 2 (Kubernetes): Use default-deny NetworkPolicies at namespace level, then fine-grained allow policies using label selectors (namespace: frontend, app, db; pod labels: app=web, role=api).- Zero-trust enhancements: mTLS between services (e.g., Istio/Linkerd) and short-lived credentials.Enforcing least privilege:- Default deny all ingress/egress in both SGs and NetworkPolicies.- SGs: restrict source by SG-id instead of wide CIDRs to bind cloud resources (prevents accidental exposure).- NetworkPolicies: allow only specific pod selectors and ports. Example: app namespace NetworkPolicy permits ingress from pods with label role=frontend on port 8080 only.- Database: allow connections only from app pods’ selector and from a backup/maintenance SG via scheduled windows.Managing rule lifecycle and auditing:- Policy-as-code in Git (templates + variables). Use GitOps to deploy policies via CI with automated validation tests: - Unit tests (policy syntax), integration tests (connectivity matrix in ephemeral clusters), and canary rollouts.- Tag-based and label-based naming conventions and ownership metadata (team, service, ticket id) embedded in policy manifests for traceability.- Automated drift detection: compare live SG/NetworkPolicy state to desired state and alert.- Auditing: enable VPC flow logs + cloud audit logs + Kubernetes network plugin logs (e.g., Calico) → ship to centralized SIEM; retain per compliance requirements; create automated reports showing allowed flows vs. actual flows.- Change governance: require PR reviews, security approver, and automated risk scoring before merge.Avoiding rule explosion:- Use hierarchical controls: coarse-grain in cloud SGs, fine-grain in k8s. Rely on label selectors and namespace scoping instead of per-pod rules.- Group similar services under shared selectors or network policy "allow lists" (e.g., internal-platform services).- Use policy templates and parameterization to generate consistent policies rather than hand-writing many bespoke rules.- Where supported, use aggregated/global policies (Calico GlobalNetworkPolicy) instead of duplicating per-namespace rules.Compliance considerations:- Retain logs and policy change history for required retention windows.- Periodic policy reviews and attestations (automated checklist).- Evidence collection: export diffs and connectivity test results as part of audit artifacts.- Provide documented exception workflow for temporary allowlist with automatic expiry and owner.Trade-offs & rationale:- SGs protect at cloud boundary and limit blast radius; NetworkPolicies enforce pod-level intent. Combining both prevents lateral movement even if one layer is misconfigured.- Using labels/namespaces reduces number of rules and aligns with CI/CD automation, at cost of stronger label governance.- mTLS adds operational overhead but significantly raises assurance for compliance-heavy customers.This approach balances least privilege, operational scalability, and auditability by combining coarse cloud controls, fine Kubernetes policies, policy-as-code lifecycle, and centralized logging/auditing.
HardSystem Design
57 practiced
Architect a database solution for an application that requires both high-volume transactional processing and near-real-time analytics. Compare a separated OLTP + OLAP approach versus an HTAP system. Include data movement strategies (CDC, streaming), storage choices, latency targets for analytics, and how to minimize impact on transactional performance.
Sample Answer
Requirements & constraints (clarify first):- Transactional throughput: e.g., 50k TPS peak, <10 ms p99 write latency.- Analytics freshness: near-real-time (1–5s) vs soft real-time (1–15 min).- Query types: complex aggregations, ad-hoc BI, and operational dashboards.- RPO/RTO, cost, team skillset.High-level options1) Separated OLTP + OLAP (classic)2) HTAP (single system serving both)Recommendation summary- If analytics freshness tolerance ≥ 1–30 minutes and you want clear resource isolation → separated OLTP + OLAP.- If analytics must be sub-second to a few seconds and you need single-copy simplicity → consider HTAP (e.g., SingleStore, TiDB with TiFlash, CockroachDB+vector engines) but accept complexity/limits and higher cost.Design A — Separated OLTP → OLAP (recommended for most enterprise clients)Architecture:- OLTP: row-oriented RDBMS (Postgres/RDS/Aurora or distributed OLTP like CockroachDB for geo-scale). Optimize for writes: minimal indexes, batched commits, connection pooling.- CDC Layer: use Debezium or DB-specific logical replication to publish change events to Kafka (or Kinesis).- Stream processing: Apache Flink / ksqlDB for enrichment, dedup, schema evolution handling, and incremental aggregates.- Data warehouse/OLAP: columnar MPP store (Snowflake, BigQuery, Redshift, ClickHouse) for heavy analytical queries and BI. Store both base tables (from CDC) and pre-aggregated materialized views.- Near-real-time dashboards: materialized views updated by stream jobs written back to a low-latency serving store (Redis, Druid, Pinot) for sub-second reads.Latency targets:- CDC→Kafka: <1s- Stream processing + write to DW: 1–30s (can tune to seconds with micro-batching)- Serving layer for dashboards: sub-second readsHow this minimizes impact on OLTP:- Asynchronous CDC decouples analytics ingestion; logical replication is read-only on primary; use logical slots with controlled retention.- Offload heavy analytical scans to OLAP stores; no long-running analytical queries on primary.- Use read replicas and query routing for reporting that can tolerate staleness.- Throttle CDC consumer workloads and isolate network/IO via dedicated replication NICs or VPC endpoints.Design B — HTAPArchitecture:- Single storage engine capable of both row and column formats (TiDB+TiFlash, SingleStore, Yugabyte+warehouse add-ons).- Real-time analytical queries run against column replicas or vectorized read paths; writes go to row store. Internal replication keeps column replicas updated.Latency targets:- Analytics freshness: sub-second to seconds (engine dependent).Pros:- Simpler logical model, single source of truth, transactional consistency for analytics.- Lower end-to-end latency (no external CDC).Cons / Trade-offs:- Higher operational cost, vendor lock-in, limited OLAP feature set vs specialized warehouses, potential contention for I/O/CPU, and complicated scaling for heavy analytical workloads.Data movement strategies — detailed- CDC (logical): Debezium / native logical replication → Kafka topic per table. Benefits: near-real-time, exactly-once patterns with Kafka+transactional writes. Use topic compaction for stateful tables.- Streaming processing: Flink/ksql for dedup, enrichment, incremental aggregates and for writing to OLAP or serving stores.- Micro-batch: For cost control, use 30s–5min micro-batches into the DW (Snowflake) for analytical queries that can tolerate slight lag.- Bulk backfills: periodic bulk loads for historical sync and schema evolution.Storage & data modeling choices- OLTP: row store, normalized schemas, short transactions, partitioning by time or tenant, minimal secondary indexes.- OLAP: columnar, wide tables (denormalized/fact tables), partitioned by date, clustered sort keys, aggressive compression.- Serving layer: key-value / time-series optimized store for dashboards.Operational controls to protect OLTP- Resource isolation: separate clusters, throttle replication consumers, QoS on network and IOPS.- Replica architecture: dedicated logical replication replicas for CDC to avoid burdening primaries.- Workload management: WLM in OLAP to bound resources; set concurrency limits and use query queues.- Monitoring & alerts: monitor replication lag, DB locks, CPU/IO saturation; auto-scale consumers.- Backpressure handling: if Kafka long lag, degrade non-critical analytics pipelines.Failure & consistency considerations- Ensure idempotent consumers for CDC to handle replays.- Use schema registry (Avro/Protobuf) for schema evolution.- Define staleness SLAs and fallbacks (cached aggregates) when stream latency spikes.- Plan for disaster recovery: retention of CDC logs, replayable snapshot workflows.Trade-offs recap- Separated: best isolation, mature tooling, cost-effective at scale; adds operational complexity of pipelines and eventual consistency.- HTAP: simpler data path and lower analytical latency but higher cost, potential resource contention, and fewer mature analytical optimizations.Decision checklist for client- If analytics latency target ≤ 5s and dataset fits HTAP offerings and budget allows → HTAP with columnar replicas.- If scale, cost-efficiency, and broad analytics features matter and >30s freshness acceptable → OLTP + CDC → stream → DW + serving layer.Concrete technologies to propose- OLTP: PostgreSQL/Aurora, CockroachDB- CDC: Debezium -> Kafka- Stream processing: Apache Flink / ksqlDB- OLAP: Snowflake / BigQuery / ClickHouse- Serving: Druid / Pinot / Redis- HTAP options: SingleStore, TiDB+TiFlash, ClickHouse (as HTAP read replica patterns) — validate against client scale and queries.This architecture balances transactional performance and near-real-time analytics by isolating heavy reads, using streaming CDC for low-latency data movement, choosing appropriate storage formats, and applying operational controls to protect OLTP.
EasyBehavioral
47 practiced
Tell me about a time you had to present technical tradeoffs to non-technical stakeholders (sales, product, or customer). Describe the situation, how you translated technical complexity into business impact, the decision you recommended, and the outcome. Focus on communication techniques you used to align stakeholders.
Sample Answer
Situation: As a Solutions Architect supporting a large enterprise sales opportunity, the customer needed a global SaaS deployment. Product and Sales pushed for a single-region rollout to meet contract timelines; engineering recommended a multi-region design for resiliency. The stakeholders (VP of Sales, Product Lead, and the customer's procurement lead) were non-technical and focused on cost and go-to-market risk.Task: I had to present the technical trade-offs so they could make an informed commercial decision: choose faster, cheaper single-region vs. higher-cost, multi-region with better SLA and lower outage risk.Action:- Translated complexity into business terms: I opened with the concrete business risks (e.g., potential revenue loss, SLA penalties, customer churn) rather than infrastructure jargon.- Used a one-page decision slide with three columns: Option, Business Impact, Likelihood & Cost. Each impact included a dollarized estimate (expected cost of downtime per hour based on their revenue figures) so trade-offs were tangible.- Employed simple analogies (single-region = single warehouse; multi-region = regional warehouses with failover) to explain availability and recovery time.- Presented a risk matrix (probability vs. impact) and mapped mitigation actions and their costs.- Ran a short demo showing failover behavior in <5 minutes to build trust.- Proposed a phased approach: start single-region with an explicit timeline and success metrics that would trigger expansion to multi-region if certain thresholds were exceeded.- Left a FAQ one-pager addressing common concerns (cost, timeline, compliance) and scheduled a follow-up decision workshop with engineering, sales, and procurement.Result: The stakeholder group approved the phased recommendation. Sales secured the contract with a 6-week initial deployment commitment and a funded expansion plan to multi-region after meeting reliability SLAs. This reduced upfront cost objections, preserved time-to-revenue, and aligned product and engineering on measurable triggers. The communication techniques—business-focused metrics, analogies, visual risk mapping, demo, and a phased mitigation plan—kept the conversation non-technical and decision-oriented, which led to consensus and a clear contract structure.
Unlock Full Question Bank
Get access to hundreds of Cloud and Infrastructure Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.