Build vs. Buy vs. Cloud vs. On Premise Trade Offs Questions
Understanding key trade-offs in technology decision-making: (1) Build vs. Buy - custom development flexibility vs. packaged software speed/cost, (2) Cloud vs. On-Premise - operational burden, control, scalability, security, cost, (3) SaaS vs. Licensed - flexibility, upgrade frequency, customization options. Understanding implications for cost, time-to-value, flexibility, control, and ongoing support.
MediumSystem Design
41 practiced
Design a real-time ingestion and processing architecture for 50,000 events/second with end-to-end processing latency under 200ms for enrichment and routing to analytics and OLTP sinks. Sketch both an on-premise implementation and a cloud-managed stack, and compare trade-offs: ops effort, scaling, cost, data locality, and failure modes.
Sample Answer
Requirements:- 50k events/sec sustained, <200ms end-to-end latency for enrichment + routing to analytics (batch/OLAP) and OLTP sinks.- Exactly-once or at-least-once depending on sink SLAs.- Enrichment (low-latency joins with reference data), routing, monitoring.High-level approach:- Use a durable, partitioned ingestion layer → low-latency stream processing with stateful operators → parallel connectors to sinks. Keep processing in-memory with small state, use async side lookups for large reference data.On-prem implementation:- Ingress: Kafka cluster (20+ brokers, SSDs, replication factor 3, partition count ~200) fronted by load balancer.- Processing: Apache Flink cluster (YARN or Kubernetes) with task managers sized to handle ~50k eps (calculate slots based on CPU & throughput). Use RocksDB state backend on local SSD + checkpointing to HDFS/NFS for durability. Enrichment: in-process keyed state for small dimension tables; for larger reference data use a highly available Redis cluster or local read-optimized replicas and async lookups with caching.- Sinks: JDBC pool to OLTP (sharded PostgreSQL), and Kafka topics / HDFS for analytics (or ClickHouse for low-latency analytics).- Observability: Prometheus + Grafana, Alertmanager, Cruise Control for Kafka.Cloud-managed stack:- Ingress: Managed Kafka (MSK / Confluent Cloud / Pub/Sub).- Processing: Managed Flink (Amazon Kinesis Data Analytics for Apache Flink, Google Dataflow with Apache Beam) or Flink on GKE (Cloud Dataflow simplifies ops). Use cloud-managed state store (S3 for checkpoints with local RocksDB).- Enrichment: Managed Redis (Elasticache) or Cloud Memorystore; for large dimension data, Bigtable / DynamoDB with DAX / in-memory caching.- Sinks: Cloud OLTP (RDS/Aurora / Cloud SQL) and analytics (BigQuery / Snowflake) via managed connectors.- Observability: CloudWatch / Stackdriver + managed tracing.Scaling and sizing notes:- Partitioning: ensure Kafka partitions >= parallelism of Flink operators. Aim for ~1-2k events/sec per task slot (test per workload).- Latency: keep operator chain in a single task slot where possible, use async enrichment with bounded timeouts, set low checkpoint interval (e.g., 1–5s) while balancing throughput.- Backpressure: add ingress throttling or dead-lettering for downstream slowdowns.Trade-offs:- Ops effort: On-prem > Cloud. On-prem requires cluster provisioning, patching, capacity planning. Cloud offloads many operational tasks.- Scaling: Cloud offers near-instant elastic scaling; on-prem requires capacity headroom and manual scaling (longer lead time).- Cost: Cloud has higher OPEX and variable costs (ingest, storage, egress). On-prem has high upfront CAPEX but lower marginal cost at scale. Total cost depends on utilization.- Data locality: On-prem is best when regulations or low network latency to internal OLTP required. Cloud is better when sources/sinks already in cloud.- Failure modes: On-prem -> hardware failures, network partitions, operator misconfiguration; requires strong runbooks. Cloud -> provider outages, quota limits, higher black-box dependencies. In both, design for graceful degradation: local buffering (Kafka TTL), DLQs, idempotent writes, and retries.- Consistency: Exactly-once achievable with Flink + Kafka transactions to sinks that support idempotence/transactions (Kafka sinks, some DBs via upserts); for OLTP may need transactional staging tables.What I’d implement first:- Prototype with a smaller cluster (simulate 50k eps), benchmark per-task throughput, tune partitioning and state backend. Implement async cached enrichment and end-to-end tracing. Define SLOs and alert thresholds for latency, lag, and errors.
HardTechnical
38 practiced
Design an end-to-end audit logging and chain-of-custody system for a regulated financial firm where data pipelines span on-prem and cloud. Specify where logs are stored, tamper-evidence measures, retention policy, access controls, and how to present provenance to auditors. Include solutions for ensuring log integrity and reconstructing data lineage during an investigation.
Sample Answer
Requirements:- Immutable, tamper-evident audit trail across on-prem and cloud for ingestion, transforms, transfers, and access; searchable for investigations; retention and WORM/chain-of-custody guarantees; strict access controls; human-readable provenance for auditors.High-level architecture:- Emit structured audit events from every component (ingest agents, Spark jobs, ETL orchestration, DB changes, user access) to a centralized streaming bus (e.g., Kafka with TLS + client auth).- Sink events to two independent immutable stores: a hot index for queries (Elasticsearch/OpenSearch or BigQuery) and a cold WORM store for forensic evidence (AWS S3 with Object Lock in Governance/Compliance mode + Glacier Vault Lock; or Azure immutable blobs; on-premise write-once NAS with hardware seal).- Maintain a cryptographic ledger: each event appended includes (timestamp, sourceID, eventType, payloadHash, prevHash). Periodically (hourly/daily) batch-sign Merkle tree roots with an HSM-backed key (Cloud HSM / on-prem HSM) and publish signed root+timestamp to an external attestation endpoint (e.g., notary service or blockchain anchor).Tamper-evidence & integrity:- Append-only writes + S3/Blob Object Lock ensure immutability.- Cryptographic chaining: per-stream hash chain + Merkle root signatures provide provable sequence integrity; auditors can verify signatures against HSM public key.- Store event payload hashes in ledger and keep raw events in cold store; keep separate replicas (cross-region, cross-account) to detect differences.- Use KMS/HSM for signing; rotate keys with multi-party approval; keep key-use logs.Retention & retention policy:- Map retention to regulatory classes (e.g., transactional logs 7+ years, access logs 10 years). Implement lifecycle: hot index retention short (90d), archived signed WORM store for full retention. Ensure policy enforced by immutable storage features and reviewed by compliance.Access controls & separation:- RBAC/ABAC via IAM and identity federation; require MFA and privileged access justifications for forensic reads.- Separation of duties: engineers cannot both alter pipeline code and grant archive access. Use time-bound, auditable access via privileged access management (PAM).- All access attempts to logs produce audit events stored in WORM.Presenting provenance to auditors:- Provide an auditor portal that: - Shows provenance chain for a dataset/record: ingestion event(s), transformations (job IDs, code commit hashes), data snapshots (dataset version IDs), transfer hashes, and access records. - Exports verifiable evidence bundle: raw events, corresponding Merkle proofs, signed root, and HSM public key certificate. Include human-readable timeline and JSONL of events.- Support API and offline signed bundles with checksum and signature for independent verification.Reconstructing data lineage during investigation:- Instrument ETL to emit provenance events: datasetID, partition, schemaHash, parentIDs, jobID, commitHash, timestamps, row-level or partition-level record hashes (sampling or full, depending on volume).- Maintain a metadata catalog (e.g., Data Catalog/Atlas) that links events to physical files and versions. Store file-level manifests with content hashes (e.g., parquet file-level checksums stored in WORM).- Forensic steps: 1. Query catalog for dataset/timeframe → retrieve all provenance events and manifests. 2. Verify signatures and Merkle proofs to assert integrity. 3. Recompute and compare payload hashes for suspect files / records. 4. Re-run transformation code at recorded commit/tag in isolated environment to reproduce outputs; compare hashes.Operational best practices:- Test rollback and verification workflows regularly (quarterly audits).- Retain independent replica (different cloud/account) for disaster and integrity comparison.- Alerting on missing/late audit events and hash mismatches.- Document chain-of-custody process: who exported data, why, approvals, and physical transfer logs.Example tech stack:- Producers: Fluentd/Beats, instrumented Spark jobs- Stream: Kafka (TLS, ACLs)- Hot store: OpenSearch/BigQuery for queries- Cold store: S3 Object Lock + Glacier Vault Lock (or Azure immutable blobs)- HSM: AWS CloudHSM / Azure Key Vault (HSM)- Catalog: Apache Atlas / DataCatalog- Portal: internal web UI + signed evidence exportThis design provides provable, auditable, and reconstructable lineage with cryptographic integrity, immutable retention, strict access controls, and practical investigator tooling for compliance in regulated finance environments.
HardSystem Design
34 practiced
Design a multi-tenant internal data platform for dozens of product teams. Compare per-tenant isolated clusters versus shared multi-tenant clusters using RBAC and quotas. Provide architecture, cost allocation (showback/chargeback) model, security boundaries, onboarding, and how the choice interacts with cloud vs on-prem deployments.
Sample Answer
Requirements (clarify): dozens of product teams, varying workloads (batch/stream), strong data governance, cost visibility, low-latency interactive queries, mixed cloud/on‑prem constraints, compliance.High-level architectures- Per-tenant isolated clusters: each team gets a logically or physically separate compute cluster (K8s namespace + dedicated Spark/YARN/Presto pool or separate VM pool) and dedicated storage namespace (S3 prefix or HDFS path + IAM).- Shared multi-tenant clusters: single set of shared compute clusters with multi-tenant orchestration, RBAC, resource quotas, namespaces, and workload isolation (cgroups/Yarn queues/Kubernetes QoS + AdmissionControllers).Compare (trade-offs)- Isolation (security/fault blast radius): per-tenant >> shared. Isolated reduces noisy neighbor risk, easier compliance boundaries. Shared needs careful cgroups, namespaces, and workload priority preemption.- Cost efficiency: shared >> isolated (higher utilization). Per-tenant increases idle capacity and management overhead.- Operational complexity: per-tenant simpler troubleshooting per cluster; shared requires robust scheduler, telemetry, and multi-tenant testing.- Agility/onboarding: shared faster (template namespaces); isolated requires provisioning automation (infra-as-code).- Performance predictability: per-tenant more predictable; shared needs quotas, fair-scheduler, or slot reservations.Security boundaries- Per-tenant: enforce network segmentation, distinct IAM roles, encryption keys per tenant, separate logging/metrics buckets, dedicated Kerberos realms if needed.- Shared: strict RBAC (K8s RBAC + cloud IAM), data-plane encryption with per-tenant envelope keys, workload identity (SPIFFE), admission controls to prevent privilege escalation, side-channel mitigations, tenant-scoped logging + alerting.Cost allocation (showback/chargeback)- Metrics: CPU-hours, memoryGB-hours, storage GB-month, I/O ops, egress, managed service charges.- Implementation: - Shared: tag workloads with tenant IDs; collect telemetry (Prometheus, CloudBilling, Spark metrics); allocate costs via proportional rules; provide dashboards (Grafana) for showback; optionally enforce chargeback via monthly invoices/charge codes. - Isolated: map cluster-level costs directly to tenant; apply overhead allocation (platform ops %).- Hybrid model: baseline platform fee + usage-based charge, with reserved capacity discounts for committed tenants.Onboarding & developer experience- Provisioning: Terraform/ArgoCD templates that create tenant namespace, IAM roles, storage prefixes, KMS keys, monitoring hooks, and sample pipelines.- Guardrails: policy-as-code (OPA/Gatekeeper) to enforce schema, dataset tagging, and quota limits. Provide SDKs and datapipelines templates, CI/CD pipelines for data jobs.- Support: self-service portal to request quotas, SLA tiers (dev/test vs prod), and automated lifecycle (suspend/delete).Cloud vs On‑Prem interaction- Cloud: easier to implement shared due to managed autoscaling, fine-grained IAM, per-resource tagging, and cost APIs. Encryption and per-tenant keys via KMS. Chargeback easier via billing exports.- On‑Prem: tighter resource constraints favor per-tenant only for critical teams; shared requires robust capacity planning and possibly virtual clusters (K8s virtual cluster projects). Chargeback needs mapping of on-prem costs (hardware amortization, power, ops) to tenants.- Hybrid: use cloud bursting for heavy jobs—set policies to prefer on‑prem for sensitive data and cloud for elastic workloads; implement unified metadata/catalog to abstract location.Recommendation for dozens of teams- Use a hybrid approach: shared multi-tenant clusters for dev/analytics with strong RBAC, quotas, fair-scheduler, and per-tenant KMS keys; offer isolated clusters (or dedicated node pools / tenant-specific virtual clusters) for high-security/compliance teams or heavy, latency-sensitive workloads.- Build centralized platform services: identity, catalog (Data Catalog/Glue/Atlas), observability, cost allocation pipeline, policy engine, and self-service provisioning.- Enforce governance early: dataset classification, mandatory tenant tags, SLOs, and automated cost/showback to drive efficient behavior.Operational checklist- Implement per-tenant telemetry and billing pipeline- Enforce policy-as-code and CI for data jobs- Provide clear SLA tiers and onboarding runbook- Automate lifecycle and disaster recovery per tenancy model chosenThis balances security, cost, and scalability while allowing teams to graduate from shared to isolated as needs grow.
MediumTechnical
65 practiced
Implement a Python function estimate_monthly_cost(compute_hours, storage_gb, egress_gb, rates) that returns a dictionary with 'cloud' and 'onprem' estimated costs. Rates is a dict like {'cloud': {'cpu': 0.05, 'storage_per_gb': 0.025, 'egress_per_gb': 0.09}, 'onprem': {...}}. Apply a 10% storage discount for volumes > 10,000 GB. Include basic unit tests for zero usage and a large-scale case.
Sample Answer
Approach: compute monthly cost per environment by multiplying compute_hours by cpu rate, storage_gb by storage rate (apply 10% discount if >10,000 GB), and egress_gb by egress rate. Return {'cloud': cost, 'onprem': cost} with costs rounded to 2 decimals.Key points:- Applies discount only to storage rate when storage_gb > 10,000.- Missing rate keys default to 0.0 to avoid exceptions.- Rounds to 2 decimals for currency-like output.Time/Space:- Time O(1), Space O(1).Edge cases:- Zero or negative inputs (negative treated mathematically; validate upstream if not allowed).- Missing environment keys handled by empty dict.Basic tests:
python
def estimate_monthly_cost(compute_hours, storage_gb, egress_gb, rates):
"""
compute_hours: float
storage_gb: float
egress_gb: float
rates: dict like {'cloud': {'cpu':0.05, 'storage_per_gb':0.025, 'egress_per_gb':0.09}, 'onprem': {...}}
"""
def single_cost(r):
cpu_cost = compute_hours * r.get('cpu', 0.0)
storage_rate = r.get('storage_per_gb', 0.0)
# apply 10% discount for volumes > 10,000 GB
if storage_gb > 10_000:
storage_rate *= 0.9
storage_cost = storage_gb * storage_rate
egress_cost = egress_gb * r.get('egress_per_gb', 0.0)
return round(cpu_cost + storage_cost + egress_cost, 2)
return {env: single_cost(rates.get(env, {})) for env in ('cloud', 'onprem')}python
def test_zero_usage():
rates = {'cloud': {'cpu':0.05,'storage_per_gb':0.025,'egress_per_gb':0.09},
'onprem': {'cpu':0.02,'storage_per_gb':0.01,'egress_per_gb':0.05}}
assert estimate_monthly_cost(0,0,0,rates) == {'cloud': 0.0, 'onprem': 0.0}
def test_large_scale():
rates = {'cloud': {'cpu':0.05,'storage_per_gb':0.025,'egress_per_gb':0.09},
'onprem': {'cpu':0.02,'storage_per_gb':0.01,'egress_per_gb':0.05}}
res = estimate_monthly_cost(100000, 20000, 5000, rates)
# cloud: cpu=100000*0.05=5000, storage rate discounted=0.025*0.9=0.0225 -> storage=20000*0.0225=450, egress=5000*0.09=450 => total=5900.0
assert res['cloud'] == 5900.0MediumSystem Design
50 practiced
Your analytics users are distributed across EU and APAC and require low-latency dashboards plus EU-only residency for some customer data. Compare multi-region single-cloud, multi-cloud, and single-cloud with on-prem edge nodes. Discuss replication, consistency, operational complexity, cost, and compliance trade-offs.
Sample Answer
Requirements clarification:- Low-latency dashboards for EU and APAC users.- Some customer data must remain EU-only (data residency/GDPR).- Global availability and reasonable freshness for analytics.- Operationally maintainable and cost-effective.High-level comparison (pros/cons):1) Multi-region single-cloud (e.g., AWS with regions in EU/APAC)- Replication & consistency: Use region-local read replicas (RDS/Aurora Global DB, cross-region object replication). Can choose async replication for scale (eventual consistency) or single-writer multi-read with global DB features for stronger consistency (higher latency on writes).- Latency: Low read latency using regional replicas; writes may be routed to a primary region -> higher write latency from remote clients unless write-local staging is used.- Operational complexity: Moderate — one cloud API, unified tooling, simpler IAM/billing, but need cross-region replication configs and failover testing.- Cost: Moderate — inter-region data transfer and replica costs, but one vendor discounts possible.- Compliance: Easier to enforce EU residency by keeping EU-only datasets in EU region and restricting replication rules; cloud provider contracts/CSPD help.2) Multi-cloud (EU on GCP, APAC on AWS/Azure)- Replication & consistency: More complex — cross-cloud replication requires custom pipelines (Kafka MirrorMaker, CDC to cloud-agnostic storage), typically eventual consistency. Harder to implement strong consistency transactions.- Latency: Excellent for local reads if data colocated; complexity for global joins or enrichments that need cross-cloud access.- Operational complexity: High — different APIs, networking, identity, monitoring, schema/versioning sync, and DR across providers.- Cost: Higher — duplicate services, egress charges between clouds, more engineering overhead.- Compliance: Strong separation possible by design (keep EU data entirely in EU cloud), but legal/compliance contracts must be managed per provider.3) Single-cloud + on-prem edge nodes (EU and APAC edge caches/processors)- Replication & consistency: Edge nodes handle local ingestion, light transformations, and caching; central cloud stores canonical data. Use tiered replication: write-local, async ship to cloud for authoritative store. Eventual consistency expected; can implement conflict resolution/CDC for reconciliation.- Latency: Best for interactive dashboards if edge serves reads from local aggregated materialized views; writes are fast locally.- Operational complexity: Highest in terms of ops footprint — manage edge hardware/software, connectivity, deployments, and reconciliation jobs.- Cost: Upfront CAPEX for edge plus ongoing ops; reduced egress if edge aggregates before shipping.- Compliance: Very strong for residency (keep EU customer raw data on-prem or in EU-edge), but increases compliance burden (physical security, audits).Trade-offs & recommendation for role:- If you want lower ops friction and acceptable write-latency tradeoffs: multi-region single-cloud with strict replication rules — keep EU-only tables blocked from cross-region replication, use regional read replicas and materialized views for dashboards.- If legal/compliance requires absolute physical separation and lowest read latency: single-cloud + edge nodes for EU residency.- Use multi-cloud only if vendor risk or specific regional services are essential — accept higher complexity/cost.Practical patterns:- Use CDC (Debezium/Kafka) for reliable async replication and clear provenance.- Serve dashboards from regional materialized views (Presto/Trino, BigQuery/Athena) refreshed via streaming micro-batches.- Implement RBAC, encryption-at-rest and in-transit, automated tests for replication rules, and monitoring/alerts for replication lag to meet SLA and compliance.
Unlock Full Question Bank
Get access to hundreds of Build vs. Buy vs. Cloud vs. On Premise Trade Offs interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.