Technology and Platform Selection Questions

Evaluation and justification of technologies services and platforms used to implement systems across the stack. Candidates should be able to select compute options including virtual machines containers and serverless platforms as well as orchestration and workflow engines messaging systems batch and streaming processing engines object and block storage data warehouses and other data platforms. The topic encompasses comparing managed services and self managed deployments cloud versus on premise hosting and choosing frameworks runtimes and overall stacks based on workload characteristics. Assessment focuses on weighing trade offs across cost operational overhead reliability latency and throughput scaling characteristics vendor lock in development velocity team familiarity and learning curve maturity and community support security and compliance and monitoring and debugging complexity. Candidates should demonstrate how system requirements map to service capabilities justify build versus buy decisions and managed service choices design proof of concept experiments and outline migration and rollout planning while making pragmatic choices that balance performance cost and operational risk.

HardSystem Design

63 practiced

You are migrating a live operational streaming pipeline from on-prem Kafka to a cloud-managed streaming service while producers and consumers remain live. Outline a zero-downtime migration strategy that covers data synchronization (dual-write or mirror), offset translation and mapping, DNS/service discovery changes, cutover verification, and rollback approach.

Sample Answer

Requirements & constraints:- Zero-downtime: producers and consumers must continue with no data loss and minimal lag.- Live migration of topic data and offsets from on-prem Kafka to managed cloud Kafka.- Ability to roll back quickly if issues arise.

High-level approach (phased, reversible):1. Prepare: provision cloud cluster, match topic configs (partitions, retention, compression), IAM/networking, monitoring, and a dual-write gateway or replication layer.2. Continuous data sync (mirror): run an always-on replication tool (MirrorMaker2 / Confluent Replicator / Kafka Connect with MirrorSourceConnector) to copy topics from on-prem → cloud. Use at-least-once semantics and preserve message metadata (timestamps, keys, headers). Monitor replication lag and throughput.3. Dual-write for producers (optional / incremental): implement a producer-side dual-write or proxy that writes to both clusters in parallel (idempotent producers or unique message IDs). Prefer a routing/proxy layer (Envoy, custom producer library, or Kafka producer wrapper) so producer changes are toggled remotely. If dual-write isn’t feasible, rely on mirror replication only.4. Offset management & translation: - While mirroring, consumer group offsets differ. Capture consumer group offsets on source periodically (kafka-consumer-groups or offset exporter). Use a mapping process to translate source offsets to target offsets: for each topic+partition, find the target offset whose message key+timestamp matches or is the first offset >= source offset’s timestamp. Tools: Kafka’s OffsetSync (Confluent), custom offset-lookup using message timestamps/keys, or use MirrorMaker2’s offset-sync feature. - Persist mapping in a small offset-mapping store (etcd/Redis/Postgres) keyed by consumer-group -> topic -> partition -> target-offset. Keep this updated until cutover.5. DNS / service discovery & cutover plan: - Introduce an abstraction for producers and consumers: logical bootstrap endpoints (VIP/load balancer or DNS name like kafka-prod.company.com). Initially point to on-prem cluster. - For producers: shift traffic by updating routing to send weighted traffic to cloud (canary) then 100% once verified. If using dual-write, simply flip a feature flag to stop dual-write and only write to cloud. - For consumers: avoid abrupt switch. Use a consumer-side migration helper that can start consuming from cloud with provided target-offsets from mapping store. Strategy: start a consumer group in “shadow-read” mode that consumes from the cloud in parallel for verification but does not commit; when ready, coordinator instructs consumers to stop on-prem consumption and resume from mapped cloud offsets so processing continues without reprocessing or gaps.6. Cutover verification: - Pre-cutover checks: replication lag near zero; message counts and checksums equal across time windows; consumer-group offset mappings validated on a small test consumer. - During progressive cutover: run end-to-end test streams, verify downstream results (hash of processed outputs), monitor consumer lag, error rates, commit rates, and business metrics. Use canary consumers to process a subset and compare outputs. - Post-cutover: assert no data loss by comparing high-watermarks and message counts; ensure consumer lags stabilize and throughput meets SLOs.7. Rollback approach (fast, deterministic): - Keep replication bidirectional or continue mirroring cloud→on-prem (short window) if rollback capability is needed, or ensure replayable logs exist. - If cutover shows problems: re-enable producers to write to on-prem (flip DNS/load-balancer back, or re-enable dual-write), stop consumers reading from cloud, restore consumer offsets on on-prem from offset mapping, and resume processing. Because replication was continuous, on-prem will have all messages up to cutover. Ensure idempotence to avoid duplicates.8. Operational controls & safety nets: - Feature flags to toggle dual-write, proxy routing, and consumer source. - Rate limiting/throttling during sync to avoid overload. - Robust monitoring: replication lag, per-partition offsets, producer acks, consumer lags, end-to-end checksums. - Run blameless rollback drills ahead of migration.

Trade-offs and notes:- Dual-write guarantees near-zero producer impact but increases complexity (idempotency required) and reconciliation work.- Mirror-only plus offset-translation is simpler for producers but requires careful consumer offset mapping; timestamp/key-based mapping can be imperfect for duplicates or key-less messages.- Maintain a long “safe window” where both clusters are kept in sync until business verifies success.

Concrete tooling suggestions:- MirrorMaker2 or Confluent Replicator for high-throughput replication.- Kafka consumer-group offset export + a small service to compute target offsets by timestamp/key.- Envoy or a DNS-based VIP for transparent routing, and LaunchDarkly/Config service for toggles.- Observability: Prometheus + Grafana, Kafka Exporter, and end-to-end validation jobs.

This phased, reversible plan keeps producers and consumers live, ensures consistent data and offsets, allows progressive verification, and provides a rapid rollback path if any step fails.

HardTechnical

85 practiced

You have two weeks to deliver a compact proof-of-concept (PoC) demonstrating a low-latency, fault-tolerant ingestion-to-analytics pipeline for leadership. Define clear success criteria, a minimal architecture (including whether to use managed vs self-managed services), test scenarios to validate latency and fault tolerance, and how you would measure scalability and operational feasibility within the timebox.

Sample Answer

Success criteria (measurable, 2-week PoC):- End-to-end median latency ≤ 500 ms and 99th percentile ≤ 2s for 1k events/sec.- No data loss under single-component failure (producer, broker, or worker) for a 5 minute outage.- Demonstrate horizontal scale to 10k events/sec with linear throughput scaling.- Dashboards + alerting and a short runbook for failover ops.

Minimal architecture (prefer managed where it speeds delivery):- Producers: simple HTTP/SDK generator (locally or containerized).- Ingestion: managed Kafka (Confluent Cloud) or AWS MSK / Kinesis Data Streams for durability and quick setup.- Stream processing: Apache Flink (managed e.g., Ververica or Flink on Kubernetes) or Spark Structured Streaming if team familiar; choose Flink for lower-latency streaming.- Cold storage: S3 (parquet) for raw/events + checkpointing.- Serving/analytics: ClickHouse hosted or BigQuery for fast OLAP; ClickHouse if on-prem-like fast query latency is needed.- Observability: Prometheus + Grafana (managed Grafana Cloud optional), OpenTelemetry traces, and error logs in CloudWatch/ELK.- Orchestration: lightweight k8s (EKS/GKE) or serverless for processors if available.

Managed vs self-managed rationale:- Use managed Kafka/MSK and managed ClickHouse/BigQuery to minimize ops overhead and hit PoC goals in 2 weeks. Self-manage only if org mandates.

Test scenarios (what to run & how):1. Baseline latency: run steady 1k ev/s for 10 minutes; record p50/p95/p99 from producer->processor->store. 2. Burst/scale: spike to 10k ev/s for 5 minutes; verify throughput and latency. 3. Failure injection: kill a broker, then a processing pod, then simulate S3 unavailability for 1–2 minutes; verify no data loss (via offsets + parity counts) and automated recovery. 4. Backpressure: slow downstream (ClickHouse) to see how stream handles buffering and bounded retention. 5. Data correctness: end-to-end checksums and sample queries to validate transformations.

How to measure scalability and operational feasibility in timebox:- Load generation: k6 or custom Go producer to ramp traffic; measure max sustainable throughput while maintaining latency SLOs.- Metrics: consumer lag, throughput (events/sec), CPU/memory, GC, network, broker partitions, Flink operator metrics (processingTime, checkpointDuration), query latency on serving DB.- Scaling tests: increase processor replicas/slots and show near-linear throughput increase; measure autoscaling times.- Cost & ops: estimate 30-day cost extrapolation, document required team time (on-call, patching), provide runbook for common failures (broker down, long GC, disk full).- Deliverables: architecture diagram, Grafana dashboards, postmortem of tests, recorded demo of failover and scaling, and recommended next steps (prod hardening: replication factor, retention, monitoring SLOs, security).

Why this is realistic:- Managed services remove undifferentiated heavy lifting and allow proving latency and fault-tolerance in two weeks. Tests focus on observable, repeatable metrics and failure modes leaders care about.

MediumSystem Design

57 practiced

A company wants to avoid vendor lock-in and explore a pragmatic multi-cloud ingestion architecture for collecting events from globally distributed sources. Outline an architecture that balances risk and cost, explain which components you would make cloud-agnostic versus cloud-specific, and list the trade-offs with respect to latency, egress costs, complexity, and operational burden.

Sample Answer

Requirements & constraints:- Global event ingestion from many producers, avoid vendor lock-in, pragmatic cost-sensitive, tolerate small added complexity, low-to-moderate end-to-end latency (< seconds–minutes), minimize egress costs.

High-level architecture (text diagram):Edge collectors → Regional buffering & pre-processing → Central multi-cloud ingestion hub → Long-term storage & processing

Components and responsibilities:- Edge collectors (cloud-agnostic): lightweight agents/SDKs or HTTP/WebSocket clients deployed near producers (on-prem, edge, or k8s). Responsibilities: batching, compression, TLS auth, local retry/backoff.- Regional buffering & pre-processing (cloud-specific but portable patterns): use managed services where cheap/available — e.g., Kafka/Confluent online, MSK, or Pub/Sub — or run self-managed Kafka on k8s (Strimzi) if portability preferred. Responsibilities: ingestion fan-in, deduplication, schema validation, minimal enrichment.- Central ingestion hub (cloud-agnostic orchestration): containerized stream processors (Fluent Bit/Fluentd, Spark Structured Streaming, Flink) orchestrated on k8s clusters that can run on any cloud; use Terraform/Helm for infra as code.- Storage tier: abstracted via a cloud-agnostic storage layer (object storage with S3 API compatibility — e.g., AWS S3, GCS with S3 gateway, or MinIO multi-cloud). Use replication/mirroring to chosen long-term store.- Cross-cloud transfer: use event routing with smart regional sinks to keep most data local; only aggregate or analytics-relevant subsets cross regions/clouds.- Observability & IAM: centralize logging/metrics (Prometheus/Grafana, OpenTelemetry) with exporters adapted per cloud.

Which to make cloud-agnostic vs cloud-specific:- Agnostic: Edge agents, stream processing code, orchestration (k8s), formatting/schema (Avro/Protobuf), monitoring, IaC templates.- Cloud-specific (use when advantageous): Managed queuing/stream services (Kinesis, Pub/Sub), native object stores for primary storage when egress cost/latency favors staying on that cloud.

Trade-offs:- Latency: Local regional buffering lowers ingest latency; cross-cloud aggregation increases end-to-end latency for global views.- Egress costs: Keeping raw data in-region/cloud and transferring only materialized subsets minimizes egress. Using managed services per cloud reduces cross-cloud traffic but increases fragmentation.- Complexity: Multi-cloud increases operational complexity (deployments, monitoring, IAM differences). Using k8s+portable tools reduces but doesn’t eliminate complexity.- Operational burden: Self-managed Kafka/MinIO increases ops cost but reduces vendor lock-in; managed cloud services lower ops but increase lock-in risk and may cost more overall.

Recommendations:- Start hybrid: use managed services for quick wins in primary cloud while architecting processing and storage to be portable (k8s, S3-compatible). Instrument clear data-routing policies to minimize egress and iterate toward more portable components only if lock-in risk outweighs operational cost.

MediumTechnical

46 practiced

You must choose ingestion connectors to move transactional data into a data lake in near-real-time. Compare CDC-based connectors (for example Debezium), batch-extraction with incremental queries, and cloud-native change streams. Discuss latency, transactional consistency, DDL handling/schema drift, and operational complexity.

Sample Answer

Start with short definitions, then compare across the four dimensions requested.

Definitions:- CDC connector (e.g., Debezium): reads database transaction logs (WAL/binlog) and emits row-level change events (insert/update/delete) in near-real-time.- Batch-extraction with incremental queries: periodic jobs that query tables for rows changed since last run (timestamp/modified flag/offset).- Cloud-native change streams: managed DB/platform streams (e.g., AWS DMS CDC, Cosmos DB Change Feed, Cloud Spanner streams) that provide change events without self-managed connectors.

Comparison:

1) Latency- CDC (Debezium): low latency (sub-second to seconds) depending on connector throughput and downstream processing.- Batch incremental: higher latency (minutes to hours) determined by schedule; not suitable for near-real-time.- Cloud-native change streams: similarly low latency to CDC; often optimized and integrated with cloud messaging.

2) Transactional consistency- CDC (log-based): can preserve transaction boundaries if you consume and apply events in order and respect transactional metadata; strong consistency for committed transactions.- Batch incremental: reads current table state per run; consistency is coarse-grained per run and may miss intermediate states; safe for final consistent snapshots but not for ordered per-row changes.- Cloud-native streams: depends on provider — many preserve ordering and txn metadata, but check guarantees (at-least-once vs exactly-once).

3) DDL handling / schema drift- CDC (Debezium): supports DDL capture for many connectors and emits schema-change events; still requires consumer logic to evolve schemas, manage backward compatibility, and migrate downstream tables.- Batch incremental: schema changes are detected at job runtime (query fails or new columns appear); simpler to handle schema drift via ETL job changes but risk of silent failures if not validated.- Cloud-native streams: often surface DDL/metadata events, but behaviors vary. Managed services may version schemas or offer integration with schema registries.

4) Operational complexity- CDC (self-managed Debezium): higher ops cost — maintain connector, Kafka/topic topology, offset management, schema registry, and careful handling of schema evolution and replay. Requires expertise in log positions, connectors, and failure recovery.- Batch incremental: lowest operational complexity to implement initially; scheduling, watermarking, and idempotency are key considerations. Easier debugging but scales poorly for high-change volumes and many tables.- Cloud-native streams: moderate complexity — less infra to manage, but you must understand provider limits, costs, and integration patterns. Easier scaling and reliability in many cases.

Recommendations:- For true near-real-time needs and per-row auditability: prefer log-based CDC or cloud-native change streams. Use Debezium if you need open-source flexibility and control; choose cloud-native streams if you want managed reliability and lower ops.- For analytical use cases tolerant to minutes/hours latency or many low-change tables: batch incremental is cost-effective and simpler.- Always plan for schema evolution: use schema registries, enforce compatibility rules, include DDL handling logic in consumers, and add testing/monitoring (SLA, lag, schema drift alerts). Consider hybrid: CDC for high-value transactional tables, batch for low-priority ones.

HardTechnical

53 practiced

Executives propose standardizing the platform on a single cloud vendor to speed delivery, but engineering teams worry about vendor lock-in. As a lead data engineer, present a framework to quantify lock-in risk (technical, financial, operational) and propose mitigation strategies that balance reduced lock-in with development velocity and operational simplicity.

Sample Answer

Situation: Executives want to standardize on one cloud to speed delivery; engineering fears vendor lock‑in affecting portability, cost and ops flexibility.

Framework to quantify lock‑in risk (three dimensions):1) Technical (portability score 0–10) - Metrics: % of services using proprietary managed services (e.g., Redshift, BigQuery, Athena), number of proprietary APIs used, data format/vendor-specific features, infra as code (IaC) portability. - Measurement: Weighted score = 0.6 * managed-service-dependence + 0.4 * API-/format-coupling. - Example: 70% managed-service usage → technical risk high (score ~7).

2) Financial (cost exposure index) - Metrics: annualized migration cost estimate, projected vendor price inflation sensitivity, multi‑year committed discounts vs. exit penalties. - Measurement: Cost exposure = (one‑time migration + 3yr price delta) / annual infra spend. - Thresholds: >0.5 → high financial risk.

3) Operational (recovery & continuity) - Metrics: mean time to replatform (MTTRP), staff skill concentration (% of team certified on chosen cloud), runbook portability, SLA dependency. - Measurement: Operational risk score combining MTTRP normalized and skill concentration.

Combine into overall Lock‑In Risk = weighted sum (example weights: technical 50%, financial 30%, operational 20%) to produce a single dashboard value and band (Low/Moderate/High).

Mitigation strategies balancing velocity and simplicity:- Short term (0–6 months): Adopt opinionated standard for core infra to gain velocity but require abstraction boundaries for data plane and APIs. Enforce standards: open data formats (Parquet/Avro), cloud-agnostic storage patterns (S3-compatible), and contract-first data APIs. Capture portability debt in sprint backlog with quantified effort estimates.- Mid term (6–18 months): Encapsulate proprietary services behind thin adapters (facade pattern). Use containerized processing (Kubernetes or managed K8s) and IaC templates with provider modules (Terraform with per-provider modules) to reduce replatform cost. Prefer open-source or cross-cloud managed offerings where throughput/latency needs allow.- Long term (18+ months): Implement automated migration runbooks, periodic portability drills, multi-cloud CI pipelines, and a “golden path” of supported components that guarantee portability. Negotiate commercial terms (exit clauses, data egress caps) with vendor and reserve budget for a contingency migration.

Trade-offs and governance:- Accept limited use of high‑value managed services when business value outweighs lock‑in risk; require ROI + sunset/exit plan.- Use the risk dashboard to set allowable thresholds and a review board for exceptions (Product/Finance/Engineering).- Measure success by reduced migration cost estimates, lowered MTTRP, and velocity metrics (lead time) — track quarterly.

Conclusion (actionable next steps):1. Run a 4–6 week inventory and scoring exercise to populate dashboard.2. Propose an initial standard with mandated abstraction rules and exception process.3. Schedule quarterly portability drills and a vendor‑terms review.

This balances immediate delivery gains with measurable, funded steps to limit lock‑in and preserve future options.

Unlock Full Question Bank

Get access to hundreds of Technology and Platform Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.