Technical Vision and Infrastructure Roadmap Questions

This topic assesses a candidate's ability to define a multi year technical vision for infrastructure, platform, and systems and to translate that vision into a practical execution roadmap. Core skills include evaluating technology choices and architecture evolution, planning migration and modernization paths, anticipating scalability and capacity needs, and balancing cost performance with resilience and operational reliability. Candidates should demonstrate approaches to managing technical debt, sequencing investments across quarters and releases, estimating resources and timelines, establishing measurable infrastructure goals and key performance indicators, and implementing governance and standards. Discussion may also cover reliability and observability, security and compliance considerations, trade offs between short term stability and long term rearchitecture, prioritization to enable business outcomes, and communicating technical trade offs to both technical and non technical stakeholders.

MediumSystem Design

43 practiced

Your company wants to move from nightly batch ETL to near-real-time streaming within 12 months. Draft a 12-month migration roadmap broken into quarters with milestones: pilot projects, staff skills, tech choices (frameworks), data validation strategies, automation, fallback paths, and how you would measure progress each quarter.

Sample Answer

Q1 (Months 0–3) — Prepare & pilot kickoff- Goals: align stakeholders, upskill core team, run 1 small pilot stream.- Pilot: ingest 1 critical dataset (e.g., user events) via Kafka or Kinesis into a staging topic.- Tech choices: evaluate Kafka (self-managed/Confluent) vs Kinesis (AWS) + stream processing candidates: Apache Flink / Kafka Streams / Spark Structured Streaming.- Skills: 2-week internal workshop + hands-on lab for streaming fundamentals, checkpoints, exactly-once semantics.- Data validation: define schema (Avro/Protobuf), implement sign-off rules, baseline data quality metrics.- Automation & fallback: CI for streaming apps, a nightly batch fallback for pilot dataset.- Metrics: pilot ingest latency, data loss rate, schema drift incidents, team training completion.

Q2 (Months 4–6) — Expand pilots & build platform- Goals: iterate on pilot, add 2 more datasets (different characteristics), build core platform components.- Platform: deploy messaging (Kafka/Kinesis), schema registry, feature store connectors, monitoring (Prometheus/Grafana), alerting.- Processing: implement streaming ETL for pilots in chosen framework, include watermarking and windowing tests.- Data validation: integrate stream-level validators (Deequ/Great Expectations streaming adapters), automated contract tests.- Automation: CI/CD pipelines for stream apps, blue/green deploy patterns.- Fallback: automated rollback and batch reprocessing runbooks.- Metrics: end-to-end latency, processing failure rate, successful rollbacks, % of datasets with streaming-ready schemas.

Q3 (Months 7–9) — Production rollouts & reliability- Goals: promote 3–5 datasets to production streaming, harden reliability, implement exactly-once or idempotent sinks.- Reliability: set SLAs, autoscaling, backpressure strategies, storage retention policies.- Observability: SLO dashboards, traceability from source→sink, alert thresholds.- Data validation: anomaly detection, golden dataset comparisons between batch and stream (outlier tolerance).- Automation: automate schema evolution approvals, drift alerts, nightly reconciliation jobs.- Fallback: maintain batch ETL as hot spare with automated switchover scripts.- Metrics: SLA attainment, data divergence rate vs batch <X%, MTTR for pipeline failures.

Q4 (Months 10–12) — Scale & optimize- Goals: migrate remaining high-priority pipelines, cost/perf optimization, org handoff.- Scale: shard topics, tune retention, optimize processing resources, adopt stateful checkpointing best practices.- Governance: data catalog integration, access controls, compliance checks.- Automation: full CI/CD, chaos tests for failover, automated reconciliation and alert remediation playbooks.- Fallback: documented, tested rollback for all migrated pipelines; runbook drills.- Metrics: % revenue/analytics-critical sources on streaming, cost per GB ingested, average end-to-end latency < target, data quality SLA 99.9%.

Cross-quarter governance & cadence- Weekly engineering syncs, monthly stakeholder reviews, quarterly postmortems.- KPIs tracked in a central dashboard; acceptance criteria for promoting pipelines to next stage.This roadmap balances pilot-driven learning, staff enablement, clear fallback paths, and measurable progress each quarter.

EasyTechnical

36 practiced

Describe the core components of a modern data platform that supports analytics and ML. Cover ingestion, storage (hot/warm/cold), processing (batch/stream), serving/warehouse, metadata/catalog, governance, and self-serve tooling. For each component give one example technology (open-source or managed) and the primary design consideration when including it in a roadmap.

EasyTechnical

39 practiced

Explain SLI, SLO and SLA in the context of data pipelines. Provide example SLIs for batch and streaming jobs (e.g., successful-run-rate, data-freshness latency, end-to-end latency), suggest SLO thresholds and explain what actions to take when SLOs are breached (page, mitigate, schedule postmortem).

Sample Answer

SLI, SLO, SLA — short definitions and how they apply to data pipelines:

- SLI (Service Level Indicator): a measurable metric that represents service behavior (e.g., percent of successful pipeline runs, data freshness latency, end-to-end processing time).- SLO (Service Level Objective): a target for an SLI you aim to meet (e.g., 99.9% successful runs per week).- SLA (Service Level Agreement): a contractual commitment to customers, usually with penalties if SLOs aren’t met.

Example SLIs and suggested SLOs

Batch pipelines (daily/weekly):- Successful-run-rate: fraction of scheduled runs that finish successfully. - SLO: ≥ 99.9% per week for critical ETL; ≥ 99% for non-critical.- End-to-end latency (ingest → usable in warehouse): time until data is ready. - SLO: ≤ 2 hours for nightly jobs; ≤ 6 hours for low-priority jobs.- Data completeness (% rows ingested vs expected): - SLO: ≥ 99.5% per run.

Streaming pipelines (near-real-time):- Data-freshness latency: time from event generation to availability. - SLO: ≤ 1 minute for low-latency features; ≤ 15 minutes for analytics.- End-to-end processing latency (event → downstream consumer): - SLO: median ≤ 500ms, 99th percentile ≤ 5s.- Error-rate per partition or per minute: - SLO: ≤ 0.1% errors.

Choosing thresholds: tie SLOs to consumer needs (analytics can tolerate higher latency than real-time features), business impact, and historical performance. Start conservative (what current systems achieve) then tighten iteratively.

What to do when SLOs are breached- Severity triage: - Critical breach (customer-facing or SLA-bound): immediate paging to on-call. - Non-critical breach: notify responsible team/channel and create priority ticket.- Immediate mitigation (runbook-driven): - Automatic: switch to fallback pipeline, replay from last checkpoint, increase parallelism, reroute traffic, promote cached snapshot. - Manual: trigger ad-hoc rerun, roll back recent deploy, increase resources.- Communication: - Post an incident summary to stakeholders with ETA and impact.- Postmortem & follow-up: - Conduct blameless postmortem within 48–72 hours: root cause, timeline, missed alerts, and runbook gaps. - Implement fixes: alert tuning, retries, backpressure handling, capacity increases, data quality checks. - Update SLOs if business needs change or if they were unrealistic.

Monitoring & tooling- Emit SLIs to metrics platform (Prometheus, Cloud Monitoring) and create dashboards and alerts on both absolute thresholds and trends.- Use both real-time alerts for immediate breaches and weekly SLO dashboards to spot drift.

This approach ties measurable indicators to business needs, prescribes clear operational responses, and ensures continuous improvement.

EasyTechnical

46 practiced

Compare blue-green and canary deployment strategies for data pipelines and platform services. Explain how each works, the pros and cons for stateful data jobs, impact on backfills and reprocessing, rollback characteristics, and give one concrete example of when to choose each strategy in a roadmap.

Sample Answer

Blue-green and canary are two deployment patterns useful for data pipelines and platform services. Briefly:

How they work- Blue-green: Maintain two full production environments (Blue = current, Green = new). Switch traffic or scheduler from Blue to Green atomically once validation passes.- Canary: Roll out new version to a small subset of workers, jobs, or partitions, monitor metrics, then progressively increase traffic until fully rolled out.

Pros/cons for stateful data jobs- Blue-green: - Pros: Clean separation — new code runs in isolated environment against a snapshot of inputs; easy to validate end-to-end without impacting current jobs. - Cons: Requires duplicate infra and state management (schemas, metadata, checkpoints). For stateful stream jobs, you must migrate or reinstantiate state (complicated for exactly-once semantics).- Canary: - Pros: Smaller blast radius, easier to test incremental compatibility with live state (e.g., run new task on subset of partitions). Lower infra overhead. - Cons: Mixed-version state can be tricky: checkpoints, offsets, and side effects may be incompatible. Risk of partial state divergence.

Impact on backfills and reprocessing- Blue-green: Safer for backfills because the new environment can run backfills against controlled snapshots; easier to keep backfill runs isolated from live production. But you must ensure consistent metadata and idempotency.- Canary: Backfills during a canary are riskier if backfill tasks run on different versions; prefer to run backfills entirely on one version or tag jobs to avoid mixed execution.

Rollback characteristics- Blue-green: Fast and deterministic rollback — switch traffic back to Blue; no need to undo state changes performed in Green if Green hadn’t been promoted. If Green made external writes, you must handle remediation.- Canary: Rollback involves stopping further ramp-up and possibly restarting failed canary jobs on the previous version. If canary mutated state, you may need compensating actions; smaller scope simplifies rollback.

When to choose each (roadmap examples)- Choose blue-green when releasing a major pipeline rewrite that changes schemas, storage formats, or checkpointing logic. Roadmap: “Q2 — replace ETL v1 with Spark v3 job that uses new checkpoints.” Use blue-green to validate full-run backfills in Green before cutover.- Choose canary for iterative, low-risk changes like optimizer tweaks, memory/config tuning, or small feature flags in platform services. Roadmap: “Q3 — roll out new serialization optimization across workers 10%→50%→100% to detect perf regressions.”

Practical tips- Always ensure idempotent writes, versioned schemas, and feature flags.- For stateful streaming, design state migration tools or use versioned checkpoint formats.- Automate monitoring (lag, error rate, data quality) and gating so both patterns can be safely observed and rolled back.

MediumTechnical

37 practiced

Estimate the resources and timeline to migrate 100 TB of historical data from on-prem HDFS to cloud object storage (e.g., S3). Include network bandwidth planning, parallel transfer strategy, compute required for transformation, validation approach to ensure correctness, costs to expect, and strategies to minimize downtime for consumers.

Sample Answer

Situation: You need to move 100 TB on-prem HDFS → S3 with minimal consumer disruption. Below is a practical plan covering bandwidth, parallel transfers, compute for transforms, validation, cost ballpark, and downtime mitigation.

1) Network & timeline- Useful throughput: assume 80% of link. Examples: - 1 Gbps link → ~100 MB/s → ~1 TB/2.8 hours → 100 TB ≈ 280 hours (~12 days) continuous. - 10 Gbps link → ~1 GB/s → 1 TB/17 minutes → 100 TB ≈ 28 hours.- Account for contention, retries, and throttling: add 20–30% overhead. So 1 Gbps realistic ~15 days; 10 Gbps ~36–48 hours.

2) Parallel transfer strategy- Use a distributed copier: Hadoop DistCp (with S3A) or AWS S3DistCp / s3cmd with multipart + parallelism. Run many mappers reading HDFS blocks in parallel.- Partition by directory/prefix to maximize parallelism (e.g., 500–2000 concurrent tasks depending on cluster/network).- Use multipart uploads (5 MB+ parts) and tune S3 request concurrency.

3) Compute for transformation- If no transform: lightweight cluster (e.g., 4–8 nodes) to run DistCp jobs.- If transformations (schema change, Parquet conversion, compression): use Spark on EMR/GKE. Estimate: to process 100 TB in ~24 hours, use ~50–100 r5.xlarge (or similar) workers depending on CPU/IO bound operations. Do a pilot to calibrate.

4) Validation approach- Produce checksums: generate per-file MD5/SHA256 on source and compare S3 ETag or store checksum metadata. For large files, use chunked checksums.- Row-level validation: sample rows/record counts and key aggregates (counts, null rates, checksums of key columns).- Reconciliation: file counts, total bytes per prefix, histogram of partition values.- Run integrity jobs in parallel with transfer for each prefix; mark prefix as “validated” when done.

5) Costs (ballpark)- Egress (if cloud provider charges ingress free but on-prem to cloud egress depends on provider/network): if using ISP, no cloud egress charge; if using cloud interconnect, pay for DS/Direct Connect charges (varies). S3 storage ~ $0.023/GB-month → 100 TB ≈ $2,300/month.- Request costs: PUT/Multipart requests modest; compute (EMR) for a 24–48h large Spark job ~ $2k–$6k depending on instance types.- Data transfer appliance (Snowball) alternative: Snowball 80TB per appliance; rental ~$300–$500 plus shipping — good if low bandwidth.

6) Minimizing downtime for consumers- Shadow copy + dual-write: Start writing new data to S3 while continuing writes to HDFS; maintain a replication window.- Read routing: Serve reads from HDFS until prefix validated; after validation, switch consumers to S3 prefix-by-prefix.- Use consistent partitioning and maintain metadata/catalog sync (Glue/Metastore) during migration to avoid schema drift.- Cutover: do small pilot cutover, then phased cutover during low-traffic windows; keep fallback for quick rollback.

7) Risk mitigation & recommendations- Run pilot on 1–5 TB to measure throughput, checksum strategy, transform cost.- Automate retries, exponential backoff, and logging.- Use lifecycle policies for storage class after migration (S3 Standard → Intelligent-Tiering / Infrequent Access).

This plan gives realistic timelines by bandwidth, an approach to parallel transfers and validation, rough cost buckets, and low-downtime strategies. Run a short pilot to finalize node counts and exact timings.

Unlock Full Question Bank

Get access to hundreds of Technical Vision and Infrastructure Roadmap interview questions and detailed answers.

Join thousands of developers preparing for their dream job.