Understanding the Company's Infrastructure Context Questions

Research the company's public infrastructure information (engineering blog, tech talks, published case studies, job description). Understand what systems they operate at scale, what problems they likely face, and what your role would contribute to.

EasyTechnical

28 practiced

You find that the company contributes an open-source Kafka connector. What immediate technical inferences can you draw about their ingestion strategy (supported sources, reliability focus, schema handling), and list 5 targeted technical questions you would ask in an interview to confirm how that connector is used in production.

HardTechnical

25 practiced

A case study describes repeated outages caused by schema changes that broke downstream jobs. Propose an automated data validation and contract-testing framework that integrates with CI/CD, handles very large datasets efficiently, minimizes false positives, and catches regressions early. Include sampling strategies, lineage checks, and how to surface failing contracts to producers.

Sample Answer

Requirements:- Catch schema/regression issues early in CI before production- Scale to TBs/day without full-scan cost- Low false-positive rate (signal actionable issues)- Surface failures to data producers with lineage + remediation steps

High-level design:1. Contract Registry (git-backed): schemas (Avro/JSON/Protobuf/Parquet), semantic constraints, column-level SLAs, allowed-evolution rules, and example records. Versioned per topic/table.2. Contract Test Runner integrated into CI/CD (GitHub Actions/Jenkins): on PR to producer repo or contract change, run lightweight validations and block merges on violations.3. Runtime Validation Framework: deployed as streaming/batch validators (Spark/Flink or Cloud Dataflow) that execute contract checks in staging and prod. Emits structured findings to a issues service.

Key components & strategies:- Pre-commit / PR checks: static schema compatibility (backward/forward/patch), syntactic validation, and sample-based checks using representative sample snapshot (see sampling).- Staging pipeline: run validators on a small window of actual data (e.g., most recent 1–24 hours) to detect integration issues before full rollout.- Production continuous checks: high-frequency lightweight checks (schema presence, null-rate thresholds) plus periodic deep checks.

Sampling strategies to handle scale:- Stratified reservoir sampling per partition key (user_id, region) to preserve rare segments.- Time-decayed sampling: higher weight to recent data to detect regressions quickly.- Sketches & probabilistic data structures (HyperLogLog for cardinality, Count-Min for heavy hitters) for O(1) summaries instead of full scans.- Bloom filters to detect unexpected new values quickly.

Lineage and root-cause:- Instrument validators to capture upstream lineage (dataset/table, job id, commit hash, run id) using OpenLineage/Marquez. When a contract fails, attach lineage metadata pointing to producer artifact (git commit, schema id) and the exact transformation/job that introduced change.

Minimizing false positives:- Multi-stage checks: require failing signal across two independent checks (e.g., schema diff + elevated null-rate) before alerting.- Use historical baselines with tolerance bands (percentile-based) rather than fixed thresholds.- Allow automatic “canary” rollouts: first deploy new schema to test partition; only escalate if canary fails.

Failing contract surfacing & remediation:- Create structured findings in a Data QA dashboard and ticket system (Slack + Jira) with: - Clear failure type (schema incompat, missing column, unexpected type, distribution drift) - Example offending records (sampled) - Lineage links: producer repo + PR, job run, dataset version - Suggested remediation (revert contract, update consumer, add migration)- Auto-open a blocking ticket for producers and notify via channel; include a one-click rollback to previous schema in the registry when safe.

CI/CD integration:- Producers run contract-lint and contract-test on PR; consumers run compatibility-check against registry.- Gate merges if tests fail. Use feature flags for schema-evolution and automated migrations (e.g., add column with default).- Publish contract versions and validator results as artifacts for audits.

Example checks:- Schema compatibility: Avro compatibility rules- Column existence/type: exact match or allowed coercion map- Cardinality & unique-key constraints via HyperLogLog- Null rate: alert if > baseline + delta- Distribution drift: KL-divergence sketch on categorical columns

Trade-offs:- Probabilistic structures reduce cost but give approximate signals—mitigate with targeted exact checks when triggered.- Sampling reduces compute but might miss ultra-rare regressions—use stratified sampling to lower that risk.

This framework catches regressions early in PR/CI, scales via sampling and sketches, links failures to producers with lineage metadata, and reduces noise through multi-signal confirmation and baseline-aware thresholds.

HardTechnical

26 practiced

Cost modeling case study: You're joining a team that processes PB-scale data and runs compute-heavy ML features. Outline a cost-modeling approach to estimate monthly cloud spend across storage (hot/warm/archive), compute (batch/stream), and network (cross-region/egress). Which public signals (documents, job runtimes, public slides) would you use to validate your estimations and reduce uncertainty?

Sample Answer

Framework — clarify scope, timeframe, and SLAs (monthly, PB-scale, retention, RTO/RPO, region footprint). Break costs into three buckets: storage, compute, network. For each, model capacity × unit price × modifiers (redundancy, compression, sustained discounts, reserved/spot mix).

Storage- Gather: ingested TB/day, retention policy by tier (hot/warm/archive), compression ratios, replication factor, object count.- Model: MonthlyStorageCost = Σ_tier (AvgBytes_tier × Price_per_GB_tier × redundancy_factor × (1 - compression))- Add operational ops charges (PUT/GET, lifecycle transitions, Glacier restore).

Compute (batch/stream)- Gather: job inventory, runtimes, concurrency, instance types, vCPU/RAM, autoscale patterns, fraction on spot/reserved, executor overhead.- Model: ComputeCost = Σ_jobs (runtime_hours × nodes_per_job × instance_hour_rate × (1 - discount) ) + managed service overhead (EMR/Dataproc/K8s control plane).- Include storage I/O and ephemeral disk costs, GPU pricing for heavy ML features.

Network- Gather: data ingress, inter-region replication, egress to customers, shuffle (between zones), cross-account pulls.- Model: NetworkCost = egress_GB × egress_rate + interregion_GB × interregion_rate + loadbalancer fees.

Uncertainty & validation- Instrumentation: use telemetry (historical CloudWatch/Stackdriver/Azure Monitor) for bytes, job runtimes, instance-hours to seed model.- Perform sensitivity analysis / Monte Carlo on key vars (ingest rate, compression, spot availability).- Build scenario rows: baseline, growth (x%), worst-case.

Public signals to validate estimates- Cloud provider pricing pages (AWS S3/Glacier, EC2/EBS, Data Transfer; GCP/Azure equivalents) — authoritative unit rates.- Provider TCO/whitepapers and public calculators (AWS Cost Explorer docs, GCP pricing docs).- Engineering blog posts / conference slides (Databricks, Netflix, Airbnb, Snowflake talks) for real-world job runtimes, compression ratios, and architecture patterns.- Open benchmarks (TPC-DS, MLPerf, Spark/EMR performance tests) to validate instance-hour per TB processing.- GitHub repos / papers describing feature store implementations (Feast, Hopsworks) for storage/serving patterns.- Public cost breakdowns in investor decks / case studies for companies operating PB-scale data.

Deliverables & ops- A parametrized spreadsheet + simple simulator that takes telemetry or assumptions and outputs monthly line items + sensitivity bands.- Dashboards showing top contributors, cost per TB processed, cost per feature served.- Runbook: how to update model with new telemetry and how to use model to trade off retention, tiering, or reserved capacity.

Example quick sanity check formula (per-month):Total = Σ_tiers(GB_tier × $/GB_month) + Σ_jobs(hours_job × $/hr) + (egress_GB × $/GB)

This approach balances concrete telemetry, provider rates, and public benchmarks to reduce uncertainty and support actionable optimizations (tiering, spot vs reserved, locality).

HardTechnical

40 practiced

You discover the company runs all data in a single cloud region and their public docs don't mention DR. Propose a disaster recovery (DR) plan for data pipelines and analytics warehouses that specifies RTO and RPO targets for critical datasets, cross-region replication strategies (synchronous vs asynchronous), and a test plan to validate failover without data loss.

Sample Answer

Requirements & constraints:- Assume single-region production today, need multi-region DR for pipelines, warehouse, metadata, and streaming. Balance cost vs criticality; business must define acceptable downtime/cost.

RTO/RPO (by dataset tier):- Tier 1 (core billing, customer profiles): RTO ≤ 1 hour, RPO ≤ 5 minutes.- Tier 2 (nearline analytics used in daily ops): RTO ≤ 4 hours, RPO ≤ 1 hour.- Tier 3 (historical, archival): RTO ≤ 24 hours, RPO ≤ 24 hours.

Replication strategy:- Streaming (Kafka/Kinesis): cross-region replica cluster + asynchronous replication with mirror consumers; if vendor supports geo-replication with exactly-once semantics use that. Synchronous across regions causes high latency — only acceptable for a tiny set of Tier 1 writes.- OLTP/source DBs: For Tier 1, use cross-region synchronous replication only if latency tolerable; otherwise use async replication with transaction log shipping and point-in-time recovery.- Data lake (object storage): Cross-region object replication (async, versioned) with lifecycle; enable object immutability for critical datasets.- Warehouse (Redshift/BigQuery/Snowflake): Native cross-region replication (e.g., snapshots/copies). For Tier 1, continuous streaming of CDC into standby region so standby can rebuild quickly; for others, periodic scheduled snapshots.- Metadata & orchestration (Airflow/metadata DB): Multi-AZ and cross-region read replicas; promote replicas on failover. Store DAGs in versioned object store replicated across regions.- Idempotency and schema evolution: Ensure pipelines are idempotent and support deduplication to allow replay.

Failover & runbook:- Automated detection + alerting trigger for failover window; runbook steps to promote standby datasets, switch DNS/endpoints, reconfigure ingestion to target standby region, validate pipelines, and cut traffic.- Rollback plan and communication templates.

Test plan to validate failover without data loss:- Quarterly DR drills with two types: 1) Planned failover rehearsal: Promote standby region, redirect a subset of traffic, run full pipeline DAGs, validate outputs. 2) Chaos tests: Simulate region outage in staging to validate automation and reclaim timelines.- Validation checks post-failover: - Schema & row-count parity for critical tables. - Checksums/hashes on partitioned data (compare last N windows). - End-to-end reconciliation: upstream message offsets vs consumed offsets; ensure no gaps. - Business-level sanity queries (key KPIs).- Measurement: record actual RTO/RPO achieved, discrepancy with targets, and remediation items.- Backups & retention: Regular immutable snapshots retained cross-region for >30/90 days depending on compliance.

Operational needs:- Monitoring: replication lag, consumer lag, checksum drift, snapshot success, runbook state machine.- Automation: IaC to provision standby, blue/green DNS, feature flags to flip endpoints.- Cost vs risk trade-offs documented per dataset; get business sign-off.

This plan provides clear RTO/RPO tiers, pragmatic synchronous vs asynchronous choices, concrete replication mechanisms, and a repeatable test suite to prove failover without data loss.

MediumSystem Design

24 practiced

An article states the company's largest monthly cost is long-term storage of raw data in object storage. Propose a retention and tiering strategy that balances analyst needs, regulatory retention, and cost: include retention windows, tier mapping (hot/cold/archive), access patterns, lifecycle rules, and how you'd get buy-in from analysts and compliance.

Sample Answer

Requirements & constraints:- Keep data available for analysts for fast queries up to a short window, meet regulatory minimums (e.g., 7 years), and reduce monthly object-store cost.- Access patterns: most reads are recent (<90 days), occasional historical lookups (90 days–2 years), rare audits (>2 years).

High-level retention + tiering proposal:- Hot (0–90 days): S3 Standard / SSD-backed data lake for daily analytics and ETL. Low latency, full read/write.- Warm (91–365 days): S3 Intelligent-Tiering or Standard-Infrequent Access (S3-IA) for lower cost but moderate latency. Keep index/metadata in hot.- Cold (1–2 years): S3 Glacier Instant Retrieval or Glacier Flexible Retrieval for batch historical analyses; retrieval minutes–hours.- Archive (>2 years up to regulatory retention, e.g., 7 years): S3 Glacier Deep Archive for audit/compliance — retrieval hours.

Lifecycle rules (AWS example):- Put all incoming raw objects in S3 Standard.- Lifecycle rule 1: Transition objects >90 days → Intelligent-Tiering / S3-IA.- Lifecycle rule 2: Transition objects >365 days → Glacier Instant Retrieval (or Glacier Flexible Retrieval depending on SLA).- Lifecycle rule 3: Transition objects >2 years → Glacier Deep Archive.- Expiration rule: Delete objects after regulatory retention window (e.g., 7 years) unless marked as legal_hold/tagged.

Access & metadata strategy:- Maintain compact metadata catalog in a fast store (e.g., DynamoDB/Glue catalog) with object locations and tags (sensitivity, retention class, owner). This enables catalog-only queries and selective restores.- For common historical queries, keep partitioned Parquet snapshots in warm tier to reduce restores from Glacier.- Implement on-demand restore API that auto-promotes restored objects to warm tier for N days.

Cost & SLA trade-offs:- Glacier Deep Archive saves ~70–90% vs Standard; trade-off is retrieval latency and per-GB restore cost. Use for rarely accessed, compliance-only data.- Intelligent-Tiering avoids lifecycle guesswork for mid-term.

Operational controls & governance:- Enforce retention through object tags (source, retention_period, legal_hold).- Prevent accidental deletion via bucket policies and MFA-delete for critical buckets.- Monitor with CloudWatch metrics: storage bytes per tier, restore frequency, retrieval costs, and number of accesses by age bins. Set alerts for unexpected access spikes.

Getting buy-in (analysts & compliance):- Run a 6-week pilot: pick a slice of data and apply tiering; measure cost savings and retrieval times; surface real-world impacts.- Create SLAs: define expected retrieval latency per tier and provide a self-serve restore portal for analysts with request tracking.- Involve compliance early: map regulatory requirements to retention/hold logic; implement audit logs (S3 access logs, AWS CloudTrail) and immutable retention for regulated datasets.- Offer training and docs: show how to tag data, request restores, and query partitioned snapshots to avoid restores.- Governance forum: monthly reviews with analysts, legal, and product owners to tune windows and handle exceptions (e.g., extend retention via tags).

Example lifecycle JSON snippet (conceptual):- Objects older than 90d -> transition to IA- Older than 365d -> transition to Glacier- Older than 730d -> transition to Deep Archive- Delete at 7 years unless tag legal_hold=true

This balances low-latency analyst needs for recent data, cost-effective storage for historical/compliance data, and transparent policies and tools so analysts and compliance trust and adopt the model.

Unlock Full Question Bank

Get access to hundreds of Understanding the Company's Infrastructure Context interview questions and detailed answers.

Join thousands of developers preparing for their dream job.