Data and Technical Strategy Alignment Questions

Evaluates a candidate's ability to reason about the technical and architectural trade-offs that shape a data platform: batch versus streaming (and hybrid) pipelines, data warehouse versus data lake versus lakehouse architecture, ETL versus ELT, schema design and partitioning for analytics and ingestion, data contracts between producers and consumers, feature stores, and metrics (semantic) layers. Good answers pick a concrete architecture or approach for a stated scale, latency, and cost profile, name the trade-offs of the alternatives, and justify the choice rather than reciting definitions.

HardTechnical

44 practiced

For an anti-fraud model with strict correctness requirements, discuss whether to design downstream analytics with eventual consistency or strong consistency. Provide a design that balances the need for fast detection, accuracy, and throughput, and indicate how you'd deal with false positives due to eventual visibility.

Sample Answer

Clarify requirements: fraud detection needs low-latency decisions (block/flag in real time), high accuracy (minimize false positives/negatives), and high throughput. For downstream analytics (e.g., aggregation, dashboards, rule-updates) use a hybrid consistency design: strong consistency for decision-critical state; eventual consistency for non-critical analytics and ML feature stores used for retraining.

Design:- Online decision path (real-time model + feature store): keep strongly consistent state for user/account risk scores and blacklists in a low-latency store (e.g., single-master Redis/etcd or strongly-consistent DB region). All blocking/decline decisions consult this authoritative store synchronously.- Stream layer: emit every event to a durable log (Kafka). Real-time processors (Flink/Beam) build enriched features and alerts; they can operate with event-timestamped, exactly-once semantics but write aggregated analytics to downstream stores with eventual visibility (Cassandra/S3).- Async reconciliation: periodic strong-consistency joins reconcile eventual aggregates back to authoritative stores; any drift triggers corrective actions (e.g., retract/override flags).Handling false positives from eventual visibility:- Default to conservative enforcement: synchronous decision store governs enforcement; downstream eventual views only influence non-blocking workflows (investigation queues, throttling scores).- Provide automated rollback/appeal paths: when reconciliation finds a false positive, issue automated remediation (release holds, credit user) and log for model retraining.- Confidence thresholds + human-in-the-loop: low-confidence detections from eventual pipelines route to manual review; high-confidence ones from strong-consistent pipeline can auto-action.Trade-offs:- Strong consistency on the critical path increases latency/operational cost but ensures correctness for blocking. Eventual consistency improves throughput and scalability for analytics and model training.- Monitor metrics: decision latency, false positive rate, reconciliation lag; tune thresholds and reconciliation frequency to balance cost vs correctness.

HardTechnical

46 practiced

Technical coding: Given a Kafka topic of events and a Flink job in Scala, write pseudocode that performs stateful enrichment keyed by user_id, deduplicates by event_id, and writes results to a transactional sink. Explain the state backend, checkpoint interval, and parallelism considerations you choose.

Sample Answer

To solve this, use keyed state by user_id, maintain per-user enrichment state and a dedup set for recent event_ids (e.g., TTL MapState). Use Flink checkpoints + a two-phase-commit transactional sink (Flink 2PC sink or Kafka transactional producer) for exactly-once.

scala

// pseudocode - Scala Flink
import org.apache.flink.streaming.api.scala._
import org.apache.flink.api.common.state._
import org.apache.flink.streaming.api.functions.KeyedProcessFunction
import org.apache.flink.util.Collector

case class Event(userId: String, eventId: String, payload: Map[String, Any])
case class EnrichedEvent(...)

val env = StreamExecutionEnvironment.getExecutionEnvironment
env.enableCheckpointing(30_000) // 30s
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(10_000)
env.setStateBackend(myStateBackend) // see explanation
env.setParallelism(8)

val kafkaStream: DataStream[Event] = // read from Kafka with exactly-once source

kafkaStream
  .keyBy(_.userId)
  .process(new KeyedProcessFunction[String, Event, EnrichedEvent] {
    // per-user enrichment cache
    lazy val enrichmentState: ValueState[UserProfile] = getRuntimeContext.getState(
      new ValueStateDescriptor("profile", classOf[UserProfile])
    )
    // dedup store with TTL (MapState eventId -> timestamp)
    lazy val seenIds: MapState[String, Long] = getRuntimeContext.getMapState(
      new MapStateDescriptor("seen", classOf[String], classOf[Long])
    )

    override def processElement(ev: Event, ctx: Context, out: Collector[EnrichedEvent]): Unit = {
      if (!seenIds.contains(ev.eventId)) {
        // mark seen with current processing time; TTL configured on state backend or managed manually
        seenIds.put(ev.eventId, ctx.timerService().currentProcessingTime())
        val profile = enrichmentState.value()
        val enriched = enrich(ev, profile)
        out.collect(enriched)
      } else {
        // duplicate -> drop
      }
      // optionally register timer to cleanup old seenIds
      ctx.timerService().registerProcessingTimeTimer(ctx.timerService().currentProcessingTime() + cleanupMs)
    }

    override def onTimer(ts: Long, ctx: OnTimerContext, out: Collector[EnrichedEvent]): Unit = {
      // remove entries older than retention
      // iterate seenIds and remove by timestamp
    }
  })
  .addSink(new TwoPhaseCommitSink(...)) // transactional sink (exactly-once)

Key choices and reasoning:- State backend: RocksDBStateBackend for large per-user state, durable disk-backed state, supports incremental checkpoints. Use filesystem (S3/HDFS) for checkpoints and savepoints.- Checkpoint interval: 30s default; balance latency vs overhead. Use minPauseBetweenCheckpoints (10s) and checkpoint timeout (e.g., 10s-1min) tuned to throughput and sink commit latency.- Dedup TTL: Keep event_id TTL equal to max out-of-order + retention window (e.g., 24h) to bound state size; use MapState with TTL or periodic cleanup timers.- Parallelism: Set parallelism based on key distribution and cluster resources. KeyBy(user_id) means skew is possible — use key salting or hotspot mitigation for heavy users. Ensure number of Kafka partitions >= parallelism for good read parallelism.- Sink: Use a 2PC/Transactional sink implementing checkpointCommit protocol to guarantee exactly-once writes (e.g., FlinkKafkaProducer with transactions or custom TwoPhaseCommitSink).- Fault tolerance: Use frequent incremental checkpoints, RocksDB incremental checkpointing, and test with failover scenarios.

Edge cases:- Hot keys causing skew — shard keys or separate processing for heavy users.- Very large dedup sets — use probabilistic structure (Bloom filter) plus occasional reconciliation.- Late events beyond TTL — decide policy (drop or reprocess).

MediumSystem Design

37 practiced

Design an architecture for a feature store that supports offline historical features for training and a high-throughput online store for inference. Explain how you maintain feature parity, freshness SLAs, and reconcile backfills with live writes. Include technologies you might use.

Sample Answer

Requirements:- Offline store for historical features (training/validation), online low-latency key-value store for inference.- Strong feature parity between offline and online representations.- Freshness SLAs (e.g., <1s for critical signals, <5m for others).- Support backfills and reconcile with concurrent live writes.

High-level architecture:- Ingestion layer (stream + batch) → Feature computation layer → Serving/Stores: - Offline store: columnar data lake (Parquet on S3) + feature registry metadata. - Online store: low-latency KV DB (Redis/KeyDB, DynamoDB, or ScyllaDB). - Materialization service to sync computed features to online store. - Streaming engine for realtime updates (Kafka + ksql/Fluent/Beam). - Orchestrator (Airflow, Argo Workflows) for batch pipelines and backfills. - Feature registry (metadata, schemas, transformations) and CI tests.

Key components & responsibilities:- Feature definitions in repo (code-first): single source of truth; transform code used for both batch and streaming.- Compute engines: Spark/Beam for batch historical joins; Flink/Beam for streaming incremental updates.- Materialization: job that writes feature vectors to offline store and issues upserts to online store via bulk + per-key incremental streams.- Serving API/gRPC layer to fetch latest feature vector, with fallback to computed-on-demand for cold keys.

Maintaining parity:- Use same transformation library (shared SDK) compiled into batch and streaming jobs to avoid logic drift.- CI checks: unit tests, integration tests that compare sample outputs between batch and streaming on the same input.- Feature registry enforces schema, types, and retention policies; store feature versioning and lineage.

Freshness SLAs:- Categorize features by freshness tier; routes: - Hot features: update via streams into online store (Flink processing) with sub-second end-to-end. - Warm features: micro-batch materialization every few minutes.- Monitor lag metrics (Kafka offsets, processing latency) and alert on SLA breaches.

Reconciling backfills with live writes:- Backfill process writes to offline store and produces a bulk snapshot to a staging namespace in the online store (namespace+version).- Dual-write strategy optional: producers write to both online and event log; materializer consumes event log.- Reconciliation steps: 1. Pause consumer-side serving of stale namespace (or use versioned keys). 2. Apply backfill in atomic swap fashion: write backfilled data to staging keys, validate checksums, then flip feature version pointer (feature registry) so reads start using backfilled keys. 3. Replay live events that occurred during backfill onto staging keys (use persisted changelog offsets) to avoid lost updates.- Maintain watermarks/offset checkpoints so replay is bounded and idempotent. Use compare-and-swap or optimistic concurrency on online store for safe upserts.

Technologies:- Storage: S3 + Parquet/Delta Lake/Iceberg for offline.- Batch: Spark or Dataproc; Delta/Iceberg to support ACID backfills.- Streaming: Kafka + Flink/Beam for low-latency transforms and exactly-once semantics.- Online DB: Redis Cluster for ultra-low latency or DynamoDB/Scylla for higher durability at scale.- Orchestration: Airflow/Argo + GitOps for feature code.- Observability: Prometheus/Grafana (latency, lag), DataDog, Great Expectations for data quality.

Trade-offs:- Exactly-once vs lower latency: aim for idempotent writes and event-sourcing to reconcile.- Complexity of atomic swaps vs acceptable short window of divergence — choose based on model sensitivity.

This design ensures a single source of transformation logic, automated testing for parity, tiered freshness guarantees, and safe backfill reconciliation with replayed live writes.

HardSystem Design

45 practiced

Design a multi-region, low-latency event-driven inference pipeline for personalized recommendations that must remain available during regional failures. Include data replication strategy, cross-region caches, model synchronization, and how you would handle GDPR data residency constraints.

Sample Answer

Requirements:- Low tail latency (<50–100ms) for inference globally- Event-driven (streaming user events → real-time personalization)- Multi-region active-active with regional failure tolerance- GDPR: user data residency and right-to-be-forgotten- Model freshness within minutes for personalization

High-level architecture:- Edge ingest via regional API Gateways → regional event broker (Kafka/Cloud Pub/Sub)- Regional inference service (stateless microservices + GPU/TPU pools) behind local LB- Per-region feature store (online) + cross-region replicated cold store (data lake)- Cross-region cache layer (read-through Redis/KeyDB clusters with CRDTs/consistent hashing)- Async model distribution service (model registry + delta model patches)- Control plane for failover, policy/GDPR enforcement, monitoring

Data replication & privacy:- Raw PII/events stored only in home region (write-local). For global features, store aggregated non-PII replicas to other regions using transformation pipelines that pseudonymize or strip identifiers.- Use asynchronous CDC to replicate only allowed columns; encrypt data at rest with region-specific keys (KMS).- For GDPR delete/rectify, maintain authoritative mapping in home region and propagate deletions via event tombstones; caches respect TTL and on-delete invalidate entries immediately via pub/sub invalidation topics.

Cross-region caches:- Serve reads from local Redis. Populate via read-through from local feature store.- Use async background sync of non-sensitive aggregates between regions with CRDT/last-writer-wins for eventual consistency.- On regional failure, traffic from affected region routed to nearest remaining region; caches there serve best-effort using replicated aggregates; degrade gracefully (fallback to cohort-level features).

Model synchronization:- Training (offline) performed centrally or per-region depending on data residency. Two-tier models: - Global base model trained on non-PII or region-approved aggregated data. - Regional fine-tuned adapters trained on local sensitive data (kept in-region).- Model registry stores versioned artifacts and small adapter deltas. Distribute via CDN/Artifact store to regions; inference composes base+adapter at runtime.- Use atomic rollout with canary + shadow testing; rollback by versioned endpoints.- Ensure deterministic seeding and schema compatibility; use checksums and signatures.

Failure modes & DR:- Active-active: health checks and DNS-based traffic steering (Geolocation + latency-aware).- If region fails: automated re-route, spin up additional inference capacity in target regions; apply rate-limiting and degrade personalization granularity.- Data loss: use cross-region backups of non-PII; authoritative home-region backups for GDPR-sensitive must be restored within compliance windows.

Operational considerations:- Observability: request traces, feature drift, model quality metrics (A/B, real-time cohort metrics), SLOs for latency/availability.- Security: mTLS, per-region KMS, VPC peering/private links for replication.- Compliance workflows: data residency manifest per user, audit logs for data access, automated erase propagation and proof-of-deletion reports.

Trade-offs:- Strong consistency across regions increases latency — use eventual consistency for non-critical features, keep critical user profile reads local.- Keeping adapters local preserves privacy but increases orchestration complexity.

This design balances low-latency, regional privacy constraints, and high availability by combining local authoritative data, replicated non-sensitive aggregates, composable models, and automated GDPR-compliant dataflows.

EasyTechnical

36 practiced

Explain the difference between ETL and ELT in the context of AI-driven products. Describe situations where you would choose ELT over ETL when building training pipelines for large-scale models, considering data volume, compute locality, transformation complexity, and downstream analytics needs. Include a short example workflow for each approach.

Sample Answer

ETL vs ELT — concise definitions:- ETL (Extract, Transform, Load): data is extracted from sources, transformed (cleaning, feature engineering, aggregation) in a staging layer or ETL engine, then the processed data is loaded into the target (warehouse or ML store). Good when transformations are heavy, schema must be enforced before consumption, or target storage is compute-limited.- ELT (Extract, Load, Transform): data is extracted and loaded raw into the target (often a scalable data lake/warehouse with compute), and transformations happen in-place using the target’s compute (SQL, Spark, or cloud-native engines). Good when storage+compute are scalable and you want flexible, iterative transforms.

When to choose ELT for training pipelines (large-scale models):- Data volume: very large raw datasets (multi-TB/PB) where moving pre-transformed data is costly — load once into a lake and transform near compute.- Compute locality: training runs on cloud GPU clusters co-located with the data lake/warehouse (minimize egress, exploit data locality).- Transformation complexity: transformations are parallelizable and can run in distributed engines (Spark, Databricks, BigQuery). Also helpful when transformations are iterative during feature experimentation.- Downstream analytics: multiple consumers need raw + derived views (researchers, monitoring, retraining); ELT preserves raw data for re-use and lineage.

When ETL is preferable:- Strict schema/enforcement before loading (regulatory constraints), lightweight datasets, or when target cannot handle heavy transformations.

Short example workflows:

ETL workflow (preprocess before load):1. Extract logs from app DB.2. Run batch ETL on dedicated cluster: cleaning, sessionization, tokenization, feature extraction.3. Load cleaned feature tables into model store (feature store) and start training on GPUs.

ELT workflow (load raw, transform in target):1. Extract raw logs and load to cloud data lake/warehouse (Parquet in S3 or BigQuery).2. Use distributed jobs (Spark/SQL) near the data to create training shards, augment, and serialize TFRecords or Arrow files.3. Launch training on GPU cluster reading data directly from the same cloud storage.

Key trade-offs: ELT favors scalability, flexibility, and experimentation; ETL favors control, pre-validation, and lighter targets. Choose based on data size, locality, transformation pattern, and governance needs.

Unlock Full Question Bank

Get access to hundreds of Data and Technical Strategy Alignment interview questions and detailed answers.

Join thousands of developers preparing for their dream job.