Designing data storage, integration, and processing architectures. Topics include relational and NoSQL database design, indexing and query optimization, replication and sharding strategies, data warehousing and dimensional modeling, ETL and ELT patterns, batch and streaming ingestion, processing frameworks, feature stores, archival and retention strategies, and trade offs for scale and latency in large data systems.
HardSystem Design
45 practiced
Architect a vector embedding index for nearest-neighbor search at large scale: billions of vectors, sub-50ms query latency, and high update throughput. Discuss indexing choices (HNSW, IVF + PQ/OPQ), memory vs disk tiers, sharding and replication, batching queries, recall vs throughput trade-offs, and how you would support incremental updates and deletions without full reindexing.
Sample Answer
Requirements & constraints:- Billions of d-vectors, sub-50ms p99 query latency, high update throughput, strong recall (configurable), multi-tenant resilience.High-level architecture:- Ingest layer (write buffer + WAL) → Indexing nodes (shards) → Query API + Router → Storage tier (memory SSD) → Background index maintenance workers. Use Faiss / custom C++ for fast kernels; orchestrate with k8s and GPUs for heavy rebuilds.Indexing choices:- HNSW: excellent recall/latency for in-memory serving; heavy RAM per vector (approx 2–8x). Use for “hot” shards where latency is critical.- IVF + PQ/OPQ: compressed, low-RAM (disk-backed) good for cold/capacity tier. IVF reduces candidate set, PQ/OPQ quantizes vectors to reduce memory and I/O; OPQ improves quantization quality.- Hybrid: store full HNSW graph for hot subset (recent/high-QPS vectors); store compressed IVF+OPQ for bulk cold storage. Query both and merge top-k.Memory vs disk tiers:- Tier 1 (RAM): HNSW graph + vector cache for top-K hot vectors; guarantees sub-50ms.- Tier 2 (NVMe SSD, memory-mapped): IVF centroids + PQ codes + async disk hits; used when Tier1 misses.- Persistent store (object store / DB): raw vectors and metadata (S3 + RocksDB for mappings).Sharding & replication:- Shard by vector id hashing or semantic routing (ANN locality-preserving). Keep shard sizes to fit HNSW in RAM (e.g., 50–200M vectors/shard).- Stateful replica pairs: one leader for writes (applies updates), multiple read replicas for queries. Use quorum or async replication for updates depending on consistency needs.- Dynamic resharding: monitor load, split/merge shards with online migration (stream vectors to new shards while forwarding queries).Batching & query flow:- Router fans-out queries to relevant shards (all-shard for global). Use batched vector-matrix multiplication on GPU for first-pass ANN or use CPU SIMD for HNSW hops.- Batch queries to leverage BLAS/GPU: group ~16–128 queries to amortize cost; ensure latency SLO by adaptive batching (small batch at high QPS).- Early-stopping: use configurable efSearch (HNSW) / nprobe (IVF) to trade recall vs latency.Recall vs throughput trade-offs:- Increase efSearch / nprobe and PQ precision → higher recall, higher latency/IO.- Use multi-stage: cheap coarse filter (IVF nprobe small) → rerank top-N with exact distance on full vectors or HNSW re-search.- Expose SLAs per tenant; auto-tune knobs based on latency window and desired recall.Incremental updates & deletions (no full reindex):- Write path: append to WAL and in-memory write buffer. For new vectors: - Insert into HNSW online (fast local graph updates). - Insert PQ codes into IVF codeset for disk tier.- Deletions: tombstones in metadata + lazy physical removal. Mark vector id deleted; exclude in search and periodically prune graph/compact (background worker).- Re-insertion/upserts: treat as delete + insert; for HNSW maintain per-shard background repairs to reconnect neighbors.- For IVF+PQ, support delta-logs: store incremental PQ codes separately and consult delta during queries; periodically merge (compaction) into main index offline or on less-loaded windows.- Background maintenance workers: rebuild subgraphs or re-quantize only affected partitions (partial reindexing). Use incremental re-clustering for IVF centroids: update centroids with streaming k-means or maintain a buffer of new vectors until a threshold triggers local recluster.Operational considerations:- Telemetry: latency, recall (A/B against exact), load per shard, RAM/IO.- Autoscaling: spawn additional read-replicas or split shards when p99 or CPU/IO thresholds breached.- Backups & consistency: persist WAL and object store snapshots; allow point-in-time recovery.- Testing: synthetic workloads with mixed queries/updates; evaluate recall vs latency curves and tune ef/nprobe/PQ bits.Summary:Combine HNSW for hot low-latency slices and IVF+OPQ for high-capacity compressed storage; use sharding + replicas for scale; implement write-ahead buffers, tombstones, delta-merge strategy, and background partial rebuilds to support high update throughput and deletions without full reindexing. Tune efSearch/nprobe and batching to hit sub-50ms while meeting recall targets.
EasyTechnical
86 practiced
List the pros and cons of using JSON-based event payloads for transport versus typed binary formats like Avro or Protobuf in the context of streaming ML pipelines. Discuss human-readability, payload size, schema enforcement, compatibility, and serializer/deserializer overhead during high-throughput ingestion.
Sample Answer
JSON (text) vs typed binary (Avro/Protobuf) — pros and cons for streaming ML pipelines:Human-readability- JSON: Highly readable, easy to debug, inspect logs and replay events without tooling.- Avro/Protobuf: Not human-readable; requires schema/tools to interpret, making ad-hoc debugging harder.Payload size- JSON: Verbose (field names, quotes), larger network/storage footprint — increases network IO and storage costs.- Avro/Protobuf: Compact binary encoding, significantly smaller payloads — lower bandwidth and faster disk IO.Schema enforcement & validation- JSON: Schemas optional (e.g., JSON Schema); weak enforcement by default — risks inconsistent records, latent data-quality bugs.- Avro/Protobuf: Strongly typed schemas with required/optional fields and types; enforce contract at producer/consumer, safer for feature pipelines.Compatibility & schema evolution- JSON: Flexible but ad-hoc compatibility; producers can change shape without notice — consumers must defensively handle missing/extra fields.- Avro/Protobuf: Built-in versioning rules (back/forward compatibility) when used correctly — safer evolution for long-lived pipelines.Serializer/deserializer (CPU) overhead at high throughput- JSON: Fast to parse in many libs but larger payloads increase I/O cost; parsing text into types is CPU work and GC-heavy in some runtimes.- Avro/Protobuf: More efficient binary parsing and less data to move; however, (de)serialization code generation and copying can add overhead but typically lower end-to-end CPU+IO cost at scale.Recommendation (AI-engineer view)- For prototyping and debugging: JSON for speed of iteration.- For production high-throughput pipelines: Avro/Protobuf (with schema registry) to reduce size, enforce contracts, and enable safe schema evolution. Consider hybrid: use JSON at ingress in dev, convert to binary in ingestion layer.
MediumTechnical
43 practiced
Create a retention and archival policy for AI training datasets, derived features, and model artifacts that balances regulatory requirements (e.g., auditability), reproducibility, and storage cost. Include classification tiers, TTLs, archival stores (e.g., S3 IA / Glacier), retrieval SLAs, and processes for legal holds and immutable snapshots for experiments.
Sample Answer
Overview: Define three classification tiers (Hot, Warm, Cold), assign TTLs, archival targets, retrieval SLAs, and processes for reproducibility, auditability, legal holds, and immutable experiment snapshots.Classification tiers:- Tier 1 — Hot (raw training data + active model checkpoints + recent derived features) - TTL: 90–180 days active; after that evaluate for promotion to Warm - Store: SSD-backed object store (S3 Standard / equivalent) - Retrieval SLA: minutes - Rationale: needed for ongoing iteration and fast retraining.- Tier 2 — Warm (validated datasets, production models, derived feature stores used in production) - TTL: 1–3 years - Store: S3 Standard-IA or regional low-cost block storage with lifecycle to Cold - Retrieval SLA: hours - Rationale: reproducibility and auditability for regulatory windows.- Tier 3 — Cold (older training snapshots, full experiment artifacts, aggregated derived features) - TTL: 3–7 years (configurable by dataset criticality) - Store: S3 Glacier Instant / Flexible Retrieval or Glacier Deep Archive - Retrieval SLA: 6–48 hours (depending on archive class) - Rationale: long-term retention at minimal cost for audits/forensics.Reproducibility & auditability:- Mandatory manifest per experiment: dataset version IDs, preprocessing code hashes, feature-store snapshot IDs, random seeds, environment/container image digest, and model hyperparameters. Store manifests in append-only ledger (e.g., object storage with object lock + versioning).- Periodic integrity checks: checksums and automated verification jobs every 30/90 days.Immutable snapshots & experiments:- For each experiment finalization, create an immutable snapshot: bundle manifest + dataset snapshot + model checkpoint + Docker image digest, then apply object-lock governance/retention (WORM) for the applicable TTL.- Store experiment bundles in Tier 2 or Tier 3 depending on age/importance.Legal holds & compliance:- Legal hold toggles prevent lifecycle transitions and deletions; admin + legal role required to release. Holds apply at dataset, model, or experiment level and override TTL.- Audit trail: every hold/release and access must be logged in an immutable audit log (centralized SIEM + cloud audit logs).Cost controls & automation:- Lifecycle policies automate transitions (Hot → Warm → Cold) with alerts before transition for datasets with ongoing active jobs.- Tagging: enforce tags (sensitivity, retention_policy, owner, regulatory_category) and periodic policy compliance reports.- Budget guardrails: automated notifications when storage cost per project exceeds thresholds; option to archive to cold or request extended retention approval.Retrieval & governance workflow:- Retrieval requests for Cold/Glacier must include justification and expected SLA; restoration triggers costs and is logged.- Emergency restore process for legal or high-priority incidents with expedited retrieval option (higher cost, approval workflow).- Access control: least-privilege IAM roles; separate roles for retrieval/restore approvals.Metrics and review:- Quarterly review of retention settings by ML privacy/compliance and owners; adjust TTLs by regulatory changes.- KPIs: average retrieval time, cost-per-GB-month by tier, number of holds, successful reproducibility checks.This policy balances auditability and reproducibility (manifesting immutable snapshots and logs) with cost (tiered storage, lifecycle automation), and provides clear legal hold and retrieval processes.
MediumTechnical
53 practiced
Describe a layered approach to enforce data quality in production pipelines: validation at ingestion, unit tests for ETL, profiling and distribution checks, anomaly detection, and automated correction workflows. Mention specific tools you would use (e.g., Great Expectations, TFDV, custom checks) and the metrics to surface via dashboards.
Sample Answer
Layered approach (defense-in-depth) to enforce data quality in production pipelines:1) Ingestion-time validation- Purpose: catch bad schemas/poisoned records as close to source as possible.- Tools: TFDV for schema inference and batch/stream schema validation; custom lightweight validators in Kafka/Beam that drop/log malformed records.- Checks: schema conformance, nulls, type coercion, allowed ranges, provenance/timestamp freshness.2) ETL/unit tests- Purpose: prevent regressions during transformations.- Tools: Great Expectations integrated into CI (pytest + GE suites) and unit tests for transformation logic.- Checks: row counts, unique key constraints, referential integrity, distribution sanity, deterministic transformation tests.3) Profiling & distribution checks- Purpose: baseline data characteristics and detect drift.- Tools: TFDV for feature histograms & skew, Great Expectations for expectations on distributions, custom jobs to compute PSI/KL, embedding drift (cosine/similarity) for NLP/vision features.- Metrics: feature means/std, null rate, cardinality, PSI per feature, KL divergence, embedding centroid shift.4) Anomaly detection & monitoring- Purpose: surface unexpected shifts in near-real time.- Tools: Prometheus + Grafana for metrics; custom detectors (e.g., streaming Z-score, EWMA) or ML-based detectors (isolation forest) for multivariate anomalies.- Metrics to surface: ingestion error rate, % invalid rows, delta of feature means, PSI, volume spikes, freshness/latency, upstream source health, data quality score (composite).5) Automated correction & workflows- Purpose: remediate common issues or quarantine for human review.- Tools: Airflow/Luigi or Cloud Composer to orchestrate rollback/reload; serverless functions for auto-imputation/repair; Great Expectations actions to send alerts/quarantine datasets.- Strategies: auto-fill missing with model-based imputations, backfill from source, apply last-known-good transformations, and create manual review ticketing.Dashboard and alerting priorities- Real-time: invalid rows/sec, ingestion error rate, pipeline latency, data freshness.- Periodic: feature PSI/KL (drift), null rates, cardinality changes, label distribution shift, embedding drift, data quality score trend, % pipelines healthy.- Alerts: threshold breaches (e.g., invalid rate > 1%), sudden PSI > 0.2, model input drift > threshold, sustained latency increases.Why this matters for AI systems- Model performance is highly sensitive to subtle data shifts (feature drift, label skew, embedding drift). Combining schema checks, rigorous ETL tests, profiling, anomaly detection, and automated remediation minimizes silent failures and preserves model reliability and reproducibility.
EasyTechnical
52 practiced
Compare row-oriented and columnar storage formats in the context of ML data pipelines. Specifically, explain when to use Parquet, Avro, and JSON for different stages (raw ingestion, feature engineering, analytics, and training). Discuss compression, read/write patterns, schema evolution, and how each format impacts downstream ETL/ELT and training performance.
Sample Answer
High-level summary:- Row-oriented (Avro, JSON) store full records together — good for write-heavy, streaming, and small-read workloads. Columnar (Parquet) stores columns separately — excellent for analytic reads, vectorized IO, and ML training where you read subsets of features.By stage:- Raw ingestion: Use JSON (or Avro for binary) as landing format. JSON is human-readable, flexible for logs/events; Avro is compact, faster to parse, and supports schema registry for guaranteed typing. Both accept append-heavy writes.- Feature engineering / ETL: Use Avro or Parquet depending on workload. For row-wise transformations and streaming pipelines, Avro is convenient (fast writes, record-level). For batch feature stores and heavy column operations (selecting, aggregating), convert to Parquet for efficient column pruning and predicate pushdown.- Analytics: Parquet is preferred — columnar compression, faster scans, lower IO for wide tables and ad-hoc queries.- Training: Parquet or specialized formats (TFRecord, RecordIO). Parquet works well when training reads many columns/features with batching and benefits from vectorized reads; TFRecord/RecordIO or sharded Avro may be better for streaming GPU pipelines and when you need serialized tensors.Key considerations:- Compression: Parquet achieves higher compression ratios and better CPU/runtime decompression thanks to column-wise encodings (RLE, dictionary). Avro supports block compression but is less space-efficient for columnar data. JSON is largest and slowest.- Read/write patterns: Parquet optimizes sequential column scans and selective column reads; poor for frequent small updates. Avro/JSON are good for frequent appends and record-level updates.- Schema evolution: Avro has explicit schema and strong support for backward/forward compatibility via schema registry. Parquet supports schema metadata and evolves but can be trickier (renames, nested changes). JSON is flexible but can create downstream parsing/consistency pain.- Downstream ETL/ELT & training impact: Using Parquet for batch stages reduces IO and speeds feature selection and model training. Start with Avro/JSON for raw ingest to preserve fidelity and ease ingestion, then normalize into Parquet after validation. Ensure lineage, schema registry, and partitioning (by date, shard) to optimize reads and GPU pipeline feeding.Practical rule: Ingest JSON/Avro → validate & canonicalize → store features/analytics in Parquet → export sharded TFRecord/optimized binary for high-throughput training.
Unlock Full Question Bank
Get access to hundreds of Data Architecture and Pipelines interview questions and detailed answers.
