Storage Services and Data Management Questions

Know primary storage options: Object Storage (S3, Azure Blob, GCS) - for unstructured data at scale, highly available, cost-effective. Block Storage (EBS, Azure Managed Disks) - for VM storage, IOPS/throughput optimized. Databases - Relational (RDS, Azure SQL, Cloud SQL) for structured data with relationships; NoSQL (DynamoDB, Cosmos DB, Firestore) for flexible schemas and scale. Understand access patterns, durability, and consistency models. Know when to use each storage type based on data characteristics and access patterns.

EasyTechnical

64 practiced

List and categorize common storage access patterns in ML systems (e.g., large sequential reads for batch training, small random reads for online inference, high-throughput streaming writes for telemetry). For each pattern recommend appropriate storage types (S3/GCS, EBS, Redis, DynamoDB, HDFS) and explain how the access pattern influences partitioning, caching, and cost decisions.

Sample Answer

Below is a concise taxonomy of common ML storage access patterns, recommended storage types, and how the pattern drives partitioning, caching, and cost trade-offs.

1) Large sequential reads (batch training, epoch scans)- Characteristics: Read-heavy, large files (TFRecords/Parquet), high throughput, sequential access.- Recommended storage: S3/GCS or HDFS (object or distributed file store); EBS for smaller on-prem training clusters.- Partitioning & layout: Shard data by time/job or by dataset slice (shards sized to fit worker memory/IO parallelism). Use file-level partitioning (e.g., by date, label) to enable skipping.- Caching & cost: Use ephemeral SSD (local instance store/EBS) or distributed caching (Alluxio) for repeated epochs to reduce egress/GET costs. Optimize object size (100 MB+ objects typically cheaper per-op).- Notes: Favor throughput-optimized instances and parallel reads (multiple threads) to saturate network.

2) Small random reads (online inference, feature store lookups)- Characteristics: Low-latency, high QPS, small payloads.- Recommended storage: Redis / DynamoDB (in-memory / low-latency key-value). EBS-backed DB only if single-node and low latency acceptable.- Partitioning & layout: Key-based sharding (consistent hashing or DynamoDB partition keys) to distribute QPS and avoid hot partitions. Design keys to spread uniformly.- Caching & cost: Heavy use of caching (Redis) in front of DB; TTLs to control staleness. Redis costs higher per GB—cache only hot features. DynamoDB offers predictable throughput pricing; right-size read/write capacity or use on-demand to avoid throttling.- Notes: Monitor hot keys and implement request coalescing or local caches.

3) High-throughput streaming writes (telemetry, training logs, feature ingestion)- Characteristics: Many small writes, append-only, eventual consistency ok.- Recommended storage: S3/GCS (write via batching), Kafka + HDFS/S3 sink, DynamoDB for realtime feature ingestion if low-latency reads needed.- Partitioning & layout: Partition by time/producer to allow parallel ingestion; use batching to reduce per-request overhead.- Caching & cost: Buffering (Kafka, Kinesis) reduces direct storage ops and cost. For S3 sinks, larger multipart uploads reduce PUT cost.- Notes: Design schemas tolerant to duplicates and out-of-order events.

4) Large random reads/writes (model checkpoints, intermediate tensors)- Characteristics: Large blobs, occasional reads/writes.- Recommended storage: EBS for single-instance low-latency, S3/GCS for durable long-term storage.- Partitioning & layout: Store checkpoints as versioned blobs with consistent naming (jobID/epoch) to ease lifecycle policies.- Caching & cost: Keep recent checkpoints on EBS/instance store for fast restore; archive older ones to S3 Glacier to save cost.

How access patterns influence decisions (summary)- Partitioning: Drive parallelism and avoid hotspots—choose shard keys that evenly distribute load (time + id hashing).- Caching: Use caches where latency matters (Redis for inference), local SSD/EBS for repeated training epochs; cache only hot data to control cost.- Cost: Object stores (S3/GCS) are cheap per-GB but have per-operation costs and higher latency; Redis/EBS are expensive per-GB but provide low-latency access. Batch small ops to reduce request costs; use lifecycle/archival policies.

Example mapping quick reference:- Batch training: S3/GCS (+local SSD cache)- Online inference: Redis / DynamoDB (+local LRU cache)- Telemetry ingestion: Kafka -> S3 / HDFS- Checkpoints: EBS for fast restore; S3 for durable archival

This approach balances performance, scalability, and cost by matching storage semantics to access patterns.

HardTechnical

75 practiced

Discuss trade-offs between POSIX-like filesystems (NFS, EFS, Lustre) and object stores (S3-like) for ML workloads that require parallel reads/writes and operations like atomic rename. Address semantics such as atomic rename, directory listing costs, small-file overhead, metadata scalability, and how these differences affect distributed training frameworks and ingestion pipelines.

Sample Answer

Start with the core semantic difference: POSIX-like filesystems present strong POSIX guarantees (atomic rename, consistent directory semantics, byte-range updates), while object stores expose an “eventually-consistent” key-value model with immutable objects and no native atomic-rename or POSIX directory operations.

Key trade-offs and impacts:- Atomic rename: POSIX rename is atomic and cheap — critical for checkpointing and safe killer-resume in distributed training. On S3-like stores you must emulate rename by copy+delete (non-atomic) or write to a temp key and use application-level marker objects + list/check semantics; this complicates failure recovery and risks partial reads.- Directory listing costs: POSIX listings are metadata-server driven, low-latency. Object store list operations are expensive (control-plane, paginated) and slower; frameworks that rely on frequent list() for discovery (e.g., parameter servers, checkpoint pruning) will see latencies and higher request costs.- Small-file overhead: POSIX handles many small inodes efficiently; object stores incur per-object overhead, higher latency, and higher cost for millions of small files. For ML this penalizes feature-store shards, many-model checkpoints, or per-sample objects — mitigate by bundling (tar/shard/LMDB/TFRecord).- Metadata scalability: Distributed POSIX systems (Lustre, EFS) can scale metadata with dedicated metadata servers but face contention at very high metadata rates. Object stores shift scale to the control plane and are massively scalable for throughput but not for operations-per-second on metadata-heavy workloads.- Parallel reads/writes: Object stores excel at high aggregate throughput and parallel reads from many clients (immutable objects, CDN-friendly). POSIX filesystems provide low-latency byte-range reads and POSIX locks, helpful for fine-grained I/O and in-place updates.

How this affects frameworks and pipelines:- Distributed training: Checkpointing patterns prefer atomic rename and consistent listings — POSIX simplifies correctness. Using S3 requires careful write-commit protocols (write temp object, write success marker, clients read marker-first) and increased coordination. For read-heavy sharded datasets, S3 scales better across many workers if dataset is packaged into large objects (shards).- Ingestion pipelines: High-volume small-record ingestion favors batching before persisting to object stores. POSIX simplifies random writes and transactional updates for downstream processing.

Practical recommendations:- Use POSIX (EFS/Lustre/NFS) when you need POSIX atomicity, many small files, in-place updates, or simple checkpoint semantics.- Use object stores when you need massive throughput, cost-effective long-term storage, global availability, and can adapt pipelines (sharding, marker-based commits).- Hybrid: store active checkpoints and small-file metadata on POSIX-mounted fast storage; archive immutable artifacts to S3. Implement robust commit patterns (temp + manifest/mutex service) and bundle small files to minimize object-store overhead.

HardTechnical

61 practiced

A company stores 50 PB of training data across multiple buckets and regions and the monthly storage/egress bill is rising unsustainably. Propose a cost-optimization strategy covering tiering (hot/warm/cold), compression and deduplication, compute placement to minimize egress, scheduling to reduce peak costs, and retention/lifecycle policies. Provide numeric examples and discuss risks to agility and reproducibility.

Sample Answer

Approach: treat this as a layered storage+compute optimization problem: move infrequently used bytes to cheaper storage, shrink bytes via compression/dedup, avoid cross-region egress by co-locating compute, smooth training demand to reduce peak egress and transient costs, and apply retention/lifecycle rules to prune irrelevant data. I'll give concrete numbers assuming current 50 PB, cloud object storage tiers and typical egress pricing.

1) Tiering (hot/warm/cold)- Define hot = <30 days access / used for active experiments (10% = 5 PB), warm = 30–180 days (20% = 10 PB), cold = >180 days (70% = 35 PB).- Move hot to Standard ($0.023/GB-mo), warm to Infrequent/IA ($0.012/GB-mo + retrieval), cold to Archive ($0.004/GB-mo but retrieval latency).- Monthly storage example (approx): hot 5PB * $23/TB = $115k; warm 10PB * $12/TB = $120k; cold 35PB * $4/TB = $140k → total ~$375k/mo vs naive 50PB * $23/TB = $1.15M — ~67% saving on storage.

2) Compression & deduplication- Apply lossless compression on raw text/tabular and image codecs for images; use chunk-level dedupe (content-defined chunking + rolling hash) across buckets.- Conservative estimate: 2x compression for images/structured, 3–5x for logs. Global dedupe can remove replicated datasets (checkpoints, copies) — assume combined 2x–3x reduction.- If effective size falls from 50PB → 20–25PB, storage + egress costs drop proportionally.

3) Compute placement to minimize egress- Require “compute-in-region” policy: schedule training only where dataset resides; move model artifacts (small) across regions rather than datasets.- Employ shared regional caches (e.g., persistent regional filesystem or FSx/Filestore) and worker-local prefetch. For multi-region teams, replicate only required slices (e.g., 1–2% sample) across regions.- Example: avoid repeated cross-region reads of 30PB; each cross-region read at $0.09/GB would be catastrophic. By co-locating, egress reduces to model artifact transfers (~<1TB).

4) Scheduling & batching- Use spot/low-cost instances and schedule large training during off-peak to take advantage of lower network contention and burstable pricing.- Implement prefetch windows and dataset caching to amortize transfer: bulk transfer of new datasets once per week instead of many small transfers reduces per-request overhead.- Example: smoothing 100 TB/day spikes into steady 20 TB/day reduces peak egress charges and transient staging costs.

5) Retention & lifecycle- Implement metadata-driven lifecycle: only keep raw data for experiments that passed a quality threshold; keep derived features and model-ready shards longer.- Enforce automated pruning: delete raw intermediary artifacts older than X months, keep consumable curated datasets. Archive checkpoints older than 1 year.- Add legal/compliance exemptions.

Implementation steps- Inventory: quantify hot/warm/cold access by last-read metrics; compute per-object access frequency and size.- Pilot: enable tiering + compression on one dataset, measure compression ratio and retrieval latency.- Rollout: enable chunk-level dedupe via object-store lifecycle + external dedupe service; deploy region-aware scheduler in training orchestration (Kubernetes/Terraform).- Monitor: track storage by tier, egress bytes by region, and time-to-reproduce experiments.

Risks to agility & reproducibility- Cold/archival retrieval latency increases iteration time; cold storage can add hours to restore—mitigation: keep small working sets hot and snapshot metadata to rehydrate quickly.- Dedup/compression may complicate partial reads or streaming training; use chunked formats (TFRecords, LMDB) that support streaming compressed chunks.- Aggressive pruning risks losing data needed to reproduce experiments: mitigate by policy exceptions (tagged experiments, immutable archives for production models), and store provenance metadata (checksums, preprocessing pipelines, seeds).- Cost vs developer velocity tradeoff: quantify cost savings vs added latency; allow on-demand “fast restore” budget for experiments that need rapid iteration.

KPIs- Target 50–70% reduction in monthly storage+egress within 3–6 months.- Track mean time-to-rehydrate, reproducibility incidents, and egress GB/month by region.

This plan balances large cost savings (tiering + compression/dedupe) with preserving reproducibility via metadata, selective hot retention, and controlled restore workflows.

EasyTechnical

69 practiced

Given a transactions table with schema (transaction_id PK, user_id INT, amount DECIMAL, occurred_at TIMESTAMP), write a PostgreSQL-compatible SQL query that computes each user's mean and standard deviation over the past 365 days and flags transactions that are > mean + 3 * stddev. Explain how you handle users with fewer than two transactions in the window.

Sample Answer

Approach: For each transaction, compute the user's mean and sample standard deviation over the trailing 365-day window ending at that transaction's occurred_at using a correlated/lateral aggregate. Use stddev_samp (sample stddev). Flag only when stddev is available (>= 2 samples); otherwise do not mark the transaction as anomalous.

sql

SELECT
  t.transaction_id,
  t.user_id,
  t.amount,
  t.occurred_at,
  stats.mean_amount,
  stats.stddev_amount,
  CASE
    WHEN stats.stddev_amount IS NULL THEN false        -- fewer than 2 txns in window
    WHEN t.amount > stats.mean_amount + 3 * stats.stddev_amount THEN true
    ELSE false
  END AS is_anomaly
FROM transactions t
LEFT JOIN LATERAL (
  SELECT
    AVG(amount)        AS mean_amount,
    stddev_samp(amount) AS stddev_amount,
    COUNT(*)           AS cnt
  FROM transactions t2
  WHERE t2.user_id = t.user_id
    AND t2.occurred_at BETWEEN t.occurred_at - INTERVAL '365 days' AND t.occurred_at
) AS stats ON true;

Key points and reasoning:- Using a LATERAL subquery lets each transaction use its own window (past 365 days up to that row's timestamp).- stddev_samp returns NULL when fewer than 2 rows — we treat those as non-anomalous (no reliable variance estimate).- If you prefer to flag when amount > mean + 3*0 for single repeated-value histories, you could COALESCE stddev to 0, but that risks false positives; sample stddev is the statistically correct choice.- Performance: ensure an index on (user_id, occurred_at) for the correlated scans; for large data consider materialized aggregates or rolling-window precomputations.

MediumTechnical

55 practiced

Explain trade-offs between columnar formats (Parquet/ORC) and row-based formats (Avro/CSV) for feature engineering and training. Cover compression effectiveness, IO patterns, predicate pushdown, column pruning, and impact on query planning for analytics workloads.

Sample Answer

For ML workflows, choice between columnar (Parquet/ORC) and row-based (Avro/CSV) formats hinges on access patterns and analytics needs.

Compression effectiveness- Columnar: Much better for analytic/feature data because values in a column are homogeneous — runs of similar values compress well (dictionary, RLE, delta, bitpacking). This reduces storage and read bandwidth for selected columns.- Row-based: Less efficient for heterogeneous row fields; compression is applied across entire row blobs so savings are smaller when schema has many varied types.

IO patterns- Columnar: Optimized for scanning a subset of columns across many rows (sequential column chunks). Good for feature selection and batch training when only feature columns + label are read.- Row-based: Better when you frequently need whole records (all fields) or write/read single records (online feature lookups, streaming ingestion).

Predicate pushdown & column pruning- Columnar: Strong support — file formats store per-column statistics (min/max, null counts) and per-row-group metadata. Engines can skip row-groups (predicate pushdown) and read only required columns (column pruning), dramatically lowering I/O for filtering and feature sweeps.- Row-based: Limited; CSV has no built-in metadata, Avro supports schema but not efficient column pruning or rich per-block statistics—engines often must scan full rows.

Impact on query planning for analytics workloads- Columnar: Query planners can plan narrow scans, push filters early, avoid expensive deserialization for unused columns, and exploit vectorized execution. This lowers CPU and I/O cost for feature engineering (aggregations, joins) and for hyperparameter sweeps that run many similar queries.- Row-based: Planners treat scans as wider, more I/O-bound, and may perform full-table reads and heavier deserialization, increasing latency for big analytic queries.

Practical ML implications- Use Parquet/ORC for training data lakes, large-batch feature engineering, offline feature stores and cross-feature joins — faster reads, cheaper compute.- Use Avro or compact binary rows for online feature stores, streaming sinks, or row-wise writes where single-record reads/writes matter.- Hybrid: store raw ingested events in row format for append-heavy pipelines, transform to columnar for downstream training and analytics. Also partitioning and proper sort-order (e.g., by event time or label) improves predicate effectiveness.

Edge cases: very high-cardinality columns compress less; too many small files negates columnar benefits—use file consolidation and appropriate row-group sizes.

Unlock Full Question Bank

Get access to hundreds of Storage Services and Data Management interview questions and detailed answers.

Join thousands of developers preparing for their dream job.