Storage Services: Object, Block, and File Storage Questions

Know the main storage categories. Object Storage (S3, Azure Blob, Cloud Storage) is best for unstructured data, files, backups, and large-scale storage; it's infinitely scalable and cost-effective but accessed by key, not mounted like filesystems. Block Storage (EBS, Managed Disks, Persistent Disks) attaches to VMs; ideal for OS and database storage; provides high performance. File Storage (EFS, Azure Files, Filestore) provides shared access across multiple instances; useful for shared data and collaborative workflows. Understand use cases: Object storage for data lakes and backups; block storage for database disks; file storage for shared application data. Know that object storage pricing is very low compared to other options.

MediumSystem Design

72 practiced

You need to support metadata-rich search over millions of media files stored in object storage. Propose an architecture that stores metadata, indexes searchable fields, and keeps metadata synchronized with objects. Discuss trade-offs of embedding metadata in objects vs an external DB or search index.

Sample Answer

Requirements (functional + non-functional):- Searchable over millions of media objects by metadata fields (text, tags, dates, numeric, facets).- Low-latency full-text + faceted search (sub-second queries).- Near-real-time sync when objects/metadata change.- Strong scalability, availability, and eventual consistency acceptable.- Auditability and bulk re-indexing support.

High-level architecture:Client/API Gateway → Metadata Service (CRUD) → Event Bus (Kafka or SNS/SQS) → Indexer Workers → Search Engine (OpenSearch/Elasticsearch) + Metadata DB (Postgres/Cassandra) → Object Storage (S3/GCS). UI queries go to Search Engine; object fetches use object storage URLs.

Core components and responsibilities:1. Metadata API: validate, enrich, store canonical metadata in DB; write-change emits events.2. Event Bus: durable event stream for decoupling, replay for re-indexing.3. Indexer Workers: consume events, transform and upsert documents into search engine; handle deletes and schema evolution.4. Search Engine: inverted indexes and aggregations for fast search/facets.5. Object Store: holds binary; store object reference (bucket/key) in metadata.6. Reconciliation Job: periodic diff between DB and index; support full re-index.

Data flow:- Create/update metadata -> write DB -> emit event -> indexer updates search index.- Object uploads: either metadata created first or object upload triggers notification -> metadata service links and emits event.

Sync consistency & failure handling:- Use at-least-once event processing with idempotent indexing (doc IDs based on object ID + version).- Store change version in DB; indexer checks version to avoid stale writes.- Use dead-letter queue and monitoring for failed events; reconciliation job recovers missed updates.

Trade-offs: embedding metadata in objects vs external DB/index- Embedded metadata (object metadata headers or sidecar JSON in object): - Pros: single source, object-level atomicity, easier to back up/replicate with object. - Cons: limited queryability (object stores are not search engines), expensive to scan/reindex, metadata size limits, slower for complex searches.- External DB + Search index (recommended): - Pros: optimized for queries, rich indexing (full-text, facets), separate scaling for search vs storage, easier schema evolution, audit/history. - Cons: eventual consistency complexity, extra components to manage, risk of divergence (mitigated with events + reconciliation).Recommendation:- Use canonical metadata in a transactional DB + OpenSearch for search. Emit events on changes via Kafka/SNS and use idempotent indexers plus periodic reconciliation. For small immutable attributes needed at read-time (e.g., content-type), store as object metadata for quick retrieval, but keep searchable fields in the external index.

HardSystem Design

70 practiced

Design a low-latency data pipeline to feed large ML training jobs that require high throughput from datasets stored in object storage. Include design for data sharding, local NVMe caches, prefetching, parallel reads across nodes, and orchestration to ensure GPUs are not starved. Discuss how to measure and optimize throughput.

Sample Answer

Requirements & constraints:- High-throughput, low-latency feeding of large datasets in object storage (S3-compatible) to many GPUs across cluster nodes.- Ensure GPUs are never starved, support scaling to hundreds of GPUs, tolerate node failures.

High-level architecture:- Data lake (S3) + manifest service → Shard manager → Orchestrator (Kubernetes + custom controller) → Per-node Data Agent (prefetcher + local NVMe cache) → Worker (training process on GPU).- Metadata store (etcd/Postgres) stores shard assignments, offsets, checksums, metrics; monitoring (Prometheus + Grafana).

Sharding & placement:- Precompute deterministic sharding (e.g., hash of sample id) into N shards where N >> total GPUs to allow flexible assignment.- Shard manager assigns shards to node-local Data Agents, balancing by data size and node capacity. Use consistent hashing for minimal rebalance on scale events.

Local NVMe cache & prefetching:- Data Agent maintains an LRU-backed NVMe cache with block-level objects (large files chunked, e.g., 64–256MB).- Two-tier prefetch: proactive (ahead-of-consumption window per worker) and reactive (on-demand parallel read). Prefetcher reads next K chunks asynchronously from S3 using multipart parallelism and writes to NVMe.- Validate with checksums; background eviction maintains headroom.

Parallel reads across nodes:- Use range GETs and multipart downloads; tune concurrency per node to saturate network without overwhelming S3 (limit via token bucket).- When multiple nodes need same data (e.g., common validation set), use coordinated cache sharing via sidecar that can serve peers over RDMA/HTTP/NGINX with peer-read fallback.

Orchestration and GPU starvation prevention:- Orchestrator monitors per-GPU consumption rate and Data Agent buffer fill level. Implement backpressure: training pods request tokens from orchestrator before consuming new minibatches; orchestrator ensures tokens only when buffer > threshold.- Autoscale Data Agent prefetch concurrency and number of data-serving sidecars based on real-time metrics.- Prioritize eviction/replica placement for hot shards.

Measuring & optimizing throughput:- Key metrics: bytes/sec served to GPUs, cache hit ratio, prefetch queue depth, end-to-end batch readiness latency, S3 read ops/sec, GPU utilization, tail latency percentiles (p50/p95/p99).- Measure with Prometheus exporters on Data Agents and training pods; correlate GPU utilization with data throughput.- Optimization levers: - Increase NVMe cache size or chunk size to reduce metadata overhead. - Tune prefetch window and parallel download concurrency. - Use instance types with higher network bandwidth; enable ENA/GPUDirect for RDMA where available. - Use warmed replicas of hot shards across nodes. - Compress serialized records (e.g., TFRecord with Snappy) to trade CPU decompression vs network IO. - If S3 becomes bottleneck, introduce a regional read-through cache (e.g., MinIO gateway or object cache tier).

Trade-offs:- Larger chunk sizes reduce GET overhead but increase wasted IO on partial reads.- Strong consistency (single-writer shard) simplifies bookkeeping; more replicas increase availability but raise cache coherence complexity.- Prefetch aggressiveness trades memory/NVMe/Network for lower latency.

Failure modes & mitigation:- Node failure triggers shard reassignment via consistent hashing; warm standby replicas prioritized.- Stale or corrupted cache entries validated with checksums and auto-refreshed.

This design balances throughput, scalability, and GPU protection using deterministic sharding, aggressive local NVMe caching with smart prefetching, parallelized S3 reads, and orchestrated backpressure and monitoring to keep GPUs fed.

EasyTechnical

67 practiced

Describe the POSIX semantics that file storage (NFS/SMB) provides and key limitations of managed cloud file services. Explain why some applications that rely on strict POSIX behavior fail when moved to cloud-managed file systems.

Sample Answer

POSIX semantics (as provided by traditional NFS/SMB) means a set of filesystem guarantees apps rely on:- Strong consistency for metadata and data (writes visible to subsequent reads in program order).- Atomicity of certain operations (rename is atomic, link/unlink behaviors).- File descriptor vs. pathname semantics: operations on an open FD are independent of concurrent renames/unlinks.- Advisory/per-file locks and flock/posix-lock semantics with predictable conflict behavior.- Byte-range I/O ordering and visibility guarantees.

Managed cloud file services (EFS, Azure Files, Filestore, etc.) often implement a subset of POSIX or relax guarantees for scalability and multi-tenant performance. Key limitations:- Eventual or weak consistency on metadata or directory listings across clients.- Non-atomic or different rename/link semantics (e.g., rename across mounts fails or is non-atomic).- Different locking implementations (no true POSIX locks, locks scoped per SMB session or not honored across protocols).- Inconsistent file-handle semantics: delete-while-open may behave differently.- Performance/latency variability and limits on metadata ops (high-churn workloads suffer).

Why apps fail when moved:- Applications that expect atomic rename for safe replace (e.g., editors, database snapshots) can see partially written files or stale listings if rename isn't atomic.- Programs using POSIX file locking for coordination will race or corrupt state when locks aren't global/strong.- Build systems, job schedulers, and databases often depend on immediate visibility of writes; weak consistency causes unexpected behavior and data corruption.- Workflows that rely on file-descriptor semantics (open->unlink->continue writing) break if cloud service implements delete semantics differently.

Recommendation: assess app file patterns (rename/locking/metadata rate), test on target managed service, or use block storage with clustered filesystems or deploy application-aware services (managed databases/object stores) when strict POSIX semantics are required.

HardTechnical

81 practiced

Design a performance benchmarking plan for a multi-tenant storage service to validate IOPS, throughput, and tail latency under production-like loads. Include workload generation, tenant isolation tests, metrics to capture, and how you would detect resource interference between tenants.

Sample Answer

Goal: validate IOPS, throughput, and tail (99th/99.9th) latency of a multi-tenant storage service under production-like loads, and detect cross-tenant interference.

1) Requirements & constraints- Define SLOs: e.g., 99th latency < 50 ms for small random reads, sustained throughput X MB/s per tenant, baseline IOPS.- Environment: use staging cluster with identical software, similar hardware and network; test both isolated and co-located tenant mixes.

2) Workload generation- Use realistic mix: small random reads/writes (4–16KB), large sequential reads/writes (64KB–1MB), metadata ops.- Model tenant behaviors: bursty (spiky IOPS), steady (throughput-heavy), mixed.- Tools: fio for block-level; YCSB or custom clients for object/metadata; k6 for API-level.- Drive workloads from distributed generators to avoid client-side bottlenecks; use workload scripts to replay production traces (sampling APM logs).

3) Tenant isolation tests- Baselines: run each tenant workload alone to capture per-tenant performance envelope.- Pairwise / combinatorial co-location: co-locate tenants with conflicting profiles (e.g., heavy random write + heavy read).- Scale tests: increase tenant count until degradation or resource saturation.- Noisy-neighbor injection: introduce one “attacker” tenant that intentionally blasts IOPS/throughput.

4) Metrics to capture- Storage: IOPS (read/write), MB/s, queue depth, active ops, latency distribution (p50/p90/p99/p99.9), service time, retry counts.- System: CPU, memory, disk utilization, disk scheduler stats, block device queue length, NIC throughput, interrupts, context switches.- Platform: per-tenant cgroup/io.stat, kernel blkio, storage pool usage, cache hit rate, GC/paging events.- Tracing/observability: per-request tracing IDs, end-to-end latency, request size, tenant ID tags.

5) Detecting resource interference- Correlate per-tenant latency spikes with system resource metrics (queue depth, disk busy %) and co-located tenant activity.- Anomaly detection: threshold and statistical methods (CUSUM) on p99 latency vs baseline.- Causal inference: run controlled experiments toggling suspect tenant load while monitoring target tenant; use difference-in-differences to confirm interference.- Use kernel-level accounting (cgroups/blkio, I/O latency histograms) to attribute I/O wait to tenants.

6) Analysis & reporting- Present SLO compliance, degradation curves vs tenant count, heatmaps of latency vs co-located tenant profiles.- Recommend mitigations: QoS (IOPS/throughput limits), token-bucket shaping, prioritized scheduling (bfq/blk-io-controller), cache partitioning, elastic throttling.- Repeatability: provide test orchestration (Terraform/Ansible), versioned workload configs, artifacted logs and dashboards.

This plan offers reproducible, data-driven validation and clear next steps to harden multi-tenant isolation.

EasyTechnical

66 practiced

Explain the difference between eventual consistency and strong consistency in object storage systems. Provide a real-world example where eventual consistency could cause surprising behavior for an application (e.g., after an overwrite or delete), and how you would mitigate it.

Sample Answer

Strong consistency means once a write returns success, all subsequent reads (from any client) immediately reflect that write. Eventual consistency allows reads to return stale data for a window of time; the system guarantees that if no new updates occur, all replicas will converge to the same value eventually.

Key differences:- Latency vs. freshness: strong consistency trades higher read/write latency or availability for up-to-date reads; eventual consistency favors low latency and high availability but permits short-term staleness.- Failure behavior: under partitions, systems offering eventual consistency remain available; strongly consistent systems may block or reject requests.

Real-world surprising behavior:Imagine an object store with eventual consistency. A client uploads object “image.jpg”, then immediately uploads a new version (overwrite) or deletes it. Another client polling the object might still read the old version or see the object present after delete, because some replicas haven’t applied the second operation yet. This can break workflows (e.g., billing systems charging for resources that were deleted) or UIs showing deleted items.

Mitigations (practical, architect perspective):- Use strong-consistency APIs where available (read-after-write guarantee) for critical operations.- Implement read-after-write by routing reads to the primary/leader replica when supported.- Use versioning and conditional operations (ETags, If-Match) to detect and prevent acting on stale data.- Add short retry with exponential backoff and validation: after write/delete, poll until a read confirms state or timeout.- Design idempotent, compensating actions: treat operations as eventual and include reconciliation jobs to correct inconsistencies.- Communicate SLA/consistency model to clients and document expected windows of staleness so application logic can adapt.

Trade-offs: prefer eventual consistency for high-throughput, global-scale reads; choose strong consistency for correctness-sensitive paths and combine both where appropriate.

Unlock Full Question Bank

Get access to hundreds of Storage Services: Object, Block, and File Storage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.