Cloud Storage Types and Trade-offs Questions

Direct answer

Store file bytes in object storage (for example Amazon S3, Google Cloud Storage, or Azure Blob Storage), addressed by content hash rather than a human path, and own everything that describes where a file lives, who can see it, and what happened to it in a separate, strongly-consistent metadata service, sharded by tenant. File storage (a mounted, POSIX, Portable Operating System Interface, filesystem) is not the right primary layer here: nothing in the requirements needs a kernel-level mount, and a POSIX metadata server does not shard across tens of millions of files as cheaply or as simply as an object store plus an external index. A hybrid is worth introducing only for one narrow slice: real-time collaborative co-editing of a single document, which needs an ordering/locking layer in front of the same object store, not a wholesale switch to file storage.

Structured elaboration

Storage choice: object vs. file vs. hybrid. Object storage wins on the stated requirements for three independent reasons: it scales to tens of millions of keys without an administrator provisioning capacity, a single PUT of immutable content is naturally idempotent and safely retryable, and content-addressing (using a hash of the file's bytes as its storage key) gives cross-tenant deduplication of identical files for free at the byte layer, while the metadata layer keeps each tenant's own reference and quota accounting completely separate (two tenants uploading the same PDF still each get billed for their own copy in quota terms, even though only one physical blob is stored). File storage would only earn its keep if a client needed to mount the filesystem and use POSIX operations like flock or atomic rename directly, which nothing in this system does: access is through an API or a sync client, not a kernel mount. The hybrid case is collaborative editing: a Google-Docs-style co-edit needs low-latency ordered writes that neither raw object PUTs nor a POSIX mount handle well, so that narrow feature gets its own operational-transform (a technique that merges concurrent edits by mathematically adjusting each incoming edit against the others made at the same time, so every client converges on the same result) or CRDT (conflict-free replicated data type) service that periodically checkpoints into the same object store, rather than pulling the whole file-sharing system onto a shared filesystem.

Metadata service. A horizontally sharded relational store (for example Postgres, sharded by tenant_id), not a plain key-value store. The reason is that ACL checks, quota enforcement, and folder listings are all transactional, indexed, point-and-range queries, and a key-value store handles those awkwardly. Core tables: Tenant, Folder, File (current-version pointer), FileVersion (append-only, one row per upload), ACLEntry (an access control list entry: a row recording that a given subject has a given permission, such as read or write, on a given resource; subject, resource, permission), and an append-only AuditEvent log. Sharding by tenant_id means a shard boundary never splits one tenant's data, which keeps every per-tenant operation (a quota check, a full-tenant export, an offboarding delete) inside a single shard.

Per-tenant isolation. tenant_id is derived server-side from the authenticated session, never accepted as client input, and is threaded through every metadata query and every object-key prefix (tenant_id/blob_hash). That is logical isolation by default. For enterprise or compliance tenants that need physical isolation, the same schema supports promoting a tenant to a dedicated bucket/prefix and, if it grows large enough to be a noisy neighbor, a dedicated metadata shard.

Quotas and enforcement. Track usage as a per-tenant counter updated in the same transaction as the metadata write that consumes quota (the FileVersion insert), using an atomic conditional update (increment-and-check, reject if it would exceed the limit) rather than a separate read-then-write. A read-quota-then-write-file sequence is a classic time-of-check-to-time-of-use (TOCTOU) race: two concurrent uploads can both read "under quota" and both proceed, overshooting the limit. Doing the check and the increment as one atomic database operation closes that race.

Shared folders and per-file ACLs. ACLEntry rows attach to either a folder or an individual file. A file without its own explicit ACL row inherits its folder's effective ACL; an explicit file-level ACLEntry overrides the inherited one. Because permission reads (every file open) vastly outnumber ACL writes (an occasional share action), maintain a materialized "effective ACL" per file, recomputed and cached on write, and invalidated whenever any ancestor folder's ACL changes. A shared folder is just a Folder row with one or more non-owner ACLEntry rows granting access to other tenant members; nothing else in the model changes to support sharing.

Versioning. FileVersion rows are immutable and append-only, each pointing at an immutable, content-addressed object. Upload order matters for correctness: write the blob to object storage first (safe to retry, since the same bytes hash to the same key), then insert the FileVersion row, and only then flip the File row's current-version pointer. A crash between any of those steps leaves the system in a recoverable state; it never exposes a half-written file as current.

Audit trails. A separate, write-once AuditEvent stream (actor, action, resource, tenant_id, timestamp), written via a transactional outbox (the event row is written to a dedicated outbox table inside the very same database transaction as the metadata change, then a separate process reads that outbox table and reliably delivers each event) alongside the metadata mutation it describes, so an audit event cannot be silently dropped if the downstream log write fails independently of the database commit. Audit events are never updated or deleted in place.

Scaling and eventual consistency, applied selectively. Object storage scales natively; the metadata store is the real bottleneck at tens of millions of files, which is why it is sharded by tenant up front rather than retrofitted later. A separate search/listing index (for example a denormalized folder-listing table or a text-search index), fed asynchronously from the metadata store, is where "eventual consistency" belongs: a search result that is a few seconds stale is a minor UX issue. ACL checks and quota decisions must not be eventually consistent: a revoked permission that is still visible for a few seconds in an eventually-consistent index is a live authorization hole, not a UX nuance. The question's "offer eventual consistency" requirement is satisfied by the index and cross-region replication layers, deliberately, not by relaxing the transactional core.

flowchart TD
    Client[Sync client or web app] -->|authenticated request| API[API / Auth layer]
    API -->|derive tenant_id from session, never trust client input| Meta[(Metadata service:
sharded Postgres by tenant_id)]
    API -->|content-addressed PUT/GET| Obj[(Object storage:
prefix = tenant_id/blob_hash)]
    Meta -->|async outbox| Audit[[Append-only audit log]]
    Meta -->|async projection| Index[(Search / listing index)]
    Audit --> Cold[Cold storage export]
    Index -.eventually consistent.-> API

Worked example

A concrete shard-sizing pass for "tens of millions of files," with versioning and ACLs accounted for, not just raw file count:

# Pinned assumptions (explicitly assumptions, not measurements):
files = 50_000_000
avg_versions_per_file = 3   # versioning requirement
avg_acl_rows_per_file = 2   # owner + one explicit share; folder defaults cover the rest

file_rows = files
version_rows = files * avg_versions_per_file
acl_rows = files * avg_acl_rows_per_file
print(f"File rows: {file_rows:,}  FileVersion rows: {version_rows:,}  ACLEntry rows: {acl_rows:,}")

target_rows_per_shard = 50_000_000   # keep autovacuum/index rebuild/PITR replay tractable per shard
shards_needed = -(-version_rows // target_rows_per_shard)   # ceil, sized on the LARGEST table
print(f"Largest table (FileVersion) = {version_rows:,} rows -> {shards_needed} shards "
      f"at a {target_rows_per_shard:,}-row-per-shard target")
print(f"-> ~{files/shards_needed:,.0f} files' worth of metadata per shard, "
      f"tenant_id hashed into {shards_needed} buckets so no shard splits one tenant")

Output:

File rows: 50,000,000  FileVersion rows: 150,000,000  ACLEntry rows: 100,000,000
Largest table (FileVersion) = 150,000,000 rows -> 3 shards at a 50,000,000-row-per-shard target
-> ~16,666,667 files' worth of metadata per shard, tenant_id hashed into 3 buckets so no shard splits one tenant

The takeaway: at this scale, the metadata layer, not the object store, is what forces a sharding decision, and the version-history table (not the file table itself) is usually the one that determines shard count, because it grows as a multiple of file count, not 1:1 with it.

Trade-offs and pitfalls

• Applying eventual consistency uniformly is a security bug waiting to happen. The requirement as stated ("offer eventual consistency") is easy to over-apply. Scope it explicitly to the search index and cross-region replication; keep ACL checks and quota decisions inside the strongly-consistent transactional core.
• Cross-tenant content deduplication saves storage bytes but must never leak quota or visibility. The object layer can share physical bytes across tenants; the metadata layer must still charge each tenant's own reference against their own quota and must never let one tenant's ACL grant visibility into another tenant's row pointing at the same blob.
• Unbounded ACL-inheritance chains get expensive without a cache. Walking from a file up through every ancestor folder on every read does not scale; the materialized effective-ACL cache is not optional at this file count, but its invalidation (a folder-level ACL change must fan out to every descendant file's cache entry) is real engineering work, not a footnote.
• A single large, hyperactive tenant on a shared shard is a noisy-neighbor risk. Monitor per-shard load by tenant, not just aggregate shard load, and have a documented path to move one tenant to a dedicated shard before it degrades everyone else on that shard.

Direct answer

Kubernetes persistent volume (PV, the cluster-level object representing a piece of provisioned storage) access modes describe how many nodes can mount a volume at once and for what purpose, not how many pods: ReadWriteOnce (RWO) restricts mounting to a single node for read-write access, ReadOnlyMany (ROX) allows many nodes to mount read-only at once, and ReadWriteMany (RWX) allows many nodes to mount read-write at once. A commonly missed detail is that RWO is a single-node guarantee, not a single-pod guarantee: multiple pods scheduled on that same node can still share an RWO volume, which is exactly the gap a fourth mode, ReadWriteOncePod (RWOP), was added to close by restricting a volume to exactly one pod cluster-wide.

Access modes mapped to underlying storage

Worked example

Consider a StatefulSet running a single-instance database with an RWO volume. Because RWO only guarantees exclusivity at the node level, a rare but real failure mode exists: if the scheduler places a replacement pod on the same node as a not-yet-terminated old pod during a bad rolling update, both pods could mount the same RWO volume at once, since they satisfy "one node" even though there are now two writers. Switching that volume's claim to ReadWriteOncePod removes this failure mode entirely, because the Kubernetes control plane enforces that only one pod anywhere in the cluster can hold the volume, regardless of which node it lands on. This distinction rarely matters for a stateless or read-heavy workload, but it matters a great deal for a workload that assumes single-writer correctness, which is why RWOP exists as its own mode rather than being folded into RWO.

Trade-offs and pitfalls

• Assuming RWO means "only one pod can ever use this volume" is the single most common misunderstanding of this feature, and it can hide a real correctness bug until two pods happen to land on the same node at the same time.
• RWX support depends entirely on the underlying storage backend; block storage generally cannot satisfy RWX at all (a small number of specialized multi-attach block products aside), so choosing RWX in a PersistentVolumeClaim (PVC) without a compatible storage class behind it will simply fail to provision.
• File-storage-backed RWX volumes trade block storage's low latency for concurrent, multi-node access; a workload that assumes block-storage-grade latency will see real IOPS (input/output operations per second) and throughput (the rate of data moved per second, as distinct from how many operations happen per second) differences on file storage, especially under many small, random operations.
• ROX is less commonly used than the other three, and driver support for it is inconsistent; verify a specific storage class actually supports ROX before designing around it, rather than assuming any storage class that supports RWX also supports ROX.

Direct answer

Treat storage strategy as a per-service decision, not one platform-wide choice: the media path gets object storage fronted by a content delivery network (CDN), since it is high-throughput and read-heavy, while each stateful service's own database gets block storage attached directly to that service's compute, since transactional reads and writes need low, predictable latency. The organizing principle underneath both choices is the same: put durable, high-scale, read-mostly data in object storage near the edge, and put low-latency, single-writer transactional data on block storage physically close to the compute that owns it.

Architecture

flowchart LR
  C[Client] --> GW[API Gateway]
  GW --> MediaSvc[Media Service]
  GW --> OrderSvc[Order Service]
  MediaSvc --> CDN[CDN]
  CDN --> OBJ[(Object storage: media)]
  OrderSvc --> Cache[(In-memory cache)]
  OrderSvc --> DB[(Database on block storage)]
  DB --> Snap[(Snapshots)]

• Compute placement and data locality: each stateful service's block volume must live in the same Availability Zone (AZ) as the compute attached to it, since block volumes typically cannot attach across zones at all; the application layer that talks to that database should also be scheduled close to it, to avoid paying cross-AZ network latency on every query. The media service, by contrast, can be stateless and placed anywhere, since it never owns local storage directly, it only talks to the CDN and the object store behind it.
• Caching layers: an in-memory cache (such as Redis) in front of a service's hot transactional reads cuts both latency and the IOPS (input/output operations per second) load the block volume has to sustain, which matters directly for cost, since higher IOPS on managed block storage is a paid, provisioned resource. On the media side, the CDN is the cache: the object storage origin should rarely serve a live read directly once content is warm at the edge.
• Backup and restore: block volumes need a defined snapshot schedule with an explicit recovery point objective (RPO, how much data loss is acceptable) and recovery time objective (RTO, how long recovery is allowed to take); snapshots themselves are commonly stored in object storage under the hood, which is part of why object storage's own durability matters even for a block-storage-centric service. Media in object storage does not need backup in the traditional hardware-failure sense, since the store's own durability already covers that, but enabling object versioning (keeping prior versions instead of overwriting) is a separate, deliberate decision that protects against an application-level mistake, a different risk than a disk failing.
• Managed versus self-managed cost trade-offs: managed block and object services cost more per unit than raw self-hosted disks, but that premium buys away the operational burden of patching, replication, and failover engineering. For most teams, especially before they have dedicated storage-infrastructure expertise, that premium is worth paying. The crossover where self-managing (for example, running an open-source object store on owned hardware) becomes cheaper requires honestly costing the team's own engineering time to build and operate that reliability, not just comparing sticker prices of hardware against a managed service's list price; most teams underestimate that fully-loaded cost.

Worked example

Walk both paths through a single request each. A media upload hits the media service, which streams the file directly into object storage and returns a CDN-backed URL; subsequent reads of that media almost never reach the origin object store, since the CDN caches it at the edge close to each viewer. An order-read request hits the order service, which checks the in-memory cache first; on a cache hit, the block-storage-backed database is never touched for that request; on a cache miss, the database serves the read from its own low-latency volume in the same AZ as the service, and the result is written back into the cache for the next request. Instead of asserting a specific improvement number here, the way to know whether this design is actually working is to measure it directly: track p99 (99th percentile) read latency for the order path before and after introducing the cache, track the cache's hit rate over a real traffic window, track object storage cost per gigabyte served against what the same traffic would have cost the origin store without the CDN in front, and, separately, actually run a restore drill against a snapshot and record the real RTO achieved, not the documented target. Those four measurements, taken from the real system, are what would make "this redesign helped" a verified claim instead of an assumed one.

Trade-offs and pitfalls

• Letting compute and a block volume land in different AZs, whether by a scheduling mistake or by not pinning the constraint explicitly, adds real latency to every single query and, on many clouds, is not even possible since the attachment itself is zone-scoped.
• A cache introduces a staleness window by design; without an explicit time-to-live (TTL) or an active invalidation path on writes, "the cache made it faster" can quietly become "the cache made it wrong."
• Backup and restore plans are frequently written down and never drilled; an RPO and RTO that have never been tested against a real restore are a documented intention, not a verified capability, and that gap tends to surface at the worst possible moment.
• The managed-versus-self-managed calculation is easy to get wrong in both directions: overpaying for managed services the team could safely run itself, or underestimating the operational cost of self-managing storage without the staffing to do it reliably; the honest version of this trade-off requires pricing the team's own time, not just the infrastructure bill.

Direct answer

Each of these five workloads has a distinct access pattern once you look past the label: how many clients touch it, how latency-sensitive it is, and whether it needs a real filesystem. Working through them in order: object storage for the backups and the streaming video, block storage for the database boot disk, file storage for the shared home directories, and object storage again for the container registry (registries are built on object storage under the hood, even though "registry" sounds like its own category).

Selection and justification

Worked example

Workloads 1, 4, and 5 look different on the surface (backups, container layers, video files) but they share the pattern that makes object storage the right call for all three: written once (or rarely), read many times by many independent clients, and never edited in place. Workload 2 is the opposite pattern: one client (the database engine), constant small random reads and writes, and correctness depends on every write landing with low, predictable latency, which only a real block device delivers. Workload 3 is different again: many clients, but this time they need to write and see each other's changes through a shared, lockable namespace, which is what makes it a file-storage workload rather than an object-storage one, even though a naive read might call "documents people are collaborating on" an object-storage use case.

Trade-offs and pitfalls

• A common mistake is treating "container registry" as its own special category instead of recognizing the underlying access pattern (immutable, content-addressed, read-many); once you see the pattern, the object-storage answer follows the same reasoning as backups and video, not a special exception.
• For workload 2, choosing object storage because "it's just database files" would leave every query and every write latency-bound by network round trips instead of local disk access, which is the kind of mistake that only shows up under real load, not in a demo.
• For workload 3, file storage's convenience comes with an operational cost: someone owns the NFS or SMB server's capacity planning, permissions model, and backup strategy, which is more to manage than object storage's largely hands-off model.
• For workloads 1 and 5, remembering to configure lifecycle rules (automatically moving old backups to a colder, cheaper storage tier, and expiring logs past a retention window) is what actually captures object storage's cost advantage; leaving everything in the default, most-expensive tier forfeits most of the savings the choice was supposed to deliver.

Direct answer

Strong consistency means a read immediately after a write is guaranteed to return that write's result. Eventual consistency means a read immediately after a write might return an old value, a partial view, or in the case of a delete, the object that was supposedly just removed, until propagation catches up. The common interview framing of this question still describes Amazon S3 (Simple Storage Service) as eventually consistent for overwrites and deletes; that has not been true since December 2020, when AWS made S3 strongly read-after-write consistent for all PUT and DELETE operations in every Region. Google Cloud Storage's current documentation states the same strong-consistency guarantee for writes, deletes, and listing today. Unlike S3, where AWS documents an exact date (December 2020) for the change from eventual to strong consistency, GCS's own consistency documentation describes the guarantee without dating when it began; what is verifiable is that GCS is strongly consistent for these operations today. The genuinely surprising eventual-consistency behavior that still exists in modern object storage lives at the cross-region replica, not at the primary copy.

Structured elaboration

Why this question still gets asked with the old framing. Before December 2020, an application that overwrote an S3 object and immediately read it back could, in rare cases, get the old bytes, and an application that deleted an object and immediately listed the bucket could still see it. That is genuinely how S3 used to behave, and a large amount of still-circulating interview material and blog content describes it as current. A candidate who repeats it as today's behavior is repeating stale information; a candidate who can say "that was true before December 2020, here is what actually still exhibits that behavior" is demonstrating they track how the underlying systems evolve rather than memorizing a fact once.

Where the surprising behavior is real today: cross-region reads. Cross-Region Replication (CRR) and equivalent features on other providers copy an object to another region only after the write to the source region succeeds, and that copy is asynchronous. Azure Blob Storage makes this explicit with its RA-GRS (read-access geo-redundant storage) option: the secondary region is documented as typically behind the primary, with an interval called the recovery point objective (RPO, the maximum acceptable gap between the primary's last write and the secondary's last received write) that can be bounded to under 15 minutes with an opt-in feature, but is not zero.

Worked example

An e-commerce catalog service uploads a new product image to a bucket in us-east-1 and the object is replicated to eu-west-1 for European users. A backend job publishes the product listing immediately after the image upload succeeds, and a European user's first page load happens to be served from eu-west-1.

def catalog_publish_flow(image_written_in_primary: bool, replica_has_caught_up: bool) -> str:
    if not image_written_in_primary:
        return "write failed, do not publish"
    if replica_has_caught_up:
        return "safe to serve from any region"
    return "serving from the lagging replica returns 404 or a stale placeholder"

print(catalog_publish_flow(True, replica_has_caught_up=False))

Output:

serving from the lagging replica returns 404 or a stale placeholder

The image exists and is durable the instant the primary-region PUT succeeds, but a read routed to the not-yet-caught-up replica can 404 or serve an older object version for as long as replication lag lasts. Two concrete mitigations: (1) hold the catalog entry back from publishing until a replication-complete signal is received (S3 Replication Time Control can emit an event when a specific object has finished replicating, bounding this to minutes), or (2) route the FIRST read after any write for that key back to the source region for a short grace window, falling back to the nearest replica afterward.

Trade-offs and pitfalls

Waiting for a replication-complete signal before publishing adds latency to the publish path, which matters for a workflow with many images; a batched or asynchronous "go live" step that only flips visibility once replication is confirmed avoids blocking the uploader while still avoiding the stale-read window for end users. The bigger pitfall is architectural: designing a multi-region read path that assumes any region's copy is always current, because that assumption was arguably safe for single-region S3 access even before 2020 (once you allow for eventual consistency's rare-but-real primary-region behavior) and is definitely unsafe the moment a second region is in the picture, regardless of how strongly consistent any single region's copy is.