Caching Strategies & In-Memory Optimization Questions

Direct answer

Score each candidate resource on how expensive it is to refetch, how often it's actually used, how large it is, how sensitive it is, and how much a slow load would visibly stall the user, then let that score decide whether it lives in a small memory cache, a larger disk cache with a time-to-live (TTL, how long a cached entry stays valid before it must be refreshed), or never gets cached at all.

Structured elaboration

The five factors, defined plainly:

• Fetch cost: how expensive it is to regenerate this data, a large hero image or a computed recommendation list costs far more than re-requesting a small JSON payload.
• Frequency of access: how often the same resource gets read, a feed thumbnail scrolled past repeatedly is a very different case from a one-time onboarding screen.
• Size: bytes it occupies. Low-end devices can have under 2GB of RAM and limited free storage shared with the OS and other apps, so size matters more here than on flagship hardware.
• Privacy: sensitive data (auth tokens, personal reading history, anything regulated) needs stricter handling (short lifetime, encryption, or exclusion from general caches) regardless of how well it scores on the other axes.
• User-perceived latency sensitivity: does a slow load visibly stall an interaction the user is looking at right now (a feed image), or is it invisible background work (a prefetch for a screen not yet opened)?

A simple framework: enumerate the resource classes in the app, score each 1 to 5 on the first three axes (cost, frequency, latency sensitivity), treat privacy as a hard override rather than a score, then assign a tier:

• Memory cache: small, frequently reused, latency-sensitive (evicted under memory pressure, cheapest and fastest).
• Disk cache with TTL: larger or less frequently reused, survives app restarts, revalidated periodically.
• Never cache (or cache only under explicit conditions): highly sensitive data, or anything large and rarely reused, where storing it costs more than refetching it.

Worked example

Prioritization for a news app on a low-end device:

Trade-offs and pitfalls

A static score goes stale as usage patterns shift (a story that goes viral suddenly gets reread constantly), so pair the framework with a runtime eviction policy like least-recently-used as a backstop rather than trusting the initial scoring forever. On low memory devices, over-caching images without an explicit cap is the single most common cause of out-of-memory (OOM, the app being killed because it tried to use more memory than the OS would allow) crashes, so set a hard limit (NSCache.totalCostLimit on iOS, a bounded LruCache on Android) instead of assuming the platform will always evict in time.

Design

Two tiers: a small, fast in-memory cache of decoded images (bounded by byte size, LRU eviction), backed by a larger on-disk cache of the original downloaded bytes (also bounded, also LRU). A memory eviction demotes an entry to disk instead of discarding it outright, since re-decoding from a locally cached file is far cheaper than a full network refetch; a disk eviction discards for real, since that's the last tier before the network.

The eviction/promotion logic, actually executed

The core cache logic below is pure JavaScript with no native dependency, so it runs as-is and the output is real, not illustrative.

// Core eviction/promotion logic for a two-tier (memory + disk) image cache.
// This part is pure logic, no native APIs, so it is genuinely runnable and the
// output below is real. The React Native-specific wiring (decode off the UI
// thread, prefetching, native-loader fallback) is shown separately below as
// architecture pseudocode, since it needs the RN/native bridge to actually run.

class TwoTierImageCache {
  constructor({ memoryLimitBytes, diskLimitBytes }) {
    this.memoryLimitBytes = memoryLimitBytes;
    this.diskLimitBytes = diskLimitBytes;
    this.memory = new Map(); // key -> { sizeBytes }, insertion order = LRU order
    this.disk = new Map();
    this.memoryUsed = 0;
    this.diskUsed = 0;
    this.log = [];
  }

  _touch(map, key) {
    const val = map.get(key);
    map.delete(key);
    map.set(key, val); // re-inserting moves it to the "most recently used" end
  }

  _evictFromMemoryUntilFits(incomingBytes) {
    while (this.memoryUsed + incomingBytes > this.memoryLimitBytes && this.memory.size > 0) {
      const oldestKey = this.memory.keys().next().value;
      const entry = this.memory.get(oldestKey);
      this.memory.delete(oldestKey);
      this.memoryUsed -= entry.sizeBytes;
      this._putDisk(oldestKey, entry.sizeBytes); // demote, don't discard outright
      this.log.push(`  memory evict -> disk: ${oldestKey} (${entry.sizeBytes}B)`);
    }
  }

  _putDisk(key, sizeBytes) {
    while (this.diskUsed + sizeBytes > this.diskLimitBytes && this.disk.size > 0) {
      const oldestKey = this.disk.keys().next().value;
      const entry = this.disk.get(oldestKey);
      this.disk.delete(oldestKey);
      this.diskUsed -= entry.sizeBytes;
      this.log.push(`  disk evict (discarded): ${oldestKey} (${entry.sizeBytes}B)`);
    }
    this.disk.set(key, { sizeBytes });
    this.diskUsed += sizeBytes;
  }

  get(key) {
    if (this.memory.has(key)) {
      this._touch(this.memory, key);
      return 'memory-hit';
    }
    if (this.disk.has(key)) {
      const entry = this.disk.get(key);
      this.disk.delete(key);
      this.diskUsed -= entry.sizeBytes;
      this._evictFromMemoryUntilFits(entry.sizeBytes); // promote back to memory
      this.memory.set(key, entry);
      this.memoryUsed += entry.sizeBytes;
      return 'disk-hit-promoted';
    }
    return 'miss';
  }

  put(key, sizeBytes) {
    this._evictFromMemoryUntilFits(sizeBytes);
    this.memory.set(key, { sizeBytes });
    this.memoryUsed += sizeBytes;
  }
}

const cache = new TwoTierImageCache({ memoryLimitBytes: 50, diskLimitBytes: 200 });
const images = [['thumb_a', 15], ['thumb_b', 15], ['thumb_c', 15], ['thumb_d', 15], ['thumb_e', 15]];

console.log('Loading images into memory tier (limit=50B):');
for (const [key, size] of images) {
  cache.put(key, size);
  console.log(`  put ${key} (${size}B) -> memory used=${cache.memoryUsed}B, disk used=${cache.diskUsed}B`);
  for (const line of cache.log) console.log(line);
  cache.log = [];
}

console.log('\nRe-requesting thumb_a (should be a disk hit, promoted back to memory):');
console.log('  result:', cache.get('thumb_a'));
for (const line of cache.log) console.log(line);

console.log('\nRequesting an image never loaded (cold miss, would trigger a real network fetch):');
console.log('  result:', cache.get('thumb_never_loaded'));

Output:

Loading images into memory tier (limit=50B):
  put thumb_a (15B) -> memory used=15B, disk used=0B
  put thumb_b (15B) -> memory used=30B, disk used=0B
  put thumb_c (15B) -> memory used=45B, disk used=0B
  put thumb_d (15B) -> memory used=45B, disk used=15B
  memory evict -> disk: thumb_a (15B)
  put thumb_e (15B) -> memory used=45B, disk used=30B
  memory evict -> disk: thumb_b (15B)

Re-requesting thumb_a (should be a disk hit, promoted back to memory):
  result: disk-hit-promoted
  memory evict -> disk: thumb_c (15B)

Requesting an image never loaded (cold miss, would trigger a real network fetch):
  result: miss

The log shows the mechanics working as designed: once memory (limit 50B) fills up, adding a new image demotes the oldest one to disk rather than dropping it; re-requesting a demoted image is a disk hit that promotes it back into memory (itself evicting whatever is now oldest in memory to make room); and a key that was never loaded at all is a clean miss.

Sizing and eviction in practice

Memory limit should be small relative to disk (memory is the scarce, shared-with-everything-else resource; disk is comparatively cheap), commonly tens of megabytes for memory versus a few hundred megabytes for disk on a typical device, tuned against the app's actual image sizes and available RAM on lower-end target devices specifically, not just a flagship test phone.

Decoding and resizing off the UI thread

Decoding a full-resolution image and resizing it to the size actually needed on screen is CPU work that must not run on the UI thread, or the app will visibly stutter. On React Native, that means either using a library that already does this off-thread (react-native-fast-image, which wraps the native image pipelines below), or explicitly deferring the work with InteractionManager.runAfterInteractions so decoding doesn't compete with an in-progress scroll or animation. Below is the shape of that wiring as architecture-level pseudocode, since it needs the native RN bridge and isn't runnable outside a real React Native project:

// Illustrative RN wiring, not executed here (needs the native bridge).
function CachedImage({ uri, width, height }) {
  const [source, setSource] = useState(null);

  useEffect(() => {
    let cancelled = false;
    InteractionManager.runAfterInteractions(async () => {
      const decoded = await imageCache.getOrDecode(uri, { width, height }); // memory -> disk -> network
      if (!cancelled) setSource(decoded);
    });
    return () => { cancelled = true; };
  }, [uri, width, height]);

  return source
    ? <Image source={source} style={{ width, height }} />
    : <PlaceholderView width={width} height={height} />;
}

// Prefetching: warm the cache for images about to be needed (e.g. the next
// few items in a list) without blocking the currently visible screen.
function prefetchUpcoming(urls) {
  urls.forEach((uri) => Image.prefetch(uri).catch(() => {})); // best-effort, ignore failures
}

Falling back to native loaders when available

When a mature native image library is already integrated (Glide or Picasso on Android, SDWebImage on iOS), prefer routing through it instead of reimplementing memory/disk caching from scratch: these libraries already handle bitmap pooling (reusing an already-decoded image's pixel-buffer memory for a new image instead of allocating a fresh buffer each time, cutting garbage-collection churn), request de-duplication (two views requesting the same URL share one decode), and platform-specific memory pressure callbacks that a hand-rolled JS cache can't observe directly. react-native-fast-image is the common bridge that wraps exactly these two native libraries, falling back to Image only when the native module isn't available on that platform build.

What an ETag is

An ETag (entity tag) is an opaque identifier, usually a hash of the resource's content, that a server returns in a response header alongside a resource. It changes whenever the resource's content changes and stays the same when it doesn't.

The end-to-end flow

1. First request: the client sends a normal GET; the server responds 200 OK with the body and an ETag: "abc123" header.
2. The client stores the ETag alongside the cached resource itself, keyed the same way the resource is looked up, ideally in the SAME on-device row or entry as the cached body, so the two can never drift apart from each other.
3. On the next request for the same resource, the client sends a conditional GET with If-None-Match: "abc123".
4. If the resource hasn't changed, the server responds 304 Not Modified with no body at all, just headers, and the client keeps using its already-cached copy. If it HAS changed, the server responds 200 OK with the new body and a new ETag, which the client stores, replacing the old one.

Why this avoids unnecessary re-downloads for large resources

The entire point of the conditional-request/304 flow is that the server confirms freshness WITHOUT resending the body. For a large resource, a big image, a video thumbnail, a large JSON payload, that's the difference between a few-hundred-byte header round trip and re-downloading megabytes the client already has, unchanged, potentially over a metered mobile connection.

Persisting and associating ETags on device

Store the ETag value in the same local database row or sidecar metadata as the cached resource itself, not in a separate structure that could get out of sync. If a cached body were deleted without also deleting its ETag, a stale If-None-Match could claim freshness for a body that no longer actually exists on device, and the app would need to handle that inconsistency as a special case instead of it being structurally impossible.

Fallback for servers that don't support ETag

Some backends never send an ETag header at all. Fall back, in order of preference:

1. Last-Modified / If-Modified-Since: an older, coarser conditional-request mechanism using a timestamp instead of a content hash. It works the same 304 way, but at second-level granularity, so it can miss a same-second double update.
2. If neither header is present, fall back to a plain TTL and simply refetch after some interval. This loses the "confirm without re-downloading" benefit but still bounds staleness.
3. For resources the client controls the identity of, an image URL that embeds a version or content hash, sidestep the problem entirely: a changed resource gets a new URL, so a cached entry for the old URL is simply never revalidated, it's either still wanted (cache hit) or the new URL is fetched fresh.

A practical note on trusting the server's support

Always send If-None-Match alongside a normal conditional GET; never assume a server correctly implements it just because it sent an ETag header once. Track, per endpoint, whether conditional requests actually return real 304s in practice, since some proxies or CDNs in the path strip conditional headers, and fall back to plain TTL revalidation for any endpoint observed to never return a 304 despite being asked correctly.

Two interceptors, two different jobs

OkHttp has two interceptor types, and this problem needs one of each. An APPLICATION interceptor (addInterceptor) runs before OkHttp's own cache logic decides whether to hit the network, so it's the right place to force cache-only behavior when offline, or to retry from cache after a failed network attempt. A NETWORK interceptor (addNetworkInterceptor) only ever sees a response that genuinely came from the network, never one OkHttp already served from its own cache, so it's the right place to fix up headers the SERVER sent, without accidentally re-touching an already-cached response.

Offline: force cache-only with a max-stale budget

CacheControl.Builder().onlyIfCached().maxStale(7, TimeUnit.DAYS).build() rewrites the request to tell OkHttp two things at once: don't attempt the network at all (onlyIfCached), and accept a cached response up to 7 days past its normal freshness window (maxStale). If nothing in the cache is even that fresh, OkHttp returns a 504 response rather than hanging or throwing, which the caller can handle as "genuinely nothing available."

Online: prefer fresh, fall back to cache on failure

The straightforward path is just chain.proceed(request), letting OkHttp's normal caching and network logic run. The interesting part is the fallback: an IOException from chain.proceed() (a timeout, a DNS failure, or a connectivity check that raced an actual drop in the instant between checking and calling) is caught and retried with the SAME cache-only, max-stale request used in the offline branch, so a flaky connection degrades to "show what we have" instead of surfacing a hard error to the user.

Cache-Control header adjustments

The offline/fallback branch adjusts the OUTGOING request's Cache-Control (via onlyIfCached()/maxStale()) to control how OkHttp is willing to read from its cache. Separately, the network interceptor adjusts the INCOMING response's Cache-Control header from the server itself: some backends omit it entirely, or set a max-age too short to be useful offline, so this normalizes it to a guaranteed minimum (one day, in the example) whenever the server didn't specify one. These are two different headers, on two different message directions, doing two different jobs, and mixing them up is the most common bug in this kind of interceptor.

The code

Shown as Kotlin since the question allows either pseudocode or Kotlin; this is not executed here (no Android/JVM toolchain available in this environment), but the OkHttp API calls used (CacheControl.Builder, onlyIfCached, maxStale, Request.newBuilder().cacheControl(...), Response.newBuilder().header(...), addNetworkInterceptor) are the real, correct API surface, not invented.

import okhttp3.CacheControl
import okhttp3.Interceptor
import okhttp3.Response
import java.io.IOException
import java.util.concurrent.TimeUnit

/**
 * Offline-first interceptor.
 *
 * - Offline: rewrite the request to accept a cached response up to max-stale 7 days
 *   old, and never attempt the network at all.
 * - Online: try the network first for a fresh response; if that call throws
 *   (timeout, DNS failure, or a connectivity check that raced a real drop), fall
 *   back to the same max-stale cache read instead of surfacing the error.
 *
 * isOnline is injected rather than read from an Android Context directly, so this
 * class is unit-testable without a device or emulator.
 */
class OfflineFirstInterceptor(
    private val isOnline: () -> Boolean,
    private val maxStaleDays: Int = 7,
) : Interceptor {

    override fun intercept(chain: Interceptor.Chain): Response {
        val request = chain.request()

        if (!isOnline()) {
            return chain.proceed(cacheOnlyRequest(request))
        }

        return try {
            chain.proceed(request) // online: prefer a live, fresh response
        } catch (e: IOException) {
            // Believed online, but the actual call failed. Fall back to cache
            // with the same max-stale budget as the offline path.
            chain.proceed(cacheOnlyRequest(request))
        }
    }

    private fun cacheOnlyRequest(request: okhttp3.Request) = request.newBuilder()
        .cacheControl(
            CacheControl.Builder()
                .onlyIfCached()
                .maxStale(maxStaleDays, TimeUnit.DAYS)
                .build()
        )
        .build()
}

/**
 * Network interceptor: normalizes the server's Cache-Control header so responses
 * stay cacheable even when the backend forgot to set one, or set one too short to
 * be useful offline. Registered with addNetworkInterceptor (not addInterceptor) so
 * it only ever sees a response that genuinely came from the network, never one
 * OkHttp already served from its own cache.
 */
class CacheHeaderNormalizerInterceptor(
    private val minFreshSeconds: Int = TimeUnit.DAYS.toSeconds(1).toInt(),
) : Interceptor {

    override fun intercept(chain: Interceptor.Chain): Response {
        val response = chain.proceed(chain.request())
        val existing = response.header("Cache-Control")
        return if (existing == null || !existing.contains("max-age")) {
            response.newBuilder()
                .header("Cache-Control", "public, max-age=$minFreshSeconds")
                .build()
        } else {
            response
        }
    }
}

// Wiring, plus a real on-disk Cache so onlyIfCached()/maxStale() have something to read:
//
// val client = OkHttpClient.Builder()
//     .cache(Cache(File(context.cacheDir, "http_cache"), 20L * 1024 * 1024)) // 20MB
//     .addInterceptor(OfflineFirstInterceptor(isOnline = { connectivityManager.isOnline() }))
//     .addNetworkInterceptor(CacheHeaderNormalizerInterceptor())
//     .build()

Why isOnline is injected instead of read from Context directly

Passing isOnline: () -> Boolean into the constructor, rather than having the interceptor reach into ConnectivityManager itself, keeps the class testable with a plain unit test (fake the lambda to return true/false) and paired with something like OkHttp's MockWebServer to simulate a failed call, without needing an emulator or a real network to exercise either branch.

How normalization works

A non-normalized GraphQL client would cache each query's response as one opaque blob, keyed by the query and its variables. A normalized cache (Apollo's normalized cache is the common example) instead flattens every object returned by ANY query into one flat store keyed by a stable identity, typically __typename:id (for example Product:42), and each query's cached result becomes a set of REFERENCES into that shared store rather than a duplicated copy of the object.

Worked example: why this raises the hit rate

Query A, getProduct(id: 42), returns { __typename: "Product", id: 42, name: "Widget", price: 9.99 }. Query B, searchProducts(...), returns a list that happens to also include product 42 among its results. Without normalization these are two separate cached blobs; if the price changes, only whichever query re-runs picks up the new value, and the OTHER screen keeps showing the old price until it also happens to refetch. With normalization, both query results point at the SAME Product:42 entry in the flat store, so updating that one entry, whether from a mutation's response, a subscription push, or a manual cache write, updates every screen currently displaying product 42, even ones reached through an entirely different query. This also means a query whose entities are ALL already present in the store can sometimes be served straight from cache even if that exact query shape has never run before, and a mutation response can be written directly into the store (for example, bumping a cached post's like count after likePost(id)) without a network refetch just to reflect an obvious, already-known update.

Additional steps for offline sync

Persist the normalized store to on-device storage (for example SQLite or the platform's key-value storage via a persistence link) so the app has real data to show on a cold launch with no network at all. Queue mutations made while offline in a durable local outbox, and replay them in order once connectivity returns, rather than losing them or silently dropping them.

Additional steps for conflict resolution

A queued offline mutation can end up applying to data that's now stale, someone else (or another of the same user's devices) changed the same entity while this device was offline. Options, in increasing complexity: last-write-wins (simplest, accepts the risk of silently overwriting a concurrent change); a server-side version or ETag check that rejects a mutation built on stale data and forces the client to refetch and let the user redo the edit against current data; or field-level merging for cases where two concurrent edits touch different fields of the same entity and both can be safely kept.

Additional steps for cache eviction

A normalized store can grow without bound as a user browses, every product ever viewed stays in the flat store forever by default, since nothing automatically knows it's no longer needed. Mobile specifically needs an explicit eviction policy: garbage-collect entities no longer referenced by any currently-active query result (Apollo's cache exposes a gc() call for exactly this), and cap the total persisted cache size so browsing doesn't quietly consume an unbounded amount of the device's limited storage over time.