Mobile System Architecture and Offline-First Design Questions

Why stateless helps scale mobile backends
Stateless services don’t store client session/data on the app server, so any instance can handle any request. For mobile apps with many short-lived API calls (auth, sync, analytics), stateless design enables horizontal scaling, simpler deployments, faster recovery, and easier A/B rollouts.

Role of components

• Load balancers: distribute incoming mobile requests across healthy instances; perform SSL termination and path/rate routing so clients don’t need to know instance topology.
• Autoscaling groups: add/remove stateless instances based on CPU, request rate or custom metrics (e.g., requests/sec from mobile users) to match demand and control cost.
• Health checks: LB + ASG probes (HTTP/ICMP) ensure traffic goes only to healthy nodes; unhealthy instances are replaced automatically, keeping user experience stable.

Example stateful component and isolation

• WebSocket server for real-time chat/notifications is stateful (maintains connections). I’d isolate it into its own service/cluster with:
  • A dedicated autoscaling group tuned to connection count (scale by active sockets).
  • A sticky-aware load balancer or a routing layer (e.g., a message router like HAProxy or NGINX with consistent-hash) so reconnects hit the same process when needed.
  • Shared state in Redis (presence, user->server mapping) and a pub/sub (Redis Pub/Sub or Kafka) to fan out messages across instances.
  • Health checks that verify socket accept loop and Redis connectivity; on failure, reassign users via Redis mapping and rely on clients to reconnect with exponential backoff.

This pattern keeps core APIs elastic and simple for mobile clients while handling connection state in a specialized, scalable service.

Approach (brief)

Use a LinkedHashMap with accessOrder = true to maintain LRU ordering and a running total size. Keep O(1) get/put by map operations and update size on insert/remove. Ensure thread-safety with a single ReentrantLock (or synchronize) to avoid race conditions during reads/writes and eviction.

Kotlin implementation

import android.graphics.Bitmap
import java.util.LinkedHashMap
import java.util.concurrent.locks.ReentrantLock
import kotlin.concurrent.withLock

class LruBitmapCache(private val maxBytes: Long) {
    private val lock = ReentrantLock()
    // accessOrder=true -> iteration order is LRU -> MRU
    private val map = object : LinkedHashMap<String, Bitmap>(16, 0.75f, true) {}
    private var currentSize = 0L

    fun get(key: String): Bitmap? = lock.withLock {
        return map[key] // accessOrder ensures this marks it as recently used
    }

    fun put(key: String, bitmap: Bitmap, sizeInBytes: Int) = lock.withLock {
        val prev = map.put(key, bitmap)
        if (prev != null) {
            currentSize -= sizeOf(prev)
        }
        currentSize += sizeInBytes
        evictIfNeeded()
    }

    private fun evictIfNeeded() {
        val iter = map.entries.iterator()
        while (currentSize > maxBytes && iter.hasNext()) {
            val entry = iter.next()
            val removed = entry.value
            currentSize -= sizeOf(removed)
            iter.remove()
            // optionally recycle bitmaps: removed.recycle()
        }
    }

    private fun sizeOf(bitmap: Bitmap): Int {
        // API-safe: bytes per row * height or bitmap.byteCount on newer APIs
        return bitmap.byteCount
    }
}

Key points & trade-offs

• Data structures: LinkedHashMap(accessOrder=true) + hashmap semantics -> O(1) lookup and ordering.
• Concurrency: ReentrantLock ensures atomic get/put/evict. For higher read concurrency, consider ReadWriteLock or lock striping, but must ensure consistent size accounting.
• Eviction: done inside put while holding lock to avoid races.
• Memory: track byte sizes precisely; handle large single items (if item > maxBytes decide to reject or clear cache first).
• Android notes: consider using Bitmap.recycle() if appropriate and watch for bitmap.byteCount availability across API levels.

Approach summary
Propose a hybrid protocol combining compact vector clocks (CVC) + dotted-version-vectors (DVV) with optional Hybrid Logical Clocks (HLC) fallback and device grouping to minimize per-device metadata while preserving causality.

Techniques

• Compact vector clocks: assign each device an index in a sparse map; store only non-zero entries per-client (sparse CVC). Good when devices have infrequent updates.
• Dotted-version-vectors: record (dot = (node, counter)) plus a summarized causal stable frontier; enables O(1) per-update metadata and precise causality for individual operations.
• Device grouping: aggregate devices by user-account or region; maintain a group clock instead of every device to reduce vector size.
• Hybrid Logical Clocks: attach a 64-bit HLC timestamp to events to provide monotonic scalar ordering; use with DVV to break ties and reduce ambiguity.

Trade-offs

• Accuracy vs size: DVV + sparse CVC preserves exact causality with smaller average size but requires more complex merge logic. Grouping reduces metadata but may lose per-device causal resolution and cause false concurrency.
• Pruning: periodically compact older entries into a stable frontier; safe if clients can accept eventual loss of fine-grained ancestry.
• Merge complexity: DVV merges require comparing dots and frontiers (O(k) where k = non-zero entries). HLC simplifies comparisons to scalar when causality isn’t ambiguous.
• Mobile constraints: prefer small on-wire metadata, incremental sync, and server-side aggregation. Use DVV for local operations, upload summaries to server; server resolves heavy merges and prunes, pushing compacted fronts to clients.

Recommendation
For mobile apps: implement DVV locally with sparse maps, HLC timestamps for fast ordering, and server-side grouping/pruning. This balances accurate causality, low per-device metadata, and acceptable merge cost while preserving battery and bandwidth.

Overview — how to choose
Pick storage by data shape, sync needs, encryption, and concurrency: key-value for small prefs/secrets, relational for structured queries and joins, object DBs for complex models/fast dev, keystore/keychain for secrets.

SQLite (iOS/Android)

• Use cases: relational data, complex queries, joins, large datasets.
• Strengths: Mature, small footprint, predictable performance, ACID.
• Weaknesses: Manual schema/migrations, boilerplate.
• Caveats: iOS uses SQLite under Core Data; Android Room abstracts migrations and threading.

Core Data (iOS)

• Use cases: object graph, undo/relationships, iOS-native model layer.
• Strengths: Integrated with Cocoa, change tracking, NSPersistentCloudKit.
• Weaknesses: Learning curve, migration complexity, less portable.
• Caveats: Background contexts required for threading; lightweight migrations possible but heavy versioning needs care.

Room (Android)

• Use cases: SQLite ORM for Android apps with compile-time queries.
• Strengths: Type-safe DAOs, integrated migrations, LiveData/Flow support.
• Weaknesses: Still SQL-based for complex queries, needs migration scripts.
• Caveats: Requires careful coroutine/dispatcher handling for background I/O.

Realm

• Use cases: Offline-first apps, reactive sync, simple object mapping.
• Strengths: Easy model mapping, fast reads/writes, built-in change notifications, optional Realm Sync.
• Weaknesses: Binary file format, vendor lock-in, larger binary size.
• Caveats: Migration API available; check background thread rules and encryption support via keys (manage securely).

LevelDB / RocksDB

• Use cases: High-write key-value, embedded storage for custom engines.
• Strengths: Fast write throughput, simple key-value semantics.
• Weaknesses: No relational queries, manual indexing, platform bindings required.
• Caveats: Rare in pure mobile apps; manage compaction and file size.

SharedPreferences (Android) / UserDefaults (iOS)

• Use cases: Small key-value config, UI state, flags.
• Strengths: Simple API, low overhead.
• Weaknesses: Not for large data or complex objects, potential blocking on main thread if misused.
• Caveats: Use DataStore (Android) or secure variants for async and consistency.

DataStore (Android)

• Use cases: Replaces SharedPreferences for async, coroutine-friendly key-value or proto.
• Strengths: Asynchronous, transactional, type-safe (Proto).
• Weaknesses: Newer; migration path from SharedPreferences needed.
• Caveats: Use protobuf for schema and migrations.

Keychain (iOS) / Keystore (Android)

• Use cases: Storing secrets, tokens, encryption keys.
• Strengths: OS-managed, hardware-backed, secure.
• Weaknesses: Limited size, access control nuances.
• Caveats: iOS Keychain persists across app reinstall with keychain-access-groups if configured; Android Keystore keys tied to device/lock-screen; handle backups and migration carefully.

Offline-first considerations

• Prefer local DB (Room/SQLite/Core Data/Realm) for full offline model and sync queue.
• Encrypt sensitive local data (use SQLCipher or platform crypto; keys in Keystore/Keychain).
• Design sync/conflict resolution (CRDTs, last-write vs merge) and background sync constraints (iOS background fetch, Android WorkManager).
• Test migrations, multi-thread access, and restore-from-backup scenarios.

Use concrete combo: Room + WorkManager + Keystore for Android; Core Data + BackgroundTasks + Keychain for iOS.

Approach overview
I’d enforce three layers of limits: per-user, per-app (app key), and global. Rate decisions happen at the edge (API gateways / notification workers) with local caches and a central store for coordination. Clients (mobile app) implement backoff/coalescing to reduce pressure.

Algorithms: token-bucket vs leaky-bucket vs sliding

• Token-bucket: good for bursts. Each key has capacity and refill rate; allow bursty notifications up to bucket size.
• Leaky-bucket: smooths out bursts into constant rate — useful if downstream services must be constant.
• Sliding-window (approx): use fixed-window counters + two-window interpolation or Redis sorted-sets for exact sliding windows; provides fairer limits across window boundaries.

Choose token-bucket for notifications (allow bursts like critical alerts) with sliding-window counters for approximate enforcement where exactness is expensive.

Scaling / implementation

• Edge nodes keep an in-memory token cache (fast local check). On hit, decrement locally and asynchronously sync with central store.
• Central coordination: Redis (INCR, EVAL scripts) or a distributed counter service to enforce global/app limits and resolve cache drift.
• Use short TTLs and probabilistic refill reconciliation (periodic reconciliation jobs) to fix inconsistencies.
• Partition keys across shards; use consistent hashing for affinity.

Graceful degradation & client UX

• Client-side: exponential backoff, jitter, batch/coalesce notifications, mark non-urgent local notifications to defer.
• Server-side: return clear HTTP 429 with headers: Retry-After, X-RateLimit-Remaining, X-RateLimit-Reset, and machine-readable body explaining which limit triggered (user/app/global).
• Prioritization: honor high-priority/urgent notifications (auth, security) and queue lower-priority ones; expose progress via push-ack or in-app banners when throttled.
• Monitoring: metrics, alerts for hit rates, throttles per key to tune limits.

This design balances low-latency checks, scalability, and good mobile UX under throttling.