Mobile System Architecture and Offline-First Design Questions

Clarify goals & constraints

• Deliver messages + presence to millions of mobile users with low latency but minimal battery/data/cost. Target latencies: critical (≤1s), interactive (1–5s), background (tens of seconds+). Support iOS/Android and flaky networks.

High-level hybrid architecture

• Mobile client chooses one of three paths: platform push (APNs/FCM) as primary background channel; persistent connection (MQTT/WebSocket) when app foreground/interactive; adaptive polling as fallback on severe restrictions.
• Server side: Ingress layer (API GW), real-time brokers (MQTT/WS clusters), push gateway (APNs/FCM integrator), presence store (in-memory distributed cache like Redis), message queue (Kafka), and orchestrator for routing.

When to use each

• APNs/FCM: background wake, low battery, when app suspended or OS-recommended. Use for low-volume, high-importance alerts or to nudge app to foreground.
• Persistent (MQTT preferred): foreground, high-frequency presence/typing/real-time messaging. MQTT advantages: lightweight binary, keepalive, LWT, QoS, built-in session persistence.
• WebSocket: browser/web or when protocol simplicity preferred. Slightly heavier than MQTT.
• Polling: use rare — for networks blocking long-lived sockets or low-frequency checks (e.g., hourly), with exponential backoff.

NAT/firewall & connectivity

• Use persistent TLS (TCP) outbound from client to server on common ports (443) to traverse NAT/firewalls. Implement keepalive/ping intervals tuned per platform:
  • Foreground: 15–30s ping
  • Background: rely on push; avoid long pings
• Use MQTT over WebSockets + TLS on 443 to bypass proxies.

Connection multiplexing & resource optimization

• Multiplex multiple logical channels over one TCP/TLS connection per device. Use subscription topics for multiple chat rooms/presence channels. Coalesce heartbeat and small messages to reduce packets.
• Use server-side connection brokers that optionally proxy connections to backend workers.

Reconnect & session strategies

• Exponential backoff with jitter for reconnects. Use session resume and message replay (MQTT persistent sessions, server-side cursor) to avoid redelivery and save battery.
• Detect network type (Wi‑Fi vs cellular) and adapt QoS/keepalive and sync frequency.

Horizontal scaling

• Stateless edge frontends (load balancers) terminate TLS and forward to stateful real-time clusters. Partition users by consistent hashing (user-id) to brokers, scale brokers with autoscaling groups. Use a metadata service for routing and service discovery. Use Redis cluster for presence with sharding and Kafka for durable message buses. For millions of devices, use multiple regions, geo-shard users, and route to nearest region.

Trade-offs & cost

• Push is cheap and battery-friendly but limited for high-frequency real-time. Persistent connections provide low latency at cost of server resources and battery. Balance by using push as default background channel and enabling persistent only when necessary (app foreground or explicit user opt-in for “always-on” experiences).

Approach summary (mobile developer view)
Mobile constraints — battery, intermittent networks, NATs, metered data — force you to minimize radio wakeups, bytes, and connection churn. Design real-time features to be network-aware, energy-aware, and adaptive.

Polling frequency

• Use exponential backoff and jitter for failed polls; increase interval when app is backgrounded or device on low power.
• Prefer server push or long-lived connection for high-frequency needs; if polling must be used, bundle multiple resources into one request and avoid fixed short intervals (e.g., 15–60s active, 5–15min background).

Keepalive intervals

• Set keepalives longer than carrier NAT timeout but short enough to detect disconnects. Typical compromise: 30–90s on Wi‑Fi, 60–120s on cellular; increase when on low-power mode.
• Use TCP/TLS ping sparingly; prefer application-level heartbeats only when necessary.

Connection multiplexing

• Reuse a single HTTP/2 or WebSocket per server origin to carry multiple channels/events to avoid extra TLS handshakes and radio transitions.
• On Android/iOS use platform networking stacks that support connection pooling (OkHttp, NSURLSession with HTTP/2).

Background work

• Rely on OS scheduling APIs (Background Fetch, WorkManager, BGTask) to batch network activity and avoid continuous background wakeups.
• Defer non-urgent syncs to when on Wi‑Fi, charging, or device idle.

When to prefer push over persistent connections

• Prefer push (APNs/Firebase) when you need occasional notifications or when app is backgrounded — pushes are OS-optimized and avoid persistent sockets.
• Use persistent connections (WebSocket/HTTP/2) when low latency and high message volume are required and app is foreground; fall back to push for background notifications.

Practical tips

• Respect "metered" flags; allow user settings for sync aggressiveness.
• Instrument: measure wakeups, bytes, and battery impact; tune intervals per real-world telemetry.

Approach (brief)
Design an SQLite-backed queue (Room) storing Change records with status, idempotency_key, attempt_count, next_retry_at. Use a single-transaction enqueue/dequeue and WorkManager to schedule uploads. Deduplicate on idempotency_key at insert. Exponential backoff computed from attempt_count.

Schema (Room entity)

@Entity(tableName = "changes", indices = [Index(value = ["idempotencyKey"], unique = true)])
data class Change(
  @PrimaryKey val id: String,                 // uuid
  val payload: String,                        // JSON
  val idempotencyKey: String,
  val status: String,                         // PENDING, IN_FLIGHT, COMPLETED, FAILED
  val attemptCount: Int,
  val nextRetryAt: Long,                      // epoch millis
  val createdAt: Long
)

DAO signatures & key queries

@Dao
interface ChangeDao {
  @Insert(onConflict = OnConflictStrategy.IGNORE)
  fun insert(change: Change): Long              // returns -1 if dup

  @Query("SELECT * FROM changes WHERE status = 'PENDING' AND nextRetryAt <= :now ORDER BY createdAt LIMIT :limit")
  fun fetchReady(now: Long, limit: Int): List<Change>

  @Transaction
  fun claimBatch(ids: List<String>) { /* set status='IN_FLIGHT' atomically */ }

  @Query("UPDATE changes SET status = :status WHERE id = :id")
  fun updateStatus(id: String, status: String)

  @Query("DELETE FROM changes WHERE id = :id")
  fun deleteById(id: String)

  @Query("UPDATE changes SET attemptCount = :attempt, nextRetryAt = :next WHERE id = :id")
  fun updateRetry(id: String, attempt: Int, next: Long)
}

Enqueue pseudocode

fun enqueue(changePayload: String, idempotencyKey: String) {
  val id = UUID.randomUUID().toString()
  val now = System.currentTimeMillis()
  val change = Change(id, changePayload, idempotencyKey, "PENDING", 0, now, now)
  val res = dao.insert(change)
  if (res == -1L) { /* duplicate detected — no-op or update payload if desired */ }
  WorkManager.enqueue(OneOffSyncWork) // ensure worker runs
}

Worker (upload loop)

class SyncWorker : CoroutineWorker(...) {
  override suspend fun doWork(): Result {
    val now = System.currentTimeMillis()
    val batch = dao.fetchReady(now, 20)
    if (batch.isEmpty()) return Result.success()

    dao.claimBatch(batch.map { it.id }) // set IN_FLIGHT in transaction

    for (c in batch) {
      try {
        val resp = api.upload(c.payload, c.idempotencyKey) // server dedups by idempotencyKey
        if (resp.success) {
          dao.deleteById(c.id)
        } else {
          scheduleRetry(c)
        }
      } catch (e: Throwable) {
        scheduleRetry(c)
      }
    }
    return Result.success()
  }

  private fun scheduleRetry(c: Change) {
    val nextAttempt = c.attemptCount + 1
    val backoff = min(60_000L * (1 shl (nextAttempt -1)), 24*60*60_1000L) // cap 24h
    val next = System.currentTimeMillis() + backoff
    dao.updateRetry(c.id, nextAttempt, next)
    dao.updateStatus(c.id, "PENDING")
  }
}

Resumability & Idempotency

• Persisted state survives restarts in Room DB.
• IdempotencyKey unique index prevents duplicate enqueue.
• Server should support idempotency keys; client includes key with upload.
• WorkManager ensures retry across app restarts; nextRetryAt prevents premature work.

Transactions & concurrency

• claimBatch runs inside a DB transaction: select FOR UPDATE semantics via Room @Transaction to set status IN_FLIGHT only for fetched ids to avoid double-processing across workers.

Edge cases & notes

• Handle long-running IN_FLIGHT on crash: a periodic job resets IN_FLIGHT older than threshold to PENDING.
• Consider tombstones for COMPLETED items retention policy.
• Encrypt payloads if sensitive.

Approach summary
Describe a lightweight RGA-like sequence CRDT adapted for mobile: compact identifiers, opportunistic sync, tombstone compression, and safe GC via consensus/causal visibility.

Identifier generation

• Use (siteId, counter, dense-position) triples where dense-position is variable-length base-64 to allow locality without long paths.
• Locally increment counter; for concurrent inserts between two identifiers, generate a fractional position by taking midpoint of base-64 strings; fall back to increasing counter to avoid long identifiers.

Core data structures

• Stored as linked list of nodes: id, value (char/chunk), tombstone flag, prev/next pointers (indices). Keep indices small (32-bit).
• Persist compact arrays to disk (SQLite / Realm) to avoid memory pressure.

Insert/Delete semantics

• Insert: create id = (siteId, counter, pos), store node with tombstone=false, send operation {ins, id, leftId, rightId, value}.
• Delete: mark node.tombstone=true, send {del, id, tombstone=true}.
• Operations are commutative: apply by locating position via id comparisons and splicing pointers; ignore duplicates.

Tombstone handling & compaction

• Store tombstones but compact adjacent tombstones and coalesce contiguous deleted ranges into a single metadata entry referencing start/end ids and count.
• Use chunking: group characters into small immutable chunks (e.g., 16-64 bytes). Deleting a range marks chunk tombstones; when chunk fully deleted, replace with a single tombstone placeholder.

Garbage collection (safe)

• Maintain per-site version vectors (VV) exchanged during sync. A node/chunk is GC-able when all known peers’ VVs include operations that causally descend from the deletion (i.e., deletion timestamp <= VV[peer]).
• Opportunistic GC: when connected to server or peers, compute global-minimum VV; delete tombstoned chunks whose delete counter <= minVV; compact storage.
• If offline long time, keep tombstones until GC confirmed to avoid resurrecting edits.

Pseudocode

function insert(leftId, rightId, value):
  id = genId(siteId, localCounter++)
  node = Node(id, value, tomb=false)
  spliceBetween(leftId, rightId, node)
  persist(node)
  broadcast({ins, id, leftId, rightId, value})

function delete(id):
  node = find(id)
  if node and not node.tomb:
    node.tomb = true
    persist(node)
    broadcast({del, id})

function applyDel(op):
  node = find(op.id)
  if node:
    node.tomb = true
    persist(node)

function garbageCollect(minVV):
  for chunk in chunks:
    if chunk.isTomb and chunk.deleteTs <= minVV[chunk.site]:
      removeChunk(chunk)  // free storage, update links

Trade-offs / mobile considerations

• Chunking reduces per-character metadata and memory churn.
• Short IDs reduce storage but risk rebalancing; fallback counters avoid starvation.
• VV exchange can be compressed (Bloom filters or sync anchors) to limit data use.
• Keep GC conservative to avoid correctness violations; prefer server-mediated global GC for aggressive cleanup.

Outcome
This yields a mobile-friendly sequence CRDT: low-memory, network-efficient, correct under concurrency, and safe compactable when peers confirm visibility.

Situation overview / goal
Quickly identify whether notifications or background syncs introduced the battery regression, quantify impact, and implement low-risk fixes that preserve functionality.

Client-side data to collect

• Per-device samples: OS version, model, app version, foreground/background state, uptime.
• Timeline logs: timestamped events for receipt of push, start/end of background job, CPU wake-locks, network transfers, and notification display.
• Energy metrics: battery drain rate before/after release, wakelock duration (Android PowerManager), iOS energy usage from Instruments.
• Telemetry counts: number of pushes received, silent pushes, background fetch runs, retries, bytes transferred.
• Sampling window: collect anonymized traces for a representative cohort (low/medium/high activity users).

Server-side data to collect

• Push send rate per user, per device token, priority flags (FCM high/normal, APNs priority).
• Payload sizes and patterns (frequency, duplication).
• Sync scheduling decisions, failure/retry rates, and server-side batching/coalescing behavior.
• Correlate server send timestamps with client-received timestamps.

Instrumentation to add

• Lightweight event tracing (timestamped) around: onMessageReceived, startJob, network request begin/end, wakelock acquire/release, notification post/dismiss.
• Counters and histograms: background job duration, bytes, CPU active time, number of pushes per hour.
• Toggleable verbose mode for deeper trace (only enabled for affected cohort).
• Remote-config flags to flip behaviors (priority handling, batching, backoff) for experiments.

Experiments / A-B tests

• A: Reduce push priority from high to normal for N% of users and measure battery + UX metrics (latency to notify).
• B: Server-side coalescing: batch updates into 1 push per X minutes vs immediate pushes.
• C: Client-side debounce: ignore duplicate pushes within Y seconds and test battery/accuracy tradeoff.
• D: Switch silent pushes to scheduled periodic sync via WorkManager (Android) / BGAppRefresh (iOS) for low-critical updates.
  Measure battery drain, notification latency, missed/duplicated notifications, user engagement.

Potential fixes to push

• Server: coalesce and rate-limit pushes per user; reduce payload size; mark non-urgent pushes as normal priority.
• Client: respect Doze/Background Restrictions; move work to JobScheduler/WorkManager (Android) or BackgroundTasks (iOS); avoid holding wakelocks—use JobScheduler for network; batch network calls; exponential backoff for retries.
• Notification UX: aggregate notifications, suppress duplicates, use silent notifications only to trigger light refreshes.
• Monitoring + rollout: enable remote-config to roll back/pivot fast and monitor post-fix metrics.

Outcome & validation
Deploy fixes to small canary, monitor telemetry (battery, background job counts, notification latency), expand rollout if positive.