Mobile System Architecture and Offline-First Design Questions

Requirements (clarify): global mobile users, personalized images/videos/dynamic fragments, low latency and bandwidth, strong cache-control & invalidation, some authenticated content, offline-friendly.

High-level architecture:

• Origin services (API + media storage, e.g., S3/Blob + signed URLs)
• Global CDN with programmable edge (Fastly/Cloudflare Workers/Akamai EdgeWorkers)
• Edge-cache + edge-compute layer for personalization
• Auth/token service and short-lived signed URLs for private media
• Purge/invalidations via CDN API and origin cache-control headers; use CDC/event bus (Kafka/SNS) to trigger selective invalidations.

Cache key strategy:

• Use hierarchical keys combining stable and variable parts:
  • Static media: {asset_id}:{version} (version from manifest or ETag) — maximizes cache hit and enables atomic invalidation by bumping asset version.
  • Personalized fragments: {fragment_type}:{user_segment}:{variant_hash} — segment not per-user where possible to improve sharing.
  • Dynamic API responses: include URL path + query normalization + Vary headers (Accept, Device-Profile) but avoid including user-id unless necessary.

Edge-compute / ESI:

• Perform personalization at edge using Workers/EdgeFuncs:
  • Fetch cached base asset, then inject small user-specific overlays (watermarks, captions) using edge CPU to avoid origin round trips.
  • Use ESI-like techniques or edge templates: cache fragments (header/footer/recommendations) and assemble at edge.
  • For video, use chunk-level caching + tokenized manifests; personalize client-side overlays where possible to reduce edge CPU.

Authenticated content handling:

• Use short-lived signed URLs or tokens for direct media fetch; CDN validates signatures at edge (JWT HMAC or signed URL).
• For per-user HTML/JSON, require edge-auth: validate session token at edge, then decide to serve cached segment + private fragment fetched from origin or generated at edge.
• Avoid caching full per-user responses; instead cache “shared” parts and fetch/generate private parts on demand.

Cache invalidation and consistency:

• Prefer versioning (asset version in URL) for media to avoid wide purge.
• For dynamic changes, send targeted CDN purge by cache key or surrogate-key tagging (attach surrogate-keys to responses so origin-triggered invalidation clears related assets).
• Use soft-TTL + stale-while-revalidate to serve stale content while refreshing in background for low-latency.

Trade-offs:

• More edge compute increases hit ratio and reduces origin egress but costs more and has limits on CPU/time.
• Per-user caching increases privacy risks; prefer segment-based personalization and client-side overlays where possible.
• Signed URLs scale well for media but require robust key rotation and clock sync.

Operational notes:

• Monitor cache-hit ratio, origin egress, and latency by region.
• Provide tooling to bump versions, run bulk surrogate-key purges, and audit signed URL usage.

Requirements:

• Allow hosts and guests to perform bookings, modifications, messaging while offline
• Ensure eventual consistency, deterministic conflict resolution, secure reconciliation, low latency UX
• Handle partial sync, large scale, and privacy/compliance

High-level architecture:
Mobile app (offline-first) ⇄ Local datastore & sync engine ⇄ Sync gateway / API layer ⇄ Event store + reconciliation service ⇄ Primary services (Bookings, Hosts, Users) + Auth/Policy store

Key components:

1. Local datastore & sync engine (on device): use SQLite/Realm with CRDT-friendly data model and an operation log (append-only). Record user intents as immutable operations (create-booking, cancel, update-pricing) with metadata (client-id, lamport/timestamp, causal dependencies).
2. Sync gateway: authenticated websocket/HTTP sync endpoint that accepts batched ops, performs validation, stores in event store, returns server-assigned versions.
3. Reconciliation service: applies ops to canonical store, resolves conflicts using deterministic rules (CRDT merges for commutative fields, last-writer-wins with vector clocks for non-commutative, domain rules for bookings like double-book prevention using optimistic locks and compensating transactions).
4. Notification/merge feedback: server returns merge outcome and intent-id → client updates local state; if conflicts require user action, surface guided UI.
5. Security & integrity: end-to-end TLS, JWT with short TTL and refresh; ops signed with device key (HMAC) to prevent replay; server-side validation & authorization; encrypt sensitive local data (device key protected by OS keystore).
6. Eventual consistency guarantees: causal ordering via vector clocks/lamport, idempotent ops, and server-side committer ensuring global ordering for conflicting resource reservations.
7. Edge cases: long offline periods handled by tombstones/compaction; large op backlog via batching and resumable sync; schema migrations with versioned ops.

Trade-offs:

• CRDTs reduce conflicts but increase model complexity; use for user-editable fields (descriptions, availability windows). Use stricter server arbitration for scarce resources (availability slots).
• Stronger consistency on critical paths (booking commit) via server-side optimistic reservation + short hold period; fallback to compensating refund if conflict detected post-reconciliation.

This design provides responsive offline UX, deterministic conflict handling, secure reconciliation, and clear escalation to users when human resolution is needed.

Requirements & constraints:

• Register/rotate APNs/FCM tokens from millions of devices; detect stale/invalid tokens; periodic cleanup; support multiple tokens per user; respect provider error codes (unregistered, invalid, throttled); low-latency registration and high-throughput sending; eventual consistency OK; GDPR/PII concerns.

High-level architecture:

• API gateway → Auth service → Device Registration service → Token Store (primary: write-optimized DB + cache) + Event stream (Kafka) → Push Sender workers → Metrics/Monitoring + Cleanup Service.

Key components and responsibilities:

1. Device Registration service

• Exposes authenticated endpoints for register/refresh/unregister.
• Validates token format, device metadata (OS, app version), client timestamp.
• Writes token record and emits events: token_created, token_rotated, token_deleted.

2. Token Store

• Schema: device_id, user_id, token, provider, last_seen, created_at, app_version, status.
• Use a horizontally scalable DB (Cassandra or DynamoDB) for writes and range scans; cache hot tokens in Redis for sending.

3. Push Sender

• Consumer groups read events or job queue for notifications.
• Sends to provider SDKs with batching and exponential backoff per-provider.
• Interprets provider error codes:
  • "Unregistered"/410 → mark token status = invalid, emit token_removed event.
  • "InvalidArgument"/BadToken → remove and notify client on next activity.
  • Rate limit / throttled → retry with backoff, track quotas.

4. Cleanup & Data Quality

• Periodic job:
  • Soft-delete tokens not seen for X days (e.g., 90d) by setting status=stale.
  • Hard-delete after retention window (e.g., 30d after soft-delete), or sooner if provider returns definitive invalid.
  • Run probabilistic validation: periodically send lightweight silent pings to low-traffic tokens to validate.
• Reconciliation:
  • Reprocess push response logs to reconcile status; run map-reduce jobs to flag anomalies (same token assigned to multiple device_ids).

5. Handling multiple tokens per user

• Allow N tokens per (user, app) with indexed queries by user_id to deliver fan-out.
• Use deduplication keys per device to avoid duplicate sends.
• Provide device management APIs to list/remove devices; surface stale-device UX in clients.

Scalability & operations

• Partition token store by user_id/device_id. Use topic sharding for senders.
• Autoscale sender pools, cap per-provider concurrency, use connection pooling for APNs/FCM.
• Emit metrics: invalidation rate, send latency, error-code distribution; alert on spikes in "unregistered".
• Cost considerations: TTLs to limit storage, compact old versions.

Trade-offs

• Immediate synchronous validation on register increases latency; prefer async validation with optimistic accept.
• Strong consistency would complicate scale; eventual consistency with reconciliation is acceptable for push.

Security & privacy

• Encrypt tokens at rest, audit deletions, expose user consent management and data retention controls.

This design supports millions of tokens, efficient rotation, clear handling of provider errors, periodic cleanup, and multi-device per user semantics while remaining operationally manageable.

Eventual consistency means that after a write, different replicas (or a client and server) may temporarily see different values, but given no new updates, they will converge to the same value within some bounded or unbounded time. In mobile apps this is common because devices may be offline, use local caches, or rely on asynchronous replication to reduce latency and improve availability.

Examples where eventual consistency is acceptable:

• Social feed likes/comments counts (a like may appear locally before propagating).
• Background analytics, telemetry, or logging.
• Collaborative documents with optimistic merges where conflicts are resolved later.

Examples where strong/causal consistency is required:

• Financial transactions (wallet balance, purchases).
• Authentication/authorization decisions (permission checks).
• Single-item inventory decrement during checkout.

How to expose consistency guarantees to frontend developers (to avoid confusion):

• Document explicit consistency SLA per API (e.g., "read-after-write: not guaranteed; convergence within <30s under normal conditions").
• Provide API primitives that indicate consistency level: strongRead(), eventualRead(), readWithStaleness(maxAge).
• Return metadata with responses: version/timestamp/vector clock, sync state (stale/fresh/partial), and conflict markers.
• Offer optimistic UI patterns with clear UX: show pending state, undo, or “syncing” indicators and explain eventual outcomes.
• Provide client SDK helpers: automatic retries, merge strategies, conflict resolution hooks, and network-state APIs so UI can choose behavior when offline.
• Recommend testing strategies: simulate latency/offline, and document user-facing error messages and fallback flows.

Trade-offs summary: eventual consistency improves availability and responsiveness on mobile but increases UX complexity. Make guarantees explicit, provide SDK support and clear UI affordances so frontend teams can design predictable, user-friendly flows.

Requirements & targets:

• RTO: <5 minutes for core messaging/control plane; <30 minutes for non-critical analytics.
• RPO: near-zero (≤30s) for transactions/state; <5 minutes for aggregated logs/analytics.
• Constraints: global low-latency, mobile clients, regulatory data residency.

High-level DR strategy:

• Active-active multi-region deployment with regional primary for latency; cross-region async replication + global consensus for critical state (e.g., leader election via Raft or managed services like Spanner/Cockroach for strongly consistent state).
• Durable event sourcing: write-ahead log (Kafka/Managed PubSub) replicated to at least two regions synchronously where possible; object storage snapshots (S3/GCS) for large blobs every 5–15 minutes.

Failover procedure (automated runbook + human-in-loop):

1. Detection: health checks + synthetic transactions + global observability trigger.
2. Isolation: route traffic away from failing region via global load balancer (Cloud DNS + Anycast + health-based routing).
3. Promotion: promote standby region’s control-plane instances; ensure latest committed offsets applied. For strongly consistent stores, perform controlled leader takeover with safety checks.
4. Reconnect: clients use SDK reconnection logic with exponential backoff + region-priority fallback. Issue a short-lived global token/redirect (via push notification or config endpoint) to point clients to new region; preserve session tokens via redis replication or JWTs accepted by new region.

Preserve critical data:

• Commit log + acknowledged writes before returning success; on failover, rehydrate in-memory state from durable logs and checkpoints.
• Use idempotent write semantics and vector clocks/timestamps to reconcile concurrent writes.
• For partial writes, run compensating transactions from audit log.

Verification post-failover:

• Run smoke tests, business transactions, data-consistency checks (hashes on critical tables), and compare offsets/checkpoints.
• Run throttled canary traffic and monitor error rates, latencies, and data divergence metrics.
• Postmortem: collect traces/logs, time-to-detect, time-to-recover, root cause, and update playbooks.

Trade-offs:

• Active-active increases complexity but reduces RTO/RPO. Synchronous replication reduces RPO but increases write latency—use selectively for critical state.
  Operational needs:
• Automated runbooks (IaC), DR drills quarterly, runbook runbooks in runbooks ;) and clear rollback criteria.