Mobile System Architecture and Offline-First Design Questions

Requirements (clarify): minimize churn, stable experiment stats, support push notifications & real-time features, large scale, safe rollouts and fast rollback.

High-level approach:

• Use feature-flag service + experimentation platform separated from product code. Flags control behavior; experiment layer handles assignment and metrics.

Membership assignment & bucketing:

• Deterministic, stateless bucketing by stable customer key (user_id or device_id) + experiment_id + salt using a consistent hash (e.g., MurmurHash → uniform bucket 0-10000). Store bucketing config (seed, traffic allocation, stratification) in central config.
• Support stratified bucketing: tier, geography, client version, platform to avoid imbalances. Allow forced holds for VIPs and opt-outs.
• Persist exposures for users in event of re-evaluation: write assignment to user profile or a low-latency store (Redis) for sticky membership (important for push sequences).

Instrumentation & observability:

• Emit deterministic exposure events (experiment_id, variant, user_id, timestamp, context) at the moment of decision—both client and server-side. For push flows, also emit delivery, open, action, and unsubscribe events linked by message_id.
• Use unique correlation IDs for real-time events. Stream events into a real-time pipeline (Kafka → processing → metrics DB) with schemas and validation.
• Monitor key guardrails: engagement, retention, error rates, unsubscribe/churn, latency. Set automatic anomaly detection and dashboards.

Rollout & ramp strategy:

• Phased rollout: 0 → 1% → 5% → 25% → 100% with health checks at each step. For push campaigns, run canary on internal/test accounts first.
• Implement kill-switch and fast rollback at flag level with sub-second propagation (CDN/in-memory caches invalidation + push to clients via SSE/WS or mobile remote-config TTLs).
• For experiments: automated halting rules (statistical or heuristic) if guardrail thresholds breached (e.g., >X% increase in unsubscribes) or if p-value/CI shows harm.

Stability & scale considerations:

• Determinism avoids split leakage. Persist assignments so users don't flip variants mid-sequence (critical for push sequences and multi-step real-time features).
• Throttle event volume with sampling for non-critical telemetry; keep 100% for guardrail events.
• Ensure idempotency and deduplication in ingestion; use compacted topic for assignment snapshot.

Rollback & safety:

• Immediate flag toggle to previous variant; revertable deployment pipelines. For push, stop further notifications and send mitigations (e.g., apology or opt-back) if needed.
• Post-rollback: run root-cause analysis, compare cohorts (exposed vs persisted), and publish learnings.

Trade-offs:

• Persisting assignments increases storage and complexity but prevents churn from re-bucketing. Server-side evaluation gives control; client-side required for offline/latency—use hybrid approach: server authoritative, client caches sticky assignment with TTL.

This design balances determinism, observability, fast control, and customer-safety for push and real-time features at scale.

Requirements & constraints:

• Functional: real-time interactive features (chat, presence, live actions) with p95 <200ms for users across NA, EU, APAC.
• Non-functional: survive a whole-region outage, acceptable consistency (e.g., read-after-write for user session data; eventual for analytics), cost/operational complexity trade-offs.

High-level approach:

• Active-active multi-region for low latency and availability of interactive features; active-passive for strongly consistent control-plane operations (billing, account changes) if needed.
• Each region runs full app stack: edge (mobile SDK -> regional gateway), real-time layer (region-local WebSocket/RTC cluster), application services, regional datastores with cross-region replication.

Replication & consistency:

• Use a hybrid model:
  • Real-time state (presence, ephemeral session state): keep region-local authoritative for latency; replicate via async CRDTs or operational transforms to other regions for convergence. CRDTs avoid global locking and handle concurrent updates with automatic merge.
  • User profile / critical metadata: multi-master with leader-election per user (sticky write-region) or lightweight consensus (Raft/ETCD) restricted to a small control-plane cluster; or use Paxos/RAFT-backed global store for metadata requiring strong consistency.
  • Durable events/commands: append-only global log replicated with Kafka MirrorMaker2 or cloud-native cross-region replication; consumers reconcile based on causal metadata (vector clocks).

Client routing:

• DNS + Anycast + SDK region affinity:
  • Use global DNS (GeoDNS) + CDN/Anycast to route initial connections to nearest region.
  • Mobile SDK establishes persistent connection to local region and pins a "home region" token for subsequent reconnects.
  • For reads that tolerate eventual consistency, SDK can accept any-region; for writes requiring strong consistency, SDK forwards to the pinned write-region via regional gateway or a control-plane proxy.

Conflict resolution:

• Prefer CRDTs for merges where correctness is mergeable (presence lists, counters).
• For non-mergeable conflicts, use last-write-wins with client timestamps + server-side validation, or application-level conflict resolution workflows (prompt user, reconcile).
• Provide monotonic session tokens to ensure read-after-write when needed (route reads to write-region until token expiry).

Failure modes & failover:

• Region outage: global load balancer/GeoDNS removes region; clients reconnect to nearest healthy region. For sessions pinned to failed region, rehydrate session from replicated event log or CRDT snapshots; if lost, fall back to log-based replay with potential slight divergence.
• Network partition: allow local progress using eventual replication; delay cross-region operations that need strong consistency.

Performance & latency optimizations:

• Edge caching and CDN for static content and non-critical reads.
• Keep real-time paths local (no cross-continent RTTs in request path).
• Use regional caches (Redis with active-active via RedisGears/CRDT or DynamoDB global tables) to reduce datastore latency.

Trade-offs:

• Active-active yields best latency and availability but increases complexity (conflict resolution, operational overhead, testing). Good for interactive features.
• Active-passive simplifies consistency (single-writer) but increases failover latency and can violate p95 during switchover.
• Strong global consistency everywhere would harm latency (synchronous cross-region consensus) — unacceptable for p95 <200ms across continents.
• Cost: more compute/replication bandwidth for active-active.

Operational & observability:

• Centralized control-plane for orchestration, global healthchecks, automated failover playbooks.
• Extensive distributed tracing, metrics (p95 per region), SLOs, synthetic transactions from each continent.
• Chaos testing (region kill, network partition) in pre-prod and regular runbooks.

Recommendation:

• Implement active-active for real-time interactive services with CRDTs and event logs; use sticky-write-region pattern for user-scoped strong consistency; keep a small strongly-consistent control-plane for account-critical operations. This balances latency, resilience, and reasonable consistency with operational feasibility.

Requirements & constraints:

• 2M concurrent WebSocket connections, low per-connection CPU/memory, realtime (sub-100ms), high availability, geo-distribution, support pub/sub and direct messages.

High-level architecture:

• Edge layer: autoscaled WebSocket edge workers (lightweight processes on many hosts) accept TCP/WSS; run minimal logic (TLS, heartbeat, auth token validation) and maintain socket I/O.
• Broker/coordination layer: stateless message routers + a small set of centralized brokers for control-plane tasks (presence, matchmaking) backed by a distributed store (Redis Cluster / TiKV).
• Message plane: messages routed peer-to-peer between edge workers; persistent messaging not required on path.

Connection brokering (edge vs centralized):

• Edge workers own live socket state (memory per connection minimal: file descriptor + small struct). No centralized process holds socket file descriptors.
• Centralized brokers handle metadata: connection → worker mapping, presence, topic partitioning. Brokers are horizontally sharded; they are not in the data path for streaming messages (except metadata lookup).

State management:

• Ephemeral per-connection state stored in-edge (in-memory). Durable metadata in Redis Cluster:
  • key: conn:<conn_id> -> {worker_id, user_id, last_seen, rate_limits}
  • pubsub channels or Redis Streams for cross-worker notifications
• Use LRU + keepalive timeouts to bound memory.

Routing messages to correct worker:

• Two-step: publisher contacts local edge; edge queries local cache for subscriber location. Cache populated via:
  • consistent-hash-based partition ownership for topics: each topic partition maps to a broker responsible; that broker maintains membership (list of worker_ids with subscribers).
  • Worker registry in Redis + pub/sub updates on connect/disconnect.
• If cache miss: edge queries broker (fast Redis GET). Once worker_id known, sender opens a short-lived RPC (gRPC) to that worker or uses a persistent inter-worker message bus (NATS/JetStream or lightweight TCP mesh) to deliver message to the target worker which then writes to socket.

Horizontal scaling plan & connection limits:

• Aim for ~2M connections by scaling edges: estimate per-host capacity conservatively (e.g., 20k-100k connections depending on kernel tunings). Use epoll/kqueue, tuned SO_REUSEPORT, ulimits, and keep per-connection memory <1KB.
• If capacity = 50k per host => ~40 hosts. Add headroom and AZ redundancy -> 80-100 hosts.
• Autoscale based on connection and CPU metrics. Use placement across zones/regions to reduce blast radius.
• Central brokers sized separately (smaller cluster of Redis + brokers), scale by partition count.

Performance & trade-offs:

• Use consistent hashing for topic partitions to minimize metadata churn. Use push from publisher to worker (short RPC) instead of routing all through brokers to reduce latency.
• Use connection affinity (sticky) from edge LB (Envoy) so re-connections hit same worker when possible.
• For cross-region messaging, use edge-to-edge relay via WAN-optimized message bus or Kafka-like streams.

Fault tolerance:

• Worker fails: periodic heartbeats; worker removed from registry; broker instructs other edges to re-resolve locations; clients reconnect (sticky attempts next best).
• Redis cluster with replicas and Raft for metadata durability.
• Graceful draining: edge signals broker before shutdown to migrate subscribers (or instruct clients to reconnect).

Security & ops:

• TLS termination at edge, mTLS for inter-worker RPCs, rate limiting at edge, per-connection quotas in Redis.
• Observability: per-connection metrics, end-to-end latency, broker cache hit rates.
• Sizing tests in staging with synthetic connections and failure injection.

Why this design:

• Keeps per-connection memory/CPU low by minimizing centralization of socket state.
• Brokers provide searchable metadata without being in the hot data path, enabling horizontal scale to 2M+ with controlled resource footprint and low-latency routing.

Scalability definition (role-appropriate): Scalability is the system’s ability to maintain functional correctness and defined performance SLAs as load (users, connections, messages) grows. For a mobile real-time messaging service serving 10M DAU, that means predictable latency, high delivery success, elastic capacity (connections/RPS), and graceful degradation under overload.

Concrete metrics and suggested thresholds / SLAs (real-time messaging, 10M DAU):

• Peak concurrent users (CCU): expected peak CCU = 10M * peak_ratio. Example assumption: 10% simultaneous => 1,000,000 concurrent connections. Design to 1.2M headroom.
• Connections/sec (new connects): support connection churn spikes e.g., 5k–50k connects/sec; scale to 100k/sec for events (app updates).
• Requests/sec / messages/sec: avg messages/sec = 10M DAU * msgs_per_user_peak; e.g., if peak = 0.1 msg/sec/user => 1,000,000 msg/sec. Design for 1.5–2× burst headroom.
• P95 / P99 end-to-end latency (send -> recipient receive): P95 ≤ 50 ms, P99 ≤ 250 ms for 90% of geographies; global worst-case SLA P99 ≤ 500 ms.
• Time-to-first-payload (connection handshake + first message): ≤ 150 ms p95, ≤ 400 ms p99.
• Delivery success rate: ≥ 99.99% (successful ack within SLA); expired/failed deliveries < 0.01%.
• Availability: 99.99% (four 9s) regional; 99.95% global multi-region.
• Error rate (5xx, timeouts): < 0.1% of requests.
• Fanout amplification / amplification factor: measure average recipients per message; provision throughput = messages/sec * amplification_factor.
• Backlog / queue depth: max tolerated queue depth per shard before shedding: set e.g., 1M messages; time-to-drain SLA after surge: < 10 minutes to return to baseline.
• Resource efficiency: connections per server (long-lived TCP/websocket): design for 50k–200k sockets per front-end depending on instance type; CPU/memory per 1k active connections baseline.
• Autoscale time: scale-up within 30–60s for sudden spikes; scale-down with hysteresis to avoid thrashing.
• Durability / persistence SLA (for stored messages): persisted and ACKed within 2s p95; retention SLA for offline delivery as agreed (e.g., 7 days).

Why these thresholds:

• Mobile users expect near-instant messaging; p95 targets ensure perceived real-time behavior, p99 accounts for tail latency.
• High CCU and message/sec targets prevent under-provisioning—10M DAU implies large concurrency and bursts.
• Availability and delivery rates reflect business impact (user retention, SLAs for enterprise customers).

How to measure and enforce:

• Synthetic traffic and chaos testing for target CCU and bursts.
• Real-time metrics: per-shard messages/sec, connection counts, CPU/memory, queue depth, p95/p99 latencies, error rates.
• Autoscaling policies tied to meaningful signals (queue depth, CPU, connection saturation, latency).
• Capacity planning: model fanout, compression, batching, and protobuf sizes to convert messages/sec to bandwidth and instance counts.
• Graceful degradation: prioritize control messages, rate-limit non-critical features, employ back-pressure and shedding.

Example capacity calc (simple):

• If peak CCU = 1,000,000 and avg messages/sec/user_peak = 0.1 → messages/sec = 100k. If average fanout = 10 → 1,000k deliveries/sec. With 100k deliveries/sec per region per 100 instances → need ~1,000 instances (adjust for message size, protocol overhead, geo distribution).

These metrics allow you to translate business requirements into concrete architecture choices (connection tier, pub/sub brokers, sharding, region placement, CDN edge relay, autoscaling rules) and measurable SLAs to discuss with stakeholders.

Situation: In a pre-sales call a strategic customer demanded “real‑time messaging for 50M concurrent users” but had strict budget limits. My goal as Solutions Architect was to translate that ask into a realistic, staged technical proposal that balances cost, risk, and time‑to‑value while keeping the door open to scale.

Approach / Sales conversation structure

• Clarify constraints & success metrics up front: define "concurrent" (active connections vs. idle), target latency percentiles, message size/rate, retention, security/compliance, and acceptable downtime/SLA.
• Align on business priorities: which features must exist Day‑1 (chat, presence, push) vs. nice‑to‑have (message history search, delivery receipts, per‑message encryption).
• Present risk/cost tradeoffs clearly: explain what’s feasible within budget and what requires staged investment.
• Commit to measurable pilot/KPIs and a roadmap with checkpoints to approve further spend.

Prioritized features (MVP)

1. Maintainable connection handling (WebSockets/HTTP2 for long‑lived connections)
2. Basic pub/sub routing and fan‑out
3. Simple auth + TLS
4. Horizontal stateless front‑end, minimal durable storage (recent history only)
5. Monitoring, rate limiting, and throttling to protect budget

MVP architecture (cost-aware)

• Edge Load Balancers + Autoscaling stateless gateway fleet (WebSocket/HTTP2) for connection termination — colocate gateways near users to reduce bandwidth costs.
• Lightweight, partitioned message router using managed Kafka/RabbitMQ or Redis Streams for short retention and durable fan‑out; prefer managed services to lower ops cost.
• Sharded session store in Redis (or managed Elasticache) for routing/presence; use TTLs to reduce memory cost.
• Downstream optional cold storage (S3) for long history, but not in MVP.
• Global DNS + regional failover; start single region with plan for geo expansion.
  Cost tradeoffs:
• Use managed services to reduce engineering headcount and ops cost (higher unit cost but lower TCO).
• Favor eventual consistency for presence and read‑after‑write for critical flows only.
• Limit message retention and per‑user history in MVP to cut storage costs.
• Implement rate limits and connection caps to keep peak cost predictable.

Roadmap to 50M concurrent and enhancements
Phase 0 (Pilot, weeks): 100k–1M concurrent in single region. Validate gateways, routing, throttling, and KPIs.
Phase 1 (Scale horizontally, months): Add sharding, autoscale policies, optimized fan‑out (hierarchical fan‑out), and replicate critical services across regions.
Phase 2 (Resilience & performance): Active‑active multi‑region, consistent hashing for sessions, use dedicated streaming infra (self‑managed Kafka or streaming layer) for cost efficiency at scale.
Phase 3 (Feature parity & ops): Add delivery guarantees (at‑least‑once/ exactly‑once patterns), end‑to‑end encryption, per‑message retention policies, analytics, and cost-optimization (spot instances, reserved capacity).
Optional enhancements (value-driven):

• Presence accuracy improvements, offline message queues, typing/delivery receipts
• Enterprise features: fine‑grained RBAC, audit logs, compliance export
• Push gateway integrations for mobile efficiency

How I present to the client

• Show a pragmatic MVP diagram and a clear cost vs. benefit table for each feature.
• Propose measurable milestones and a small initial commitment (pilot) with options to expand when KPIs met.
• Offer SLA tiers and estimated TCO ranges for each scale target, plus tradeoff choices (lower latency vs. lower cost).
• Reassure with risk mitigations: throttling, quotas, rollback plans, and monitoring dashboards.

Outcome expectation

• Get alignment on deliverables, budget for pilot, and objective KPIs. This approach reduces up‑front spend, de‑risks scale assumptions, and provides a clear path to 50M concurrent users with predictable investments.