Scalability for Millions of Concurrent Mobile Users Questions
Designing systems supporting millions of concurrent mobile users. Push notification infrastructure: real-time delivery to millions, handling scale. Real-time features: WebSockets, Server-Sent Events, optimized for mobile. Data consistency at scale: handling eventual consistency, conflict resolution. CDN strategies for mobile content delivery. Analytics and telemetry collection from millions of clients. Monitoring and observability for mobile apps at scale.
EasyTechnical
64 practiced
As a Solutions Architect designing systems for mobile apps, how do you define "scalability" for a service that must support millions of concurrent mobile users? List concrete scalability metrics (examples: concurrent connections, requests/sec, p95/p99 latency, time-to-first-payload) and explain measurable thresholds or SLAs you would propose for a real-time messaging feature used by 10M daily active users.
Sample Answer
Scalability definition (role-appropriate): Scalability is the system’s ability to maintain functional correctness and defined performance SLAs as load (users, connections, messages) grows. For a mobile real-time messaging service serving 10M DAU, that means predictable latency, high delivery success, elastic capacity (connections/RPS), and graceful degradation under overload.Concrete metrics and suggested thresholds / SLAs (real-time messaging, 10M DAU):- Peak concurrent users (CCU): expected peak CCU = 10M * peak_ratio. Example assumption: 10% simultaneous => 1,000,000 concurrent connections. Design to 1.2M headroom.- Connections/sec (new connects): support connection churn spikes e.g., 5k–50k connects/sec; scale to 100k/sec for events (app updates).- Requests/sec / messages/sec: avg messages/sec = 10M DAU * msgs_per_user_peak; e.g., if peak = 0.1 msg/sec/user => 1,000,000 msg/sec. Design for 1.5–2× burst headroom.- P95 / P99 end-to-end latency (send -> recipient receive): P95 ≤ 50 ms, P99 ≤ 250 ms for 90% of geographies; global worst-case SLA P99 ≤ 500 ms.- Time-to-first-payload (connection handshake + first message): ≤ 150 ms p95, ≤ 400 ms p99.- Delivery success rate: ≥ 99.99% (successful ack within SLA); expired/failed deliveries < 0.01%.- Availability: 99.99% (four 9s) regional; 99.95% global multi-region.- Error rate (5xx, timeouts): < 0.1% of requests.- Fanout amplification / amplification factor: measure average recipients per message; provision throughput = messages/sec * amplification_factor.- Backlog / queue depth: max tolerated queue depth per shard before shedding: set e.g., 1M messages; time-to-drain SLA after surge: < 10 minutes to return to baseline.- Resource efficiency: connections per server (long-lived TCP/websocket): design for 50k–200k sockets per front-end depending on instance type; CPU/memory per 1k active connections baseline.- Autoscale time: scale-up within 30–60s for sudden spikes; scale-down with hysteresis to avoid thrashing.- Durability / persistence SLA (for stored messages): persisted and ACKed within 2s p95; retention SLA for offline delivery as agreed (e.g., 7 days).Why these thresholds:- Mobile users expect near-instant messaging; p95 targets ensure perceived real-time behavior, p99 accounts for tail latency.- High CCU and message/sec targets prevent under-provisioning—10M DAU implies large concurrency and bursts.- Availability and delivery rates reflect business impact (user retention, SLAs for enterprise customers).How to measure and enforce:- Synthetic traffic and chaos testing for target CCU and bursts.- Real-time metrics: per-shard messages/sec, connection counts, CPU/memory, queue depth, p95/p99 latencies, error rates.- Autoscaling policies tied to meaningful signals (queue depth, CPU, connection saturation, latency).- Capacity planning: model fanout, compression, batching, and protobuf sizes to convert messages/sec to bandwidth and instance counts.- Graceful degradation: prioritize control messages, rate-limit non-critical features, employ back-pressure and shedding.Example capacity calc (simple):- If peak CCU = 1,000,000 and avg messages/sec/user_peak = 0.1 → messages/sec = 100k. If average fanout = 10 → 1,000k deliveries/sec. With 100k deliveries/sec per region per 100 instances → need ~1,000 instances (adjust for message size, protocol overhead, geo distribution).These metrics allow you to translate business requirements into concrete architecture choices (connection tier, pub/sub brokers, sharding, region placement, CDN edge relay, autoscaling rules) and measurable SLAs to discuss with stakeholders.
HardSystem Design
65 practiced
Discuss stateful vs stateless real-time servers. For a stateful architecture that stores per-connection session data (presence, subscriptions), describe how you'd design failover, session replication, and seamless reconnect behavior so users do not lose important session state during server crashes or rolling upgrades.
Sample Answer
Requirements:- Preserve per-connection session state (presence, subscriptions, last-ack’d message, sequence numbers) across server crashes and rolling upgrades- Low reconnect latency (<1s perceived outage), strong consistency for presence where required, eventual consistency acceptable for non-critical metadata- Scale to many concurrent connectionsHigh-level design:- Keep real-time frontends (WebSocket/TCP) stateless regarding durable session state; persist session state in a highly-available session store with fast read/write. Use in-memory replica on the active host for performance.- Components: Load balancer with consistent hashing / sticky affinity token, Real-time servers (connection handlers), Session store (Redis Cluster with AOF + clustering OR CockroachDB for strong consistency), Replication/coordination layer (Raft/etcd for leader election or Redis primary-replica with Sentinel), Message broker (Kafka) for durable event stream.Session lifecycle & replication:1. On connect, client presents a resume token (session ID + last-seq). If new, create session record in session-store; assign a short-lived lease owned by the handling server.2. Active server keeps session hot-copy in local memory for latency. All state changes are persisted immediately (write-through) to session-store and appended to per-session event stream in the broker. Optionally use Redis streams or Kafka per-session partitioning.3. Replication: session-store runs in clustered mode (multi-AZ Redis Cluster or CockroachDB) so writes are durable and accessible to any server. For sub-100ms reads, local server uses a write-behind cache but guarantees persistence before acknowledging critical ops.Failover & seamless reconnect:- Lease/heartbeat: server renews a short lease in session-store every N seconds. If lease expires, other servers may claim session.- Client reconnect flow: client reconnects to load balancer; LB routes by session ID via consistent-hash cookie to likely previous server or any server that can claim lease. New server reads persisted session and event-stream, replays missed events up to last-ack using sequence numbers, restores subscriptions and presence.- Resume tokens and sequence numbers ensure idempotent replay; server sends only events with seq > client-last-seq.- Presence: for strong presence, use ephemeral keys with TTL in session-store and a central presence topic; on handover, new server emits join/leave events only when lease transitions occur to avoid flapping.- Rolling upgrades: perform graceful drain—stop accepting new connections, wait for in-flight ops, persist final state, transfer lease, then close connections. Use canarying and short TTL leases to limit risk.Consistency and trade-offs:- Trade-off: write-through persistence adds latency; mitigate with batching non-critical updates and local cache for reads. Use stronger consistency (CockroachDB / Redis with WATCH for CAS) for presence and order-sensitive state; use eventual-consistency CRDTs for multi-subscriber lists if conflict tolerance helps scale.- For massive scale, shard sessions by consistent hash to reduce cross-node contention; use Kafka for durable replay and replay speed.Operational notes:- Monitor lease expirations, replication lag, and client resume failures.- Test chaos scenarios (kill server, network partitions) and measure recovery time objective (RTO) and message duplication/ordering guarantees.This design guarantees fast reconnects, durable session state, and controlled trade-offs between latency and consistency.
HardTechnical
84 practiced
You expect occasional viral events that multiply traffic by 10x for short periods. As a Solutions Architect, outline a plan to handle such spikes: capacity planning, autoscaling policies, pre-warming, graceful degradation strategies, circuit breakers, and a runbook for operational response during the spike.
Sample Answer
Situation overview: We expect short, infrequent 10x traffic spikes (viral events). The goal is to remain available and cost-effective while protecting core functionality.1) Capacity planning- Model baseline RPS, p95/p99 resource use, and 10x peak. Calculate CPU/memory, DB connections, and network needs for peak QPS.- Identify choke points: DB writes, auth, external APIs.- Plan reserve: keep a small warm pool (10–20% extra instances) and optional fast scale targets (e.g., container cluster node groups, provisioned concurrency for serverless).2) Autoscaling policies- Use multi-dimensional autoscaling: request-based (targeted CPU/RPS), queue-depth for background workers, and custom metrics (p50 latency, error rate).- Fast scale-up: aggressive thresholds (e.g., 70% CPU or 70% of request concurrency) with short evaluation window (1–2 min) and scale increment steps of 2–4 instances.- Conservative scale-down: longer cooldown (10–15 min) to avoid thrash.- Horizontal scaling preferred; use target tracking + step policies for predictable jumps.3) Pre-warming- For predicted virality (marketing events), pre-warm autoscaling groups, container node pools, DB read replicas, and CDN edge cache rules.- For serverless, use provisioned concurrency or scheduled warm invocations.- Warm caches: populate caches with top endpoints/content.4) Graceful degradation & throttling- Define core vs non-core features; degrade non-essential UI components, async analytics, image processing.- Implement rate limits per-user and global (leaky-bucket/token-bucket) with 429 responses and Retry-After header.- Return cached/stale content when upstream is overloaded.5) Circuit breakers & bulkheads- Implement per-dependency circuit breakers (failure thresholds, automatic retry windows), bulkheads to isolate subsystems (auth, payments).- Use client-side/time-limited retries with exponential backoff + jitter; avoid cascading retries.- Fail fast for non-critical downstreams; queue requests for async processing.6) Observability & alerts- Dashboards: RPS, error rate, latency p50/p95/p99, CPU, queue depth, DB CPU/connections, cache hit-rate.- Alerts: sustained 50%+ above baseline RPS, p95 latency > threshold, error rate spike > X%.- Integrate synthetic probes and real-user monitoring.7) Runbook (operational response)- Triage: confirm spike source and validity (analytics, marketing).- Immediate actions: a) Enable rate limiting / increase thresholds for anonymous users. b) Activate pre-warmed capacity or manually scale critical ASGs. c) Turn off or throttle non-core features via feature-flag service. d) Enable CDN/edge caching overrides. e) Open incident channel, assign owners: infra, app, DB, networking.- Mitigation: monitor impact of each action; if DB is saturated, switch to read replicas, enable write-queueing.- Recovery: once traffic falls and metrics stabilize for 15–30 min, gradually scale down, re-enable features, and close incident.- Post-incident: run blameless postmortem, update capacity model, refine autoscaling rules and runbook.Trade-offs: faster scaling increases cost and risk of overshoot; conservative policies save cost but risk throttling. Mitigate by combining pre-warm for planned events and rapid autoscale + graceful degradation for unplanned spikes.
MediumTechnical
85 practiced
Cost optimization: For a service that needs real-time push and persistent connections for millions of mobile users, compare serverless (FaaS), container-based (Kubernetes), and specialized managed services. Discuss cost models, connection handling limitations, and when each model is appropriate.
Sample Answer
Situation: A customer needs real-time push + persistent connections for millions of mobile users and asks which hosting model optimizes cost and meets scale.Overview — three options:1) Serverless (FaaS)- Cost model: pay-per-invocation + short execution time; cold-starts can add latency. Good for bursty event-driven logic.- Connection handling: FaaS platforms are not designed to hold millions of long-lived TCP/WebSocket connections; providers limit connection lifetime and concurrency. Often requires offloading persistent socket handling to a gateway or managed service.- Appropriate when: business logic is sporadic, traffic is spiky, and persistent connections are handled by a separate managed layer (e.g., push notifications, webhooks, or ephemeral websocket handlers).2) Container-based (Kubernetes)- Cost model: pay for provisioned nodes (VMs) and cluster overhead; better utilization if you can bin-pack workloads, autoscale via HPA/Cluster-Autoscaler reduces idle cost but baseline capacity needed for connection endpoints.- Connection handling: can host long-lived connections directly (sticky services or scale via load balancers); requires careful capacity planning (connection counts per node), connection proxies, and state management (sticky sessions or external state).- Appropriate when: you need full control over runtime, custom protocols, predictable high sustained load where amortized node cost is lower.3) Specialized managed services (e.g., AWS AppSync/IoT Core/Managed WebSocket gateways, Firebase Cloud Messaging, Pub/Sub)- Cost model: typically per-connection + per-message or per-GB; higher unit cost but minimal operational overhead and built-in scaling.- Connection handling: built for millions of persistent connections, built-in reliability, backpressure, regional replication.- Appropriate when: scale requirement is very high, time-to-market and operational simplicity matter, or when predictable per-connection pricing is acceptable.Trade-offs & recommendation:- For millions of persistent mobile connections, a hybrid is common: use a managed persistent-connection service or WebSocket gateway for sockets + serverless for transient business logic, or Kubernetes if you must control protocol/state or reduce per-connection costs at scale.- If cost-sensitive and sustained heavy connections -> Kubernetes with optimized networking and autoscaling (higher ops burden).- If ops/time-to-market-sensitive and predictable pricing OK -> Managed service.- If workload is event-driven with few persistent connections -> Serverless.Key considerations for final decision:- Estimate sustained vs peak connections, messages/sec, message size, retention and throughput.- Model TCO: instance costs, per-connection fees, ops headcount, monitoring, and failover design.- Pilot with representative traffic to validate cost and scaling assumptions.
MediumTechnical
46 practiced
How would you design A/B testing and feature-flag rollout mechanisms for push notifications and real-time features that minimize client churn and ensure stable experiments at scale? Include membership assignment, bucketing, instrumentation, and rollback strategies.
Sample Answer
Requirements (clarify): minimize churn, stable experiment stats, support push notifications & real-time features, large scale, safe rollouts and fast rollback.High-level approach:- Use feature-flag service + experimentation platform separated from product code. Flags control behavior; experiment layer handles assignment and metrics.Membership assignment & bucketing:- Deterministic, stateless bucketing by stable customer key (user_id or device_id) + experiment_id + salt using a consistent hash (e.g., MurmurHash → uniform bucket 0-10000). Store bucketing config (seed, traffic allocation, stratification) in central config.- Support stratified bucketing: tier, geography, client version, platform to avoid imbalances. Allow forced holds for VIPs and opt-outs.- Persist exposures for users in event of re-evaluation: write assignment to user profile or a low-latency store (Redis) for sticky membership (important for push sequences).Instrumentation & observability:- Emit deterministic exposure events (experiment_id, variant, user_id, timestamp, context) at the moment of decision—both client and server-side. For push flows, also emit delivery, open, action, and unsubscribe events linked by message_id.- Use unique correlation IDs for real-time events. Stream events into a real-time pipeline (Kafka → processing → metrics DB) with schemas and validation.- Monitor key guardrails: engagement, retention, error rates, unsubscribe/churn, latency. Set automatic anomaly detection and dashboards.Rollout & ramp strategy:- Phased rollout: 0 → 1% → 5% → 25% → 100% with health checks at each step. For push campaigns, run canary on internal/test accounts first.- Implement kill-switch and fast rollback at flag level with sub-second propagation (CDN/in-memory caches invalidation + push to clients via SSE/WS or mobile remote-config TTLs).- For experiments: automated halting rules (statistical or heuristic) if guardrail thresholds breached (e.g., >X% increase in unsubscribes) or if p-value/CI shows harm.Stability & scale considerations:- Determinism avoids split leakage. Persist assignments so users don't flip variants mid-sequence (critical for push sequences and multi-step real-time features).- Throttle event volume with sampling for non-critical telemetry; keep 100% for guardrail events.- Ensure idempotency and deduplication in ingestion; use compacted topic for assignment snapshot.Rollback & safety:- Immediate flag toggle to previous variant; revertable deployment pipelines. For push, stop further notifications and send mitigations (e.g., apology or opt-back) if needed.- Post-rollback: run root-cause analysis, compare cohorts (exposed vs persisted), and publish learnings.Trade-offs:- Persisting assignments increases storage and complexity but prevents churn from re-bucketing. Server-side evaluation gives control; client-side required for offline/latency—use hybrid approach: server authoritative, client caches sticky assignment with TTL.This design balances determinism, observability, fast control, and customer-safety for push and real-time features at scale.
Unlock Full Question Bank
Get access to hundreds of Scalability for Millions of Concurrent Mobile Users interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.