Third-Party and SaaS Integration Questions

Definition & why it matters (mobile perspective)
Idempotency means making an API operation safe to repeat: executing the same request multiple times has the same effect as doing it once. For mobile apps — where networks drop and clients often auto-retry — idempotency prevents duplicate charges, repeated orders, or duplicate resources.

Examples

• Idempotent:
  • GET /order/123 (safe, read-only)
  • PUT /order/123 with full order body (replaces resource; repeated PUT yields same state)
  • DELETE /order/123 (repeated deletes still result in resource gone)
• Non-idempotent:
  • POST /payments (creates new payment each call)
  • POST /orders (creates duplicate orders if retried)

How servers & clients support idempotency

• Idempotency keys: client generates a UUID per logical operation, sends header (e.g., Idempotency-Key). Server stores key → result; if repeated, server returns same response without reprocessing. Mobile: persist key until final success/failure then drop.
• Use PUT/PATCH semantics where appropriate (use PUT to create-or-replace at known id).
• Design safe retry strategy: exponential backoff, detect network errors vs HTTP 4xx/5xx, avoid blind retries for non-idempotent endpoints.
• Server-side dedup logic: transactional checks (unique constraint on business keys) and idempotency logs.

Practical tip for mobile devs
Generate stable idempotency keys for user actions (UUID + timestamp or client-generated order id), persist across app restarts until server confirms. Coordinate with backend: document which endpoints accept idempotency keys and expected semantics.

Approach overview
Describe client-side throttling as a cooperative layer: limit request rate locally (token/leaky bucket), respect server signals (Retry-After, 429), queue non-urgent work, and surface graceful degradation to users.

Token-bucket vs Leaky-bucket

• Token-bucket: tokens added at rate r, request consumes token; supports bursts up to bucket size. Good for user-initiated actions that can burst (e.g., search).
• Leaky-bucket: fixed outflow rate; smoothes traffic strictly — simpler if you must strictly pace background syncs.

Practical mobile implementation (Kotlin pseudo)

// simple token-bucket
class TokenBucket(val capacity: Int, val refillPerSec: Double) {
  var tokens = capacity.toDouble()
  var last = System.nanoTime()
  fun tryConsume(n:Int=1):Boolean {
    val now = System.nanoTime()
    tokens = min(capacity.toDouble(), tokens + (now-last)/1e9*refillPerSec)
    last = now
    if (tokens >= n) { tokens -= n; return true }
    return false
  }
}

Retry / Backoff

• Honor Retry-After and 429: parse header, schedule retry after suggested delay.
• If missing, use exponential backoff with full jitter to avoid stampedes:
  backoff = random(0, base * 2^attempt)

Queueing & Background Work

• Classify tasks: real-time (UI) vs opportunistic (sync, prefetch).
• Keep a persisted queue (SQLite/Room, file) for background jobs; drain when network unconstrained and tokens available.
• Integrate with OS background constraints (WorkManager / BGTasks).

UX & Graceful Degradation

• Surface minimal, actionable feedback: inline "Try again later" or spinner with countdown when Retry-After known.
• Disable or de-prioritize non-critical features (image prefetch, analytics) under throttling.
• Offer offline fallbacks: show cached content, save actions locally and sync later; show clear status (e.g., "Limited connectivity — background sync paused").

Monitoring & Telemetry

• Log 429s, Retry-After usage, queue length and retry attempts to help tune token rates server-side.

This approach preserves battery/network, avoids server penalties, and keeps users informed while maximizing useful behavior.

Situation & goal
Receive intermittent network failures on some Android devices — need to reproduce, collect evidence, isolate root cause (client, carrier, server), and propose fast mitigations.

1) Reproduce

• Identify affected cohort: OS versions, OEMs, carriers, app versions, geographic regions, time-of-day.
• Create focused test plan: same app/build, replicate network conditions (2G/3G/4G/5G, carrier roaming, VPN, captive portals), toggling Doze/Battery Saver, toggling metered networks.
• Use device farm + local devices matching affected models.

2) Logs & telemetry to gather

• Client-side: Android logcat (network, ConnectivityManager, TelephonyManager), app-level network stack logs (OkHttp/Retrofit), timestamps, request IDs, retry counts, DNS resolution times, certificate errors.
• Metrics: failure rates by device/carrier/region, latency, RTT, TLS handshake times, DNS failures.
• User-visible: screenshots, repro steps, timestamps, cellular signal strength (RSSI), cell tower IDs.

3) Use device farms & packet captures

• Run deterministic tests on Firebase Test Lab / AWS Device Farm for affected models.
• On real devices: tcpdump/pcap via adb or use Android’s built-in VPN-based packet capture or rely on a tethered laptop with adb reverse and capture via Wireshark. Capture DNS, TCP/TLS handshakes, retransmissions, RSTs, ICMP.
• Correlate packet timestamps with app logs and server logs.

4) Isolate client vs carrier vs server

• Client: reproduce failures on Wi‑Fi vs cellular. If Wi‑Fi works but cellular fails on same device, suspect carrier or radio handling. Inspect app stack for improper network callbacks, threading, socket timeouts, keepalive.
• Carrier: failures only on specific carriers/regions and visible in pcap as DNS hijack, proxy insertion, packet drops, NAT timeouts, or TCP resets.
• Server: server logs show errors, 5xx, or TLS renegotiation failures; failures across many carriers/devices point to server-side.
• Perform controlled tests: route traffic through a known-good VPN — if issue disappears, likely carrier. Use curl from server-side to validate endpoints.

5) Quick mitigations to ship

• Short-term: increase client-side timeouts, add retry with exponential backoff and jitter, switch to alternate DNS (app-level DNS over HTTPS), enable robust network error handling and user-friendly retry UI.
• Fallbacks: degrade to HTTP/long-polling if TLS path problematic (only temporarily, consider security), disable multiplexing features (HTTP/2) if causing issues.
• Telemetry: ship enhanced logging flags and a safe diagnostic mode to capture full traces from affected users.

6) Long-term fixes

• Patch client bugs found, coordinate with carrier or CDN for fixes, harden server timeouts, and add monitoring (SLOs, alerting by device/carrier). Document reproductions and add automated tests on device farm.

This plan prioritizes quick data-gathering, deterministic repro, clear isolation steps, and safe short-term mitigations while preserving security and UX.

Situation / Goal
Design a pragmatic monitoring & instrumentation plan for mobile network requests that enables end-to-end correlation, performance SLOs, and privacy-safe logs.

What to log (no PII)

• Request metadata: HTTP method, URL path (mask query params), status code, response size, content-type
• Non-PII IDs: X-Request-ID (UUID), X-Session-ID (generated per app session), app version, device model, OS version, network type (wifi/cellular)
• Timestamps: client send, client receive, server receive, server respond (when available)
• Error context: HTTP error body hashes, network error types, stack trace id (not full trace if sensitive)

Correlation

• Client generates X-Request-ID for each request and adds X-Session-ID per app session.
• Send both to server; server logs them and returns X-Request-ID in response for client confirmation.
• Use these IDs to join mobile telemetry with backend traces.

Example header injection (Android/Kotlin):

val requestId = UUID.randomUUID().toString()
val sessionId = getSessionId()
request = request.newBuilder()
  .addHeader("X-Request-ID", requestId)
  .addHeader("X-Session-ID", sessionId)
  .build()

Performance metrics

• Latency percentiles: p50, p90, p95, p99 on client-side and server-side
• Error rate by endpoint and network type
• Retries, timeouts, throughput (requests/sec)
• Cold start time for background services (if applicable)

Sampling strategy

• Full logging for errors and slow requests (above p95)
• 1–5% sampling for success paths; increase for releases/experiments
• Adaptive sampling: raise sample on regression detection

Tools

• Client-side: Firebase Performance for traces, Sentry for crashes & breadcrumbs, custom metrics via Firebase Analytics
• Server-side: New Relic or Datadog for APM and traces
• Unified view: Export IDs to backend logs (ELK/Datadog), link to Sentry errors and tracing systems (OpenTelemetry)

Avoiding sensitive data

• Never log bodies/headers containing auth tokens, cookies, PII (emails, phone, GPS).
• Mask or redact query params and request bodies before logging.
• Use hashing for identifiers when needed (sha256) and store mapping only in secure vault if required.
• Enforce policy via linting and automated tests that scan logs for patterns (regex for emails, SSNs, tokens).

This plan gives end-to-end visibility, actionable SLIs, and preserves user privacy while enabling rapid debugging.

Approach (brief)

Design a RetryFramework that:

• Distinguishes idempotent vs non-idempotent calls
• For non-idempotent: generates/sends idempotency key header on first attempt and persists it so retries survive restarts
• Supports configurable backoff, maxAttempts, and key cleanup policy

Kotlin pseudocode

// Interfaces
interface Request { val url: String; val headers: MutableMap<String,String>; val body: ByteArray? }
interface HttpClient { suspend fun send(req: Request): HttpResponse }
data class RetryConfig(val maxAttempts: Int = 5, val baseDelayMs: Long = 200, val maxDelayMs: Long = 5000,
                       val jitter: Boolean = true)

// Simple persistence (file/DB abstraction)
interface KeyStore { fun save(keyId: String, key: String); fun load(keyId: String): String?; fun delete(keyId: String) }

// Retry framework
class RetryFramework(private val client: HttpClient, private val keyStore: KeyStore, private val config: RetryConfig) {
  suspend fun sendWithRetry(req: Request, idempotent: Boolean, keyId: String? = null): HttpResponse {
    var attempt = 0
    val idempKey = if (!idempotent) {
      // reuse existing persisted key if present, or generate+persist
      keyId?.let { keyStore.load(it) } ?: run {
        val generated = UUID.randomUUID().toString()
        keyId?.let { keyStore.save(it, generated) }
        generated
      }
    } else null

    while (true) {
      attempt++
      // attach idempotency header only for non-idempotent
      idempKey?.let { req.headers["Idempotency-Key"] = it }

      val resp = try { client.send(req) } catch (e: Throwable) { null }

      if (resp != null && resp.isSuccessful()) {
        // success -> cleanup persisted key when server confirms idempotent outcome
        if (idempKey != null && keyId != null) keyStore.delete(keyId)
        return resp
      }

      if (attempt >= config.maxAttempts) {
        // give up; keep key persisted so external retry after restart can continue (policy)
        throw RetryFailedException("failed after $attempt")
      }

      // backoff with optional jitter
      val delayMs = calculateBackoff(config.baseDelayMs, attempt, config.maxDelayMs, config.jitter)
      delay(delayMs)
    }
  }

  private fun calculateBackoff(base: Long, attempt: Int, max: Long, jitter: Boolean): Long {
    val exp = min(max.toDouble(), base * 2.0.pow(attempt - 1)).toLong()
    return if (jitter) (exp / 2 + Random.nextLong(exp / 2)) else exp
  }
}

Key points & trade-offs

• Persist idempotency key keyed by a developer-provided keyId (e.g., orderId) so restarts can continue retries.
• Cleanup only after confirmed success to avoid accidental duplicate side-effects.
• Keep maxAttempts local but allow long-running/retry-on-restart via persisted key and external orchestration.
• Consider secure storage (KeyStore) and TTL-based cleanup (GC job to remove old keys).