Structured Problem Solving and Decomposition Questions

A symptom is an observed effect (what you see); the root cause is the underlying reason that produces that effect (why it happens). Fixing a symptom may temporarily hide the problem; fixing the root cause prevents recurrence.

Example: A production web service's p95 latency jumped from 150ms to 800ms after a deployment.

Steps to go from symptom to root cause and verify the fix:

1. Observe & quantify (symptom)

• Confirm metrics: p50/p95/p99, error rate, throughput, deployment timeline, and any correlated alerts.
• Note scope: all users or specific region/endpoints?

2. Narrow the blast radius

• Compare versions, traffic slices, and hosts. Check recent commits and rollout windows.

3. Collect traces & logs (diagnostics)

• Use distributed tracing to find where time is spent (frontend, auth, DB call). Inspect slow traces to identify slow spans.
• Check application logs for exceptions, timeouts, or retries.

4. Correlate infra metrics

• CPU, memory, GC, thread pool saturation, connection pool exhaustion, database slow queries, and network latency on affected instances.

5. Form hypothesis

• E.g., p95 spike caused by a new code path making synchronous DB calls in a hot endpoint, exhausting DB connections and causing retries.

6. Test hypothesis

• Reproduce in staging with similar load and the new version. Enable sampling to capture slow requests.
• Run targeted experiments: disable the new code path (feature flag) or increase DB pool temporarily and observe latency.

7. Prove root cause

• If disabling the code path or reverting the deploy returns p95 to baseline while other variables remain constant, that proves causation.
• Cross-check logs/traces: slow DB queries and connection queue lengths should drop when the code path is disabled.

8. Implement fix addressing root cause

• Options: make DB calls asynchronous/batched, add caching, optimize query/indexes, increase pool with backpressure protection.

9. Confirm fix in production

• Deploy fix to a canary, monitor p95, error rate, DB metrics, and tracing. Verify sustained improvement and no new regressions.

10. Postmortem & prevention

• Document root cause, roll-out safeguards (feature flags, circuit breakers), add alerting on the true leading indicators (connection queue length, slow DB queries), and add regression tests or load tests.

Key idea: always move from symptom → measurement → hypothesis → controlled experiment → proven fix, and instrument to detect the root cause early next time.

Situation: At my previous company we were asked to build a new analytics dashboard to surface user engagement across mobile and web. The product brief was high-level—“single pane of glass for engagement”—with no clear scope, and a three-quarter timeline.

Task: As the engineer owning delivery, I had to decompose the work into actionable sprint-sized items, pick the first deliverable to unblock stakeholders, estimate effort for each subcomponent, and create a measurable plan for iterative rollout.

Action:

• I held a 2-hour discovery with PM, designer, and a data engineer to list core capabilities (data ingestion, ETL, API, frontend widgets, auth, reporting).
• I applied the “riskiest-first” principle and chose the first deliverable: a read-only prototype showing weekly active users (WAU) for web only. Rationale: smallest end-to-end vertical slice that validated data pipeline, API, and UI assumptions while giving stakeholders immediate insight.
• I decomposed that slice into subcomponents: data extraction (4 dev-hours), ETL transform and validation (8h), simple API endpoint (6h), frontend chart + UI (8h), tests & infra (6h). Estimates came from historical velocity, pair-estimates with the data engineer, and padding (20%) for unknowns.
• Implemented the prototype in a single two-week sprint, instrumented metrics (pipeline latency, API 95th percentile, chart render time) and added a feedback channel with PM and two power users.

Result:

• Prototype delivered end of sprint; stakeholders validated WAU definition and data quality. Metrics showed ETL latency <2 min and API 95th <120ms. Based on feedback (need mobile data next, and CSV export), we prioritized adding mobile ingestion and export for the next two sprints. The staged approach reduced rework, and final dashboard shipped in three iterations with 30% less estimated effort than an initial monolithic plan.

Requirements & constraints:

• 100M users, zero downtime, strong consistency for reads/writes, limited DB IOPS/CPU, restore point strategy, rollback window.

Plan (phases):

1. Discovery

• Inventory address fields, nullability, indexes, FK, triggers, and usage (read/write patterns). Estimate row size, total bytes, and throughput budget (IOPS/transactions/sec).
• Create sample queries to quantify:
  SELECT COUNT(*) FROM users WHERE address IS NOT NULL;
  SELECT AVG(LENGTH(address)) FROM users;
  EXPLAIN ANALYZE SELECT id FROM users ORDER BY id LIMIT 1; -- baseline

2. Schema deploy (non-disruptive)

• Add new columns/table (address_v2) with defaults NULL and appropriate indexes off-peak. Use rolling schema changes if DB supports (online DDL).

3. Dual-write or CDC

• Prefer CDC (Debezium/Kafka) to avoid app changes risk. If app update feasible, implement dual-write guarded by feature flag: write to old and new schema atomically (service-level transaction or two-phase writes with compensating on failure).
• Example pseudocode: begin tx; UPDATE users SET address=..., address_v2=to_struct(...); commit;

4. Backfill with throttling

• Backfill via batched, id-range or primary-key cursors with per-worker rate-limits and concurrency cap.
• Example scheduler: shards = 1000; per-shard batch_size=1000; sleep_between_batches = max(0, desired_latency_window - elapsed)
• Use token-bucket: allow X writes/sec globally (X = measured safe baseline * 0.6).

5. Read versioning / serving

• Implement read routing: prefer address_v2 if present else fallback to address. Add a feature flag to switch entirely when ready.
• Query pattern:
  SELECT id, COALESCE(address_v2_json->>'line1', address) as address_served FROM users WHERE id = :id;

6. Reconciliation & validation

• Reconcile samples and full checks via queries:
  -- Count mismatches where both present but different
  SELECT COUNT(*) FROM users WHERE address IS NOT NULL AND address_v2 IS NOT NULL AND address != address_v2_normalized;
  -- Find structural mismatches
  SELECT id FROM users WHERE address_v2 IS NULL AND address IS NOT NULL LIMIT 100;
• Normalize before compare (trim, lower, canonicalize zip). Run progressive checks: sampled (1M rows random), then full scan in parallel small shards.
• Auto-fix rules for trivial mismatches (whitespace, punctuation) executed in separate backfill jobs; flag complex ones for manual review.

7. Monitoring & alerts

• Track: backfill throughput (rows/sec), DB CPU, latency, replication lag, error rate, mismatch rate, queue backlog.
• Dashboards + alerts when CPU > 70% or latency > SLA, or mismatch rate increases unexpectedly.
• Canary: run on 1% users first; monitor business metrics (checkout failures, address validation rejections).

8. Rollback strategy

• If using dual-write: flip reading back to old column; stop backfill; revert feature flag; run compensating writes if needed.
• With CDC-only: pause consumers, roll back consumer offsets, or replay corrected events.
• Prepare point-in-time restore and export snapshots of both schemas before destructive steps.

Scheduling/throttling specifics

• Baseline: measure safe write QPS during off-peak. Set global cap = baseline*0.5. Start with 1% of shards, increase geometrically (1%,5%,25%,100) with 1–2 hours observation windows.
• Use dynamic feedback: if DB metrics exceed thresholds, reduce rate by factor 0.5; if healthy for N intervals, increase by 2x up to cap.
• Backfill sample worker implementation (Python pseudocode):

# token-bucket throttle
while shards:
  if tokens.consume(batch_size):
    rows = read_next_batch()
    write_batch_to_v2(rows)
  else:
    sleep(0.1)

Outcome & acceptance criteria

• Mismatch rate < 0.01% after normalization, successful canary for 24–48h, read path switched, rollback tested, and monitoring in place.

This plan minimizes risk by using CDC/dual-write, throttled backfills, progressive rollout, automated reconciliation, and clear rollback paths.

Overview / goals: Deliver a lightweight, reliable "Save for later" across web and mobile in two weeks. Prioritize core UX: user can save/un-save items and view their list. Defer complex sync/merge logic and offline-first features to later phases.

Decomposition:

• Backend API: CRUD endpoints for saved items, simple auth, pagination.
• Mobile clients (iOS/Android): add Save button + Saved list screen, local cache for immediate feedback.
• Web UI: Save button, Saved items page, small toast/confirmation.
• Sync logic: MVP = server source-of-truth; clients push actions; basic client-side polling or pull-on-open.
• Analytics: events for save/un-save, view saved-list, conversion after restore.
• Testing: unit tests, API contract tests, end-to-end happy path flows, basic manual QA on devices.

MVP scope (2 weeks):

• Server: authenticated endpoints: list, add, remove, exists.
• Clients: add UI controls, call APIs, show list; optimistic UI with rollback on error.
• Analytics: track events.
• Tests: automated API/unit tests + smoke test matrix for primary devices.

Minimal API contracts (JSON):

• POST /saved
  Request: { "user_id": "...", "item_id": "...", "metadata": {...} }
  Response: 201 { "saved_id": "...", "item_id": "...", "created_at": "..." }
• DELETE /saved/{item_id}
  Response: 204
• GET /saved?user_id=...&page=...&limit=...
  Response: 200 { "items":[{ "item_id","metadata","saved_at"}], "next_page": null }

Phased plan for cross-device sync & edge cases:
Phase 1 (weeks 3–5): Add last-modified timestamps per save, client-side delta-sync (GET /saved?since=timestamp). Add conflict policy: server-wins; log conflicts for monitoring.
Phase 2 (weeks 6–9): Offline queue with background sync, exponential backoff, merge strategy for concurrent edits (client merge for non-conflicting fields, server tombstones for deletes), optimistic UI improvements.
Phase 3 (weeks 10+): Real-time sync via push (WebSockets/FCM), deduplication, large-scale performance tuning, comprehensive integration tests.

Trade-offs:

• Simplicity over completeness: server as source-of-truth avoids complex merge logic initially, enabling fast delivery.
• No real-time push in MVP to save time and infra work; use pull-on-open to keep UX acceptable.
• Server-wins conflict resolution reduces client complexity but may surprise power users; mitigated by analytics and phased improvements.
• Minimal metadata stored to reduce schema changes; expand later when usage patterns justify it.

Success criteria:

• Feature live on web + mobile, <2% error rate on saves, >90% happy-path pass rate in QA, instrumentation capturing usage for next-phase prioritization.

Scope & goals:

• Restore checkout reliability quickly, protect revenue, preserve UX while designing durable fixes and contractual protections.

Immediate (hours) — short-term mitigations

1. Circuit breaker + rate limiter: stop calling provider when error spike > threshold (e.g., 1% failure over 5m).
2. Retry with idempotency and exponential backoff (max 2 retries, jitter). Ensure transactions are idempotent to avoid double-charges.
3. Fallback: route eligible traffic to fallback path (local tokenization / queued offline processing / alternate provider) for low-risk payments.
4. Feature-flag deployment for toggling retries/fallbacks quickly.

Example retry pseudocode:

# idempotent retry with exponential backoff
def call_payment(payload, idempotency_key):
    for attempt in range(3):
        resp = provider.charge(payload, idempotency_key)
        if resp.success: return resp
        if not transient_error(resp): break
        sleep((2**attempt) * 0.2 + random_jitter())
    raise PaymentError(resp)

Medium-term (days–weeks)

1. Root-cause analysis: collect request traces, provider logs, HTTP codes, latency, geographic patterns, and payload shapes.
2. Harden integration: strict validation, timeout tuning, improved error classification (4xx vs 5xx vs network).
3. Automated canaries & synthetic transactions exercising edge cases.

Long-term (weeks–quarters)

1. Multi-provider architecture: abstract payment gateway with strategy pattern, dynamic routing, reconciliation, and per-provider feature matrix.
2. SLA negotiation & penalties: demand SLOs (availability, P95 latency), credits, and response-time commitments. Add audit/logging access.
3. Business continuity: documented runbooks, regular failover tests, and contractual exit clauses.

Monitoring & alerting

• Metrics: success rate, error rate by code, latency percentiles, payment throughput, % of retries, fallback usage, reconciliation failures.
• Alerts: page on >0.5% failure sustained for 10m or >2% for 2m; escalation for reconciliation mismatches.
• Dashboards: real-time flows with drill-down by region, merchant, and payment type.
• Logs & traces: include idempotency key, provider request/response IDs, correlation IDs.

Rollback & incident playbook

1. Trigger: detection of spike or failed canary.
2. Actions: enable circuit breaker → route to fallback/alternate provider → disable nonessential features → notify stakeholders.
3. Rollback: revert to previous stable integration code or switch routing to alternate provider via config flag; verify reconciliation.
4. Post-incident: blameless postmortem with timeline, RCA, and action items (within 72h).

Customer communication templates

• Short outage alert (email/in-app):
  “We’re aware of an intermittent payment issue affecting checkout for some customers. Our team is applying fixes now; most purchases should succeed with retries. If you experienced a failed charge, please check your bank — we’ll notify you if any duplicate charges are found. Estimated resolution: 2 hours.”
• Follow-up/resolution:
  “The payment issue has been resolved. If you experienced a failed transaction, you may retry or contact support (link). We will proactively refund or reconcile any duplicate charges. We apologize and appreciate your patience.”

Decision criteria for switching providers (quantitative + qualitative)

• Quantitative:
  • Persistent availability < 99.5% over 30 days despite mitigations.
  • Error rate after retries > 0.5% for key geos or payment types.
  • Unacceptable MTTR: average incident resolution > 4 hours or > 3 incidents/month.
  • Business impact: revenue loss or chargeback increase > predefined threshold.
• Qualitative:
  • Poor transparency or lack of logs/traceability.
  • Uncooperative support or missed SLA commitments.
  • Incompatibility with required features (3DS, tokenization, fraud tools).
  • Cost & onboarding complexity vs benefit.
    If one or more thresholds met and contractual remediation fails, start procurement: RFP, parallel testing, data migration plan, and phased cutover with reconciliation.

Ownership & timeline

• Incident lead (on-call SE) for hours; payment product manager for vendor negotiations; engineering to deliver multi-provider abstraction in 4–8 weeks with canary rollouts and reconciliation automation.

This plan balances rapid containment and customer-facing mitigations with engineering investments and contractual risk control to avoid recurring outages.