End-to-End Feature Design and Development Questions

API contract (HTTP/JSON)

• Endpoint: GET /api/v1/products/{productId}/related?limit=8&context=cart
• Success: 200 OK

{
  "productId":"123",
  "generatedAt":"2026-03-02T12:00:00Z",
  "strategy":"collab_filter_v2",
  "items":[ {"id":"234","reason":"also_bought","score":0.87}, ... ]
}

• Errors: 5xx when service down; 503 with Retry-After for throttling; 200 with empty items allowed.

Service responsibilities

• Recommendation service: compute and serve ranked related items + explainers and generation timestamp.
• Product service: enrich IDs with display metadata (price, image) via batched calls.
• API gateway: auth, input validation, rate-limit, response caching.

Latency SLOs

• P95 ≤ 120ms for cache hits, P95 ≤ 400ms for cold path. Target client render-safe timeout = 600ms.

Caching policy & TTL

• CDN/edge cache: public GET TTL = 60s for high traffic pages.
• Edge-stale-while-revalidate: 30s.
• Service-level in-memory cache (Redis): TTL = 30s, hot items = 5–10m with LRU.
• Client-side: cache response for 15s; revalidate on focus.

Freshness requirements

• Near-real-time for inventory-sensitive fields (price/availability) — always rehydrate those from product service when displaying. Recommendations can be slightly stale (≤ 60s acceptable).

Fallback behavior

• If recommender unavailable: API returns 200 with fallback items: top-sellers or category-based recommendations with "strategy":"fallback_static" and generatedAt.
• If enrichment fails, return IDs with minimal metadata and frontend shows skeletons. Log incidents and emit alerts.

Evolving recommendation logic

• Versioned API path (/v1, /v2) and a "strategy" field in response so clients can opt-in.
• Backward-compatible schema: additive fields only; never remove keys without deprecation cycle.
• Feature flags and A/B test gates for rolling new algorithms.
• Contract tests + consumer-driven contract CI to detect breaking changes.
• Deprecation plan: announce, support both versions for 3 months, provide migration docs and server-side translation layer if needed.

Clarify requirements & goals

• Detect and fix all mismatches between our Order Service and Payment Provider (settlement status, amounts, timestamps, refunds).
• Automate safe reconciliations, surface high-severity issues, preserve immutable audit trail, and monitor regressions.

High-level design

• Daily and near-real-time reconciliation pipelines:
  • Batch job (nightly) compares full-day aggregates and per-transaction state.
  • Streaming listener (webhooks / payment events) reconciles within minutes.

Detection methods

• Strong canonical model: Order {id, amount_cents, currency, payment_id, status, updated_at}.
• Pull provider records via paginated API and webhook events; normalize fields.
• Detect: missing payments, amount mismatches, duplicate captures, status drift, partial refunds.
• Use checksums and a reconciliation key (order_id|payment_id|amount) per txn.

Automated reconciliation steps

• For safe fixes: idempotent state transitions (e.g., mark order PAID if provider shows capture and local pending) via a Reconciler service.
• For risky fixes: create compensating transactions (issue refund), or queue for review.
• Implement retry with exponential backoff and dead-letter queue for persistent failures.

Prioritization & escalation

• Severity matrix: financial-impact (amount) × business-impact (order lifecycle). Examples:
  • High: captured amount ≠ order amount by >$100 or missing settlement for shipped order → auto-pause fulfillment, notify on-call, create incident ticket.
  • Medium: small amount drift → auto-create adjustment and send report.
  • Low: metadata mismatch → background fix.
• Escalation: integrate with PagerDuty/Slack; attach immutable audit id and playbook.

Immutable audit logs

• Write append-only events to an audit store (WAL/append-only DB or S3+Parquet with immutability + signed checksums).
• Each reconciliation action writes: before, after, source, reason, operator, signature, timestamp.
• Expose read-only UI for compliance with filtering and export.

Monitoring & regression prevention

• Metrics: reconciliation rate, unmatched count, time-to-reconcile, auto-fix success rate, DLQ size.
• Alerts on trends (e.g., >1% daily mismatch) and regression tests in CI that mock provider API changes.
• Post-incident: blameless postmortem, automated test added (contract tests), and schema/versioning for provider integrations.

Implementation notes (full-stack)

• Backend: Node/Go service for Reconciler, PostgreSQL for canonical model, Kafka for events.
• Frontend: admin dashboard (React) showing metrics, audit trail, manual reconcile UI.
• Security: encrypt PII, strict RBAC for manual fixes, signed audit entries.

Outcome: automated majority-of-cases fixes, fast detection, clear escalation for high-risk issues, and immutable traces for compliance.

Clarify requirements & constraints

• Global low-latency delivery, multi-channel (email, push, SMS), per-channel rate limits, dedupe, backpressure, retries + exponential backoff, DLQ, cost control (avoid SMS overuse), regional regulations.

High-level architecture

• API Gateway → Notification Service (ingest) → Router → Channel Worker Pools (email/push/sms) → Delivery Providers (SES/APNs/Firebase/Twilio)
• Global traffic routed to nearest region; central control plane for policies and metrics.

Service boundaries

• Ingest/Validation Service: auth, schema, dedupe-key generation, policy checks (cost, opt-outs).
• Router/Orchestrator: computes channel list, enforces per-user/channel throttles, places into queues.
• Channel Workers: per-channel executors implementing provider adapters, rate limiting, retries.
• Control Plane: rules, rate limits, cost budgets, analytics.
• DLQ Service: stores failed messages for manual/automated inspection.

Queueing model & backpressure

• Use regional durable message queues (Kafka or SQS FIFO per channel+region) partitioned by customer/tenant to preserve ordering.
• Token-bucket rate limiter per channel implemented at worker-side and by router (global view via Redis/Envoy rate-limit service).
• Backpressure: if queues approach thresholds, Router returns 429 / batches messages into lower-priority cold-queue; apply circuit-breaker to slow producers.

Deduplication & persistence

• Generate deterministic dedupe-id (hash of user+payload+time-window). Store in Redis with TTL (e.g., 24h) to reject duplicates at ingest.
• Persist message metadata in transactional store (Postgres) for audit, retries, cost accounting; store payload in object store (S3) if large.

Retries, exponential backoff & DLQ

• Workers on transient failure: schedule retry with exponential backoff (base * 2^n + jitter) using delayed queue or scheduled job. Track attempts in metadata.
• After max attempts, send to DLQ (Kafka topic or S3) with failure reason; trigger alerting and optional automated remediation.

Cost-control

• Per-tenant budgets in control plane; pre-send checks to block or downgrade channels (use push/email instead of SMS) when budget approaches.
• Sampling, batching (combine multiple notifications into one email/digest), and suppression windows for noisy events.

Failure handling & observability

• Health checks, circuit breakers on provider errors, failover providers for critical channels.
• Metrics (delivery, latency, error rates), tracing (OpenTelemetry), alerts, and dashboards.
• Reconciliation job to compare provider receipts with persisted state and refund/cleanup as needed.

This design balances global scale, per-channel limits, deduplication, robust retry semantics and cost control while keeping clear service boundaries suited for a full‑stack engineer to implement end-to-end.

Overview & goals
Design a shared-docs system that supports low-latency real-time editing, offline edits with guaranteed convergence, multi-region availability, fine-grained permissions, and safe incremental rollout.

High-level architecture

• Clients (web/mobile) run an editing engine and local persistence (IndexedDB).
• Real-time layer: WebSocket/HTTP2 to regional gateway → collaboration service.
• Durable store: regional write-master + cross-region replicated object store (S3/Blob) + metadata DB (CockroachDB/Spanner).
• Background sync workers for cross-region replication and garbage collection.

OT vs CRDT (comparison & choice)

• OT: compact ops, well-known for text editors, requires central transform server and preserved operation history; non-trivial for multi-site offline edits and many concurrent editors.
• CRDTs: strong eventual consistency, peer/edge-friendly, easier offline merges—higher metadata overhead (tombstones, identifiers).
• Choice: Use a hybrid—CRDT (sequence-based, e.g., RGA or Logoot-Undo) on client for offline/resilient merges and conflict-free convergence; use server-side compaction to reduce tombstones.

Storage & sync model

• Client persists local CRDT state + op-log to IndexedDB; sends ops to regional gateway.
• Regional gateway persists ops to durable append-log and applies to regional materialized view.
• Cross-region: async replication of compacted snapshots + op-log; causal vector clocks to ensure ordering.
• Periodic snapshotting and server-side compaction to reduce metadata; clients can bootstrap from nearest region.

Security & permissions

• Auth via OAuth2/OIDC; clients receive short-lived JWTs with doc-scoped claims.
• Permission checks:
  • Client enforces UI-level permission (read/write/comment).
  • Each op signed and authorized at gateway using token and ACL lookup in the metadata DB.
  • Server rejects unauthorized ops; audit logs written immutable.
• Encryption: TLS in transit; at-rest encryption per-region; optional E2EE for private docs with client-managed keys.

Incremental rollout & testing

• Start with single-region, read-only collaborative preview.
• Canary real-time for internal teams, then small % of users.
• Tests: unit (CRDT invariants), integration (multi-client sync), chaos testing (network partitions, clock skew), load testing (ops/sec), correctness fuzzing.
• Monitoring: op lag, conflict rate, replication lag, error rates; automatic rollback on SLA breaches.

Edge cases & trade-offs

• CRDT metadata growth vs. OT server complexity; compaction solves growth at cost of occasional client resync.
• Latency vs. strong ordering: prefer low-latency local edits with causal guarantees.

This approach fits full-stack responsibilities: client CRDT implementation, gateway auth and op persistence, server compaction, and deployment/observability.

Situation & short answer
Eventual consistency is acceptable for non-critical, highly-read features where slight delays don't break user workflows — e.g., global follower counts, like counters, view counters. It lets you scale reads/writes across caches/CDNs/replicated datastores.

User-visible trade-offs

• Inconsistency window: counts may lag or briefly be lower/higher than actual.
• Confusion risk: users might click again or think action failed.
• UX perception: frequent small discrepancies are less harmful than rare large ones.

When stronger guarantees are needed

• Read-your-writes: when the same user expects to immediately see their action (posting, editing, transactional operations).
• Safety-critical flows: billing, permissions, inventory reservations.

Mitigation patterns (practical, full‑stack)

• Client hints: optimistic UI updates (immediately increment a like locally) + mark as pending.
• Refresh: background refresh or short-poll after action to reconcile authoritative count.
• Reconciliation: server-side deltas and idempotent operations; reconcile client view on reconnect.
• Hybrid: strongly consistent write to primary, eventual reads from replicas with short TTLs for critical users (the actor sees primary-read).

From a full‑stack perspective, combine optimistic front-end updates, clear pending state indicators, and short revalidation to balance scalability and UX.