Architectural Patterns and Anti-Patterns Questions

Clarify requirements & constraints

• p95 search latency <100ms globally; updates must be visible within 5s; high QPS; eventual consistency tolerable except 5s SLA; multi-region deployment.

High-level approach

• Serve most queries from edge/CDN + regional app caches; keep authoritative state in a global search index (near-real-time) with a change-propagation pipeline that invalidates/updates caches within 5s.

Indexing

• Use a distributed search engine (Elasticsearch/Opensearch or managed vector/text store) with near-real-time refresh interval ~1s.
• Index write path: listing update → write to primary DB → publish event to Kafka (with listing id + change type) → indexer consumes and updates search index asynchronously (ack on success).

Cache layers

• CDN (edge): cache query responses with short TTL (e.g., 2s) and use cache key = query fingerprint + region. Edge handles most cold reads.
• Regional app cache (distributed, e.g., Redis cluster): store popular query results or ID lists; TTL 5s; fallback to search index.
• Local in-process cache (LRU): per-instance hot-item caching for microseconds speed; TTL 1s.

Invalidation mechanics

• Event-driven invalidation: indexer emits invalidation events to a cache-invalidation topic. Consumers:
  • CDN: use surrogate keys / PURGE by listing-id or query-key via CDN API (or use cache-control + stale-while-revalidate).
  • Regional Redis: delete keys for affected query patterns; use precomputed reverse-index of listing-id → query keys for targeted invalidation; if not available, increment a region-wide version token per listing and let cache check token on read.
• Optimistic update: when listing is updated, write-through to DB and immediately update regional app cache and local cache with new listing; then indexer updates search index—this reduces visible lag.
• TTLs as safety net ensure eventual consistency if targeted invalidation misses.

Handling partial failures / partitions

• If indexer or Kafka delayed: serve stale results up to 5s (slightly extend TTL) and surface a telemetry metric/alert. Provide read-after-write guarantee for the user who updated: write-through updates local & regional caches so they see change instantly.
• If CDN purge fails: fallback to edge stale-while-revalidate and serve cached result while async refresh occurs; mark responses with stale indicator for observability.
• Circuit-breakers: if search index unavailable, fall back to app cache or DB-backed ad-hoc search; degrade gracefully and return 503 only when necessary.
• Reconciliation: periodic full-sync (diff by version numbers) and a background repair job to catch missed invalidations.

Operational considerations

• Monitor end-to-end latency, index lag, invalidation queue depth, hit rates, and correctness windows.
• Run chaos tests (partition CDN/API, delayed indexer) and define SLIs/SLOs and runbooks.
• Trade-offs: targeted invalidation complexity vs TTL simplicity. As EM, prioritize visibility (metrics, alarms), automation for runbooks, and incremental rollout to validate behavior.

Clarify goals & constraints

• SLO: p95 read latency < 50ms globally for listings read API.
• Requirements: read-dominant, occasional writes (creates/updates), acceptable eventual consistency with good UX.

High-level strategy

• Deploy multi-region read replicas in every major user region (CDN edge for static assets + regional DB replicas for listings).
• Use geo-DNS / Anycast + regional API gateways to route reads to nearest replica; writes route to a single writable primary or a small set of regional primaries depending on conflict model.

Replication & lag mitigation

• Use asynchronous geo-replication with fast WAN (private links) and change-data-capture (CDC) pipeline to apply updates quickly.
• Monitor replication lag metrics per region; if lag > threshold, fall back to stronger sources or serve cached versions with stale-while-revalidate.
• Consider quorum-aware write (semi-synchronous for nearby replicas) for critical updates.

Consistency & UX techniques

• Provide session-aware "read-your-writes" using client session tokens that route subsequent reads to the write’s primary or to a replica that has applied the write.
• Offer tunable consistency endpoints: strong-consistency path (higher latency) for critical flows, eventually-consistent path (low latency) for browsing.
• Use lightweight conflict resolution (last-write-wins with versioning) or application-level reconciliation for complex merges.
• Use read fallback patterns: if regional replica is behind, return cached listing with “recent updates may be delayed” or fetch from primary and accept higher latency.

Operational & team considerations

• Define SLOs, alerts for p95 latency and replication lag; run game days for failover scenarios.
• Cross-functional plan: infra team handles network/private links, data team owns replication pipelines, app teams implement session tokens and consistency modes.
• Rollout gradually with canary regions and measure user metrics (latency, stale reads, error rates) and iterate.

Approach / framework
As an EM I’d treat this as both an engineering and operational problem: short-term mitigation to protect bookings + medium-term fixes for reliability. I’d coordinate engineering, SRE, product and support, set metrics and a runbook.

Immediate operational steps

• Throttle traffic to gateway (rate-limit per-second/global) to stop cascading failures.
• Enable circuit breaker around gateway calls with short error window and exponential backoff.
• Route new payments to alternate gateway if available (failover).
• Communicate status to product/support and show a clear UX message.

Resilience patterns & trade-offs

• Circuit breaker
  • Pro: prevents repeated failing calls, allows recovery.
  • Con: may block legitimate requests if thresholds too low; needs careful tuning.
• Retries with jitter + idempotency keys
  • Pro: recovers transient errors.
  • Con: increases load if misconfigured; use capped retries and jitter.
• Backpressure / queueing
  • Pro: smooths bursts; preserves system stability.
  • Con: increases latency; requires capacity planning and persistence.
• Fallback flows (deferred/guest checkout, save-and-charge-later, alternate gateway)
  • Pro: preserves conversion and user trust.
  • Con: adds business complexity, more edge cases and reconciliation work.

Implementation & ops

• Metrics: success rate, latency, queue length, breaker state; alerting + dashboard.
• Runbook: when to open breaker, failover checklist, SLAs for manual intervention.
• Post-mortem: root cause, load tests, capacity/perf improvements, SLA with gateway.

Outcome focus
Protect bookings and user experience by failing fast with graceful degradation, minimizing lost conversions while avoiding system-wide instability.

Decision criteria

• Business need: measure read latency, throughput, and user impact (p50/p95, error rates, UI TTI). If reads dominate cost or user experience and writes are much less frequent, denormalize is favored.
• Consistency tolerance: can the product accept eventual consistency? If not, avoid denorm or add compensating logic.
• Complexity & operational cost: estimate engineering effort, storage cost, query simplicity, and testing surface.
• Failure modes: risk of stale/incorrect data, harder updates, undo/compensation complexity.

Migration plan (phased, low-risk)

1. Discovery & design (1–2 sprints): define denormalized model, ownership of fields, invariants, and reconciliation rules.
2. Read-only shadow (2–4 wks): build denormalized store populated from current system; route a small % of reads to compare results without affecting users.
3. Gradual rollout (canary -> 50% -> 100%): route more traffic, monitor correctness, latency, and cost.
4. Write strategy: keep normalized writes as source of truth; implement event-driven updates (CDC or events) to populate denorm store; include idempotent handlers.
5. Reconciliation & alerts: periodic background full/partial reconciliation and automated alerts on divergence thresholds.
6. Cutover & cleanup: switch default reads if metrics meet targets, then retire unused indices and code paths.

Measurement & success criteria

• Performance: p95 read latency reduced by target (e.g., 30–50%), throughput improved, decreased DB CPU/IO.
• Correctness: divergence rate < threshold (e.g., <0.01% mismatches) and MTTR for fixes < defined SLA.
• Cost: total cost (storage + ops) within budgeted delta vs benefits.
• Business impact: improved user metrics (conversion, retention) or backend SLA achievement.
• Operational burden: mean time to detect/reconcile inconsistencies and number of on-call incidents not increased.

If metrics fail, rollback plan: revert read routing to normalized store, pause event producers, and fix reconciliation logic before retry.

Search endpoint — SLO: 99.9% successful responses within 200 ms p95 per 30-day window

• Alert thresholds:
  • Warning: 99.7% over 24h or p95 > 250 ms for 1h
  • Critical: 99.5% over 24h or p95 > 350 ms for 15m
• Runbook actions:
  • Check service health, index/cluster metrics, cache hit-rate, query queue length. Roll back recent search-weight or query planner changes. Increase read replicas / scale search nodes if CPU or IO constrained. Engage search owner if index corruption or long GC.
• Error budget policy:
  • If >50% of monthly error budget spent → freeze non-critical feature releases affecting search; schedule immediate reliability sprint items (caching, autoscaling). If <20% spent, allow one small-risk experiment per week with automatic rollback.

Booking completion flow — SLO: 99.95% end-to-end success (payment confirmed and reservation persisted) within 60s per 30-day window

• Alert thresholds:
  • Warning: 99.9% over 6h or failure spike >0.5% absolute over baseline for 30m
  • Critical: 99.8% over 1h or queue growth/backpressure >2x for 15m
• Runbook actions:
  • Inspect booking service logs, DB write latency, transactional queues. Fail-open vs compensating transactions check. Run manual reconciliation for stuck bookings, enable degraded path (place-hold) while fixing root cause. Roll back recent booking-affecting deploys.
• Error budget policy:
  • Any booking-related outage consuming >25% budget triggers freeze on UX changes and mandates cross-team incident postmortem and remediation backlog.

Payments orchestration service — SLO: 99.99% successful orchestration (request accepted + routed to provider) within 1s and provider confirmation within 30s per 30-day window

• Alert thresholds:
  • Warning: 99.995% over 24h or routing failures >0.1% for 30m
  • Critical: 99.99% over 1h or provider error surge >1% for 15m
• Runbook actions:
  • Verify queue length, circuit-breaker state, provider connectivity, secrets/keys validity. Failover to secondary provider, disable risky routing rules, increase worker capacity. Notify payments compliance and ops.
• Error budget policy:
  • Conservatively protect payment budget: if >10% spent, block all non-security feature releases touching payments; prioritize resilience (retries, idempotency, observability).

How I’d run this as an Engineering Manager:

• Make SLO ownership clear per service, tie alerting into on-call rotations and playbooks, review error-budget burn weekly with product to balance features vs reliability, and require a documented rollback and test plan for any release that would consume >5% of the month’s remaining error budget.