Performance Cost Optimization & Resource Efficiency Questions
Optimizing for the money and resources a given level of performance consumes, not just raw speed. Covers cost-per-request reasoning, right-sizing compute and memory, efficiency of resource utilization, and trading performance against spend. Emphasizes treating cost and resource efficiency as first-class performance objectives.
HardSystem Design
74 practiced
You are designing a multi-tenant service with significant per-tenant traffic variance. Describe a right-sizing and billing-aware capacity strategy that minimizes wasted resources while protecting SLOs for large tenants. Include isolation, throttling, and burst strategies.
Sample Answer
Requirements & constraints: preserve per-tenant SLOs for large/paid tenants, minimize idle reserved capacity, allow bursts, and make billing reflect capacity guarantees vs best-effort.High-level strategy:1) Reservation tiers + shared pool- Offer explicit "reserved" capacity plans (SLA-backed) for large tenants: X RPS / bandwidth. Reserve that capacity on cluster nodes (soft reservation via scheduler + admission control).- Maintain a global shared pool for best-effort tenants that autos-scales more aggressively.2) Isolation- Logical isolation: per-tenant namespace/queue and rate-controller (token-bucket) per tenant.- Physical isolation for top-tier tenants: colocate reserved tokens on dedicated instances or node-pools to guarantee CPU/memory/network slices.- Use cgroups/kubernetes resource quotas and Pod Topology to enforce.3) Throttling & burst model- Token bucket per tenant: steady-rate = reserved rate (if purchased) + baseline free rate. Burst capacity = accumulated tokens up to a burst window.- Burst beyond reserved uses shared pool tokens; tracked and billed as overage or deprioritized during contention.- Admission control: when overall cluster load high, drop or delay best-effort bursts; prioritized queueing for reserved tenants.4) Billing-aware enforcement- Meter three dimensions: reserved usage, burst-from-shared, dropped/rejected. Bill reserved as recurring, burst as metered at premium, rejected requests not billed.- Charge for committed minimum (even if unused) but offer short-term credits or autoscaling discounts to reduce perceived waste.5) Autoscaling & right-sizing- Autoscale node-pools for shared pool based on aggregated demand signal and conservatively for reserved pools based on committed capacity plus statistical headroom (e.g., p95+ kσ).- Use predictive scaling per tenant using historical patterns; convert frequent bursty tenants into higher-tier reservations with automated upsell.6) SLO protection & fairness- Implement priority-based scheduler + weighted fair queuing across tenants when contention occurs.- Fast failure modes: graceful throttling, 429 with Retry-After, client-side SDKs with exponential backoff.- Circuit breakers per tenant to prevent noisy neighbor from overwhelming system.Operational & metrics:- Track per-tenant p50/p95 latency, error rate, reserved vs burst vs denied counts, cost per request.- Alerts on capacity saturation, abnormal burst patterns, and billing anomalies.- Quarterly right-sizing review: recommend reservation changes based on usage and cost trends.Trade-offs:- Reserved capacity wastes money but guarantees SLOs; statistical multiplexing reduces waste but raises risk of SLO interference—mitigate via prioritization and physical isolation for highest tiers.- Billing complexity increases; keep pricing simple with clear tiers + metered bursts to incentivize right-sizing.Example: Tenant A buys 100 RPS reserved. Token-bucket steady=100 RPS, burst bucket=300 tokens (allow 3s burst). If sustained >100, first use burst tokens (metered as overage). If shared pool saturated, Tenant A is served from its reserved node-pool to preserve SLO; Tenant B (best-effort) is throttled to fair share.
HardSystem Design
101 practiced
You need to reduce cost by moving cold data from primary disk to cheaper archival storage but still allow occasional reads with reasonable latency (seconds). Design a retrieval path that minimizes cost while keeping implementation complexity moderate. Include caching, prefetch, and user experience considerations.
Sample Answer
Requirements & constraints:- Move cold data off primary (expensive) disk to archival storage (cheap, e.g., S3 Glacier or on-prem tape-like tiers).- Occasional reads allowed with latency target: seconds (not minutes).- Minimize cost and keep implementation complexity moderate.- Support read-through caching, prefetch, and good UX (progress/soft-fail).High-level design:Primary disk holds hot data + metadata index. Cold objects are stored in archival tier (low cost). Add a retrieval service (Retriever) and a caching layer (warm cache) on moderately-priced storage (e.g., S3 Standard / EBS gp3 or a managed Redis/ElastiCache for small objects). Use lifecycle policies to migrate objects and keep metadata mapping (object ID → location & last-accessed).Components & responsibilities:- Metadata DB: lightweight key-value store (DynamoDB/Postgres) storing location, checksum, size, lastAccess, prefetch hints.- Retriever service: accepts read requests, checks metadata, routes to cache or archive; orchestrates restore when needed.- Warm cache: medium-cost storage with TTL and LRU; serves reads in seconds.- Prefetcher: background worker that uses access patterns (recent reads, user-session, object bundles) to proactively restore likely-needed objects into warm cache.- Archive backend: archival storage supporting bulk/expedite restores (e.g., S3 Glacier Deep Archive with expedited restores or a staged cache for restores).Data flow:1. Client requests object.2. Retriever checks metadata: - If in warm cache: return immediately. - If marked archived: issue expedited restore request to archive and return placeholder response (or block up to a short timeout like 5–10s).3. Retriever polls restore status; once restored, move object into warm cache and return to client.4. Prefetcher uses heuristics (user session, related object graph, sequential access) to start restores opportunistically when object is archived.Caching & prefetch strategies:- Cache sizing based on access skew (80/20). Use small object cache (Redis) for metadata/keys and warm storage for larger objects.- Adaptive TTL: keep recently accessed items longer; evict by size-aware LRU.- Prefetch heuristics: session-based, time-of-day, ML-model scoring optional. Start prefetch when probability > threshold to avoid wasted restore costs.- Batch restores: group related objects into single restore job to amortize archive retrieval overhead.User experience:- For synchronous reads: allow blocking up to a short SLA (e.g., 5–10s). Show progress indicator, estimated time, and option to receive async notification when ready.- For interactive apps: return low-fidelity placeholder quickly (metadata preview, thumbnails) while full object restores.- For bulk reads: provide background “prepare” endpoints to pre-warm objects.Cost/complexity trade-offs:- Use managed archive with expedited restores for seconds-latency but limit use to avoid high cost—prefetch and caching reduce need.- Simpler option: always restore to warm storage on first access (acceptable if accesses are very rare). More complex option: ML prefetch reduces restore cost but adds modeling work.- Keep Retriever stateless; use queues for restore jobs to simplify scaling and retries.Metrics & ops:- Instrument restore-request rate, cache hit ratio, average retrieval latency, restore cost per object.- Alerts on rising restore cost, low cache hit rate.- Periodic tuning: adjust cache size, prefetch thresholds, and lifecycle windows.Edge cases:- Very large objects: stream partial content (range requests) and restore only needed ranges.- Multiple simultaneous requests: coalesce restores to one job and fan-out responses.- Data consistency: validate checksums after restore; use versioning in metadata.This design minimizes storage cost by keeping long-term data in cheap archive while meeting second-level read latency through a warm cache, batching/coalescing restores, and smart prefetching—balancing cost and moderate implementation complexity.
MediumTechnical
77 practiced
Implement (pseudocode is fine) an LRU cache with O(1) get and put operations. Emphasize memory accounting to support an approximate max-bytes capacity rather than item count, and explain eviction behavior when item sizes vary widely. Use any language you prefer.
Sample Answer
Approach: Use a doubly-linked list for recency (head = MRU, tail = LRU) and a hashmap key->node for O(1) lookup. Each node stores key, value, and size_bytes. Maintain current_bytes; capacity_bytes is the approximate max. On put, if new item causes current_bytes>capacity, evict from tail until under capacity (or until only the new item if it itself exceeds capacity — handle per policy).Key points:- get/put are O(1) average: hashmap ops + constant pointer updates.- Memory accounting: each node tracks size; curr is adjusted on add/remove.- Eviction behavior: when sizes vary, a single very large item may evict many small ones. Choose policy: - Accept large item by evicting all others (simple, ensures insertion succeeds). - Reject items larger than cap (prevents cache pollution). - Use admission policies (e.g., only admit if larger than some hit-rate threshold).- Edge cases: concurrent access (use locks), approximate size estimation for complex values, fragmentation when sizes change on update.
python
class Node:
def __init__(self, key, value, size):
self.key = key
self.value = value
self.size = size
self.prev = None
self.next = None
class LRUCacheByBytes:
def __init__(self, capacity_bytes):
self.cap = capacity_bytes
self.curr = 0
self.map = {} # key -> Node
self.head = None # MRU
self.tail = None # LRU
def _remove_node(self, node):
# unlink node
if node.prev: node.prev.next = node.next
else: self.head = node.next
if node.next: node.next.prev = node.prev
else: self.tail = node.prev
node.prev = node.next = None
self.curr -= node.size
del self.map[node.key]
def _add_to_head(self, node):
node.next = self.head
if self.head: self.head.prev = node
self.head = node
if not self.tail: self.tail = node
self.map[node.key] = node
self.curr += node.size
def get(self, key):
node = self.map.get(key)
if not node: return None
# move to head (MRU)
self._remove_node(node) # removal updates curr & map
self._add_to_head(node) # re-add updates curr & map
return node.value
def put(self, key, value, size):
# If exists, remove old instance first
if key in self.map:
self._remove_node(self.map[key])
node = Node(key, value, size)
# Evict until there is room. Policy: allow storing item even if bigger than capacity by evicting all others;
# alternatively, skip insertion if size > cap (choose based on requirements).
while self.curr + size > self.cap and self.tail:
self._remove_node(self.tail)
# If still doesn't fit (item > cap and cache empty), decide: store single large item or reject.
self._add_to_head(node)HardTechnical
77 practiced
You observe significant allocation churn causing GC pressure in a Java microservice. Suggest a prioritized action list (quick wins to deeper changes) including instrumentation, small code changes, JVM tuning, and longer-term refactors. Explain expected outcomes for each step.
Sample Answer
Start with measurement, then iterate from low-risk quick wins to deeper changes. For each step I list expected outcomes.1) Instrumentation (immediate)- Add async allocation/GC telemetry: enable -Xlog:gc*, use GC ergonomics, JFR (Flight Recorder), async-profiler or YourKit with allocation recording, and heap/opp sampling in production.- Outcome: precise hotspots (types, sites, allocation rates), baseline metrics for impact validation.2) Short-term code fixes (minutes–hours)- Eliminate obvious temporary objects: reuse StringBuilder, avoid repeated boxing/unboxing, prefer primitive arrays or IntStream alternatives.- Replace frequent toString/format calls with precomputed templates.- Outcome: immediate drop in allocation rate; measurable GC pause reduction.3) Fast JVM tweaks (hours)- Right-size heap and GC algorithm: try G1 with tuned pause targets or ZGC/Epsilon depending on latency needs; increase young-gen to reduce promotion churn; tune survivor ratios and MaxGCPauseMillis.- Outcome: fewer full GCs, reduced pause times while you fix code.4) Targeted profiling + micro-optimizations (days)- Use allocation flame graphs to find hot call sites; optimize hot-path data structures (use primitive collections, off-heap buffers, object reuse only when safe).- Outcome: meaningful allocation rate reduction without large design changes.5) Architectural refactors (weeks–months)- Introduce streaming, batching, or immutable flyweight patterns; redesign high-frequency paths to use pooled buffers or thread-local caches; offload work to native or async pipelines.- Outcome: qualitatively lower sustained allocation, improved throughput and tail-latency.6) Verification & guardrails (ongoing)- Add regression tests, JFR snapshots in CI, allocation budgets, and alerts on allocation/G1 pause thresholds.- Outcome: prevent regressions and maintain SLOs.Trade-offs: avoid premature global object pooling (memory leaks, complexity). Prioritize measurement, then low-risk fixes, then JVM tuning, then deeper refactor.
EasyTechnical
77 practiced
List the main cloud cost drivers for a typical microservice: compute, storage, network egress, managed databases, and monitoring. For each driver, give one common optimization that reduces cost without drastically harming performance.
Sample Answer
Compute — Right-size and use autoscaling groups with burstable or spot instances where appropriate.Why it saves: reduces idle CPU costs by matching capacity to traffic; spot instances cut compute price substantially.Trade-off: potential preemption or slower scaling; keep critical tasks on stable instances and use spot for stateless workers.Storage — Use lifecycle policies and choose tiered storage (e.g., S3 Standard → Infrequent/Archive).Why: moves cold data to cheaper tiers automatically, lowering monthly storage bills.Trade-off: retrieval latency/costs for archived data; ensure frequently accessed objects stay in hot tier.Network egress — Cache responses at CDN/edge and compress payloads.Why: reduces repeated origin traffic and bytes transferred, cutting egress charges.Trade-off: cache invalidation complexity and slightly stale content; tune TTLs for correctness.Managed databases — Use right-sized instances, read replicas, and offload heavy read/analytic queries to replicas or read-optimized stores.Why: reduces expensive primary DB scaling and leverages cheaper read instances or serverless DB options.Trade-off: added replication lag and operational complexity; ensure consistency needs are met.Monitoring — Sample or aggregate high-cardinality metrics and use log retention tiers.Why: reduces ingestion, storage, and query costs by sending only necessary telemetry and keeping long-term logs in cold storage.Trade-off: less granular historical data for debugging; keep detailed telemetry for critical components only.For each optimization, measure cost and performance impact with experiments and alert thresholds so you don’t degrade user experience while saving money.
Unlock Full Question Bank
Get access to hundreds of Performance Cost Optimization & Resource Efficiency interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.