Recommendation and Ranking Systems Questions

Designing recommendation and ranking systems and personalization architectures covers algorithms, end to end system architecture, evaluation, and operational concerns for producing ranked item lists that meet business and user objectives. Core algorithmic approaches include collaborative filtering, content based filtering, hybrid methods, session based and sequence models, representation learning and embedding based retrieval, and learning to rank models such as gradient boosted trees and deep neural networks. At scale, common architectures use a two stage pipeline of candidate retrieval followed by a ranking stage, supported by approximate nearest neighbor indexes for retrieval and low latency model serving for ranking. Key engineering topics include feature engineering and feature freshness, offline batch pipelines and online incremental updates, feature stores, model training and deployment, caching and latency optimizations, throughput and cost trade offs, and monitoring and model governance. Evaluation spans offline metrics such as precision at k, recall at k, normalized discounted cumulative gain, calibration and bias checks, plus online metrics such as engagement, click through rate, conversion and revenue and longer term retention. Important product and research trade offs include accuracy versus diversity and novelty, fairness and bias mitigation, popularity bias and freshness, cold start for new users and items, exploration and exploitation strategies, multi objective optimization and business constraint balancing. Operational considerations for senior level roles include scaling to millions of users and items, experiment design and split testing, addressing feedback loops and data leakage, interpretability and explainability, privacy and data minimization, and aligning recommendation objectives to business goals.

EasyTechnical

58 practiced

In Python, implement an item-based recommendation helper: given a user-item interaction matrix as a dense 2D NumPy array (users x items), compute cosine similarity between items and return top-k recommended item IDs for a given user based on items they have interacted with. Signature: def recommend_item_based(interactions, user_id, k=10). Assume binary interactions (0/1). Explain complexity and possible scalability issues.

Sample Answer

We can compute item-item cosine similarities using the item columns of the user-item matrix, then score unseen items for a user by summing similarities to items the user has interacted with. Here's a clear, vectorized implementation using NumPy:

python

import numpy as np

def recommend_item_based(interactions: np.ndarray, user_id: int, k: int = 10):
    """
    interactions: 2D array shape (n_users, n_items), binary 0/1
    user_id: index of user to recommend for
    k: number of item IDs to return (indices)
    Returns: list of item indices recommended (length <= k)
    """
    # Validate
    n_users, n_items = interactions.shape
    assert 0 <= user_id < n_users

    # Items as column vectors (n_items, n_users) transpose to (n_items, n_users)
    items = interactions.T.astype(float)  # shape (n_items, n_users)

    # Compute norms and avoid division by zero
    norms = np.linalg.norm(items, axis=1)
    zero_mask = norms == 0
    norms[zero_mask] = 1.0

    # Cosine similarity matrix: items @ items.T / (norms[:,None]*norms[None,:])
    sim = items @ items.T
    sim = sim / (norms[:, None] * norms[None, :])

    # For items with zero interactions, set similarities to 0
    sim[zero_mask, :] = 0
    sim[:, zero_mask] = 0

    user_items = interactions[user_id]  # binary vector length n_items
    interacted_idx = np.where(user_items > 0)[0]
    if interacted_idx.size == 0:
        return []  # no history -> can't score

    # Score each item by sum of similarities to items the user interacted with
    scores = sim[:, interacted_idx].sum(axis=1)

    # Do not recommend already-interacted items
    scores[interacted_idx] = -np.inf

    # Get top-k indices
    top_k = np.argsort(-scores)[:k]
    # Filter out -inf results (e.g., when few candidates)
    return [int(i) for i in top_k if scores[i] > -np.inf]

Key points:- Vectorized NumPy calculation avoids Python loops.- Time complexity: O(n_items^2 * n_users) for building the full item-item similarity matrix (since items @ items.T costs n_items^2 * n_users in arithmetic). Memory: O(n_items^2) to store sim.- Per-user recommendation after sim is O(n_items * n_interacted).- Scalability issues: computing/storing full similarity matrix is infeasible when n_items is large (e.g., millions). Remedies: - Use sparse representations (scipy.sparse) and sparse cosine computation. - Compute top-N nearest neighbors per item (approximate NN, locality-sensitive hashing, FAISS) and store only sparse similarities. - Precompute similarities offline, shard by item, or compute on-the-fly using only user's interacted items.Edge cases handled: users with no interactions, items with zero vectors, ties.

MediumTechnical

83 practiced

Describe a practical architecture to provide feature freshness for a ranking model that needs last-click and last-10-minutes features at 100 ms latency. Include choices between streaming (Kafka/Flink), online feature store stores, caching, and fallback behavior when online features are missing.

Sample Answer

Requirements:- Serve ranking model requests with 100 ms p99 latency.- Freshness: last-click (most recent event) and aggregates for last 10 minutes.- High availability, graceful fallback when online features missing.

High-level architecture:1. Event ingestion (user interactions) → Kafka (partitioned by user_id) for durability and ordering.2. Stream processing → Flink (or Kafka Streams) computes rolling state: - Last-click: store last event timestamp and metadata. - Last-10-minutes: sliding-window aggregate (counts, sums) updated in real time. Flink writes incremental updates to an online feature store / low-latency store (Redis / DynamoDB / Cassandra with DAX) keyed by user_id.3. Online feature store API: - Highly-available KV store with <5ms read. Supports upserts from Flink and gets from inference layer.4. Serving layer (inference): - On request, read features from online store. To achieve 100 ms p99, use: - Local LRU cache per inference pod (size tuned, TTL ~30s) to avoid repeated network hops. - Batch GETs for multi-user or multi-features.5. Fallback behavior: - If online feature missing: fallback to computed cold features from offline store (feature-engineering batch features in data warehouse) or model-defaults (population averages). - If freshness-critical features missing (e.g., last-click): mark request and use conservative ranking or soft-penalize confidence; emit telemetry for misses.6. Observability: - Metrics: feature-latency, miss-rate, staleness distribution. - Alerts on sink lag, Kafka consumer lag, or high miss-rate.

Trade-offs:- Redis = lowest latency; DynamoDB scales but slightly higher tail latency.- Flink offers strong window semantics; Kafka Streams is lighter weight.- Caching lowers latency but increases staleness—tune TTL and use cache invalidation on updates for hot keys.

Edge cases:- Partition skew: use hot-key strategies (replication, consistent hashing).- Flink failures: enable checkpointing and write-ahead logs to avoid lost updates.This design balances freshness, latency, and reliability while providing safe fallbacks and observability.

HardTechnical

82 practiced

Propose a cost-optimization plan for serving large embedding tables for ranking: include strategies like quantization, pruning, caching hot rows, asynchronous fetch, model distillation, autoscaling policies, and cost-vs-latency trade-offs. Provide concrete metrics you would monitor to evaluate cost savings impact.

Sample Answer

Approach: balance memory/compute costs vs. latency and ranking quality by applying a layered strategy: reduce table size (quantization/pruning/distillation), reduce access cost (caching/hot rows/asynchronous fetch), and right-size infra (autoscaling + async pipelines). Measure cost vs. quality/latency trade-offs with specific metrics.

Plan:1. Quantization- Use 8-bit or 4-bit quantization (e.g., QAT for minimal accuracy loss). Expect ~2–4x memory reduction at <1% NDCG loss for many embeddings.- Validate with A/B: track delta NDCG@10 and recall@k.

2. Pruning & sparse/dense hybrid- Identify low-usage or low-contribution dimensions via L1/L2 importance or sensitivity; remove or sparsify them.- Target 10–30% dimension pruning with retrain/fine-tune to keep quality.

3. Model distillation- Train a smaller embedding model (student) to mimic teacher. Reduce embedding size (e.g., 512→128) to cut storage and compute by ~4x; measure end-to-end ranking loss.

4. Caching hot rows- LRU or LFU cache in RAM/SHM for top X% of lookups (e.g., top 1% rows produce 50–80% of hits). Store quantized but fast-to-dequantize representations.- Metric: cache hit rate, p50/p95 fetch latency, reduced egress/storage ops.

5. Asynchronous fetch & batching- Pre-fetch embeddings for predicted candidates; non-blocking background loads for cold rows; batch RPCs to storage (coalescing).- Metric: tail latency reduction, queue lengths, miss penalty.

6. Autoscaling policies- Scale based on combined signals: request QPS, cache miss rate, and CPU/GPU utilization. Use predictive autoscaling (time-series + recent traffic) to avoid cold starts.- Policy example: maintain p95 latency < 50ms; scale out when miss rate > 5% and CPU>70%.

Trade-offs & validation:- For each knob (quant bits, embedding size, cache size), run gated experiments and plot cost per QPS vs. NDCG loss and p95 latency.- Expected targets: reduce storage/serving cost 2–6x with <1–2% relative ranking degradation and p95 latency increase <10ms.

Concrete metrics to monitor:- Cost metrics: $/M requests, storage $/GB-month, compute $/hour- Quality: NDCG@10, recall@k, model calibration drift- Performance: p50/p95/p99 fetch & end-to-end latency, cache hit rate, cold-miss penalty- Utilization/operational: CPU/GPU utilization, memory usage, RPC latency, queue lengths- Autoscaling signals: scale events, time-to-scale, scale failures- Business impact: CTR, conversion lift from A/B tests

Implementation cadence:- Phase 1: measure baseline and top-k hot rows; add caching + async fetch.- Phase 2: apply 8-bit quantization with QAT; monitor quality.- Phase 3: pruning/distillation and predictive autoscaling.- Run continuous A/B and rollback thresholds tied to NDCG/CTR and latency SLOs.

MediumTechnical

75 practiced

Implement the scoring part of a two-tower retrieval model training step in PyTorch: given user embeddings U (batch_size x d), positive item embeddings V_pos (batch_size x d), and a set of negative item embeddings V_neg (batch_size x n_neg x d), compute dot-product scores and a softmax-based cross-entropy loss that treats each positive as the correct item among (1 + n_neg) candidates. Signature: def two_tower_loss(U, V_pos, V_neg) -> loss scalar.

Sample Answer

Approach: compute dot-product logits between each user and the (positive + negatives) items per row, then apply cross-entropy where the positive is the correct class (index 0 after concat). This is equivalent to an InfoNCE / softmax negative-sampling loss.

python

import torch
import torch.nn.functional as F

def two_tower_loss(U: torch.Tensor, V_pos: torch.Tensor, V_neg: torch.Tensor, temperature: float = 1.0) -> torch.Tensor:
    """
    U: (B, d)         user embeddings
    V_pos: (B, d)     positive item embeddings (aligned per user)
    V_neg: (B, n, d)  negative item embeddings per user
    temperature: scaling factor for logits

    Returns:
      scalar loss (torch.Tensor)
    """
    B, d = U.shape
    assert V_pos.shape == (B, d)
    assert V_neg.dim() == 3 and V_neg.shape[0] == B and V_neg.shape[2] == d

    # Positive logits: (B, 1)
    pos_logits = torch.sum(U * V_pos, dim=1, keepdim=True)  # dot product per row

    # Negative logits: (B, n)
    # U -> (B, 1, d); bmm with V_neg.transpose -> (B, 1, n) -> squeeze -> (B, n)
    neg_logits = torch.bmm(V_neg, U.unsqueeze(2)).squeeze(2)  # (B, n)

    # Concatenate so index 0 is positive candidate
    logits = torch.cat([pos_logits, neg_logits], dim=1)  # (B, 1 + n)

    # Optional scaling
    logits = logits / float(temperature)

    # Targets: positive is class 0
    targets = torch.zeros(B, dtype=torch.long, device=U.device)

    # Cross-entropy: applies log-softmax internally
    loss = F.cross_entropy(logits, targets, reduction='mean')
    return loss

Key points:- Using dot-product directly; keep embeddings normalized externally if using cosine similarity.- Complexity: O(B * n * d) time, O(B * n) extra logits memory.- Edge cases: if n == 0, logits shape (B,1) and CE with targets=0 works (loss=0 if pos large); ensure gradients flow.- Alternatives: in-batch negatives (treat other batch items as negatives), sampled softmax or noise-contrastive estimation for large item vocabularies.

EasyTechnical

85 practiced

Explain pointwise, pairwise, and listwise learning-to-rank approaches. For each, describe typical loss functions, their assumptions, and which production scenarios they are best suited for (e.g., CTR prediction vs. search ranking). Mention computational considerations during training and inference.

Unlock Full Question Bank

Get access to hundreds of Recommendation and Ranking Systems interview questions and detailed answers.

Join thousands of developers preparing for their dream job.