Specialized Topics: Bandits, Transformers, LLMs, Reward Modeling, RLHF Questions

Advanced topics within Machine Learning & AI covering bandit algorithms, transformer architectures, large language models (LLMs), reward modeling, and reinforcement learning from human feedback (RLHF). This topic explores both theoretical concepts and practical considerations for deploying and fine-tuning modern ML systems, including exploration-exploitation strategies, attention mechanisms, scalable training, and human-in-the-loop optimization.

EasyTechnical

60 practiced

What is chain-of-thought prompting? Explain how it encourages LLMs to produce intermediate reasoning steps and give examples of tasks where chain-of-thought improves performance. Also describe downsides (latency, exposing incorrect steps) in production systems.

MediumTechnical

44 practiced

Implement Thompson Sampling for Bernoulli rewards in Python. Function signature: def thompson_sampling(arms: int, trials: int, reward_fn) -> List[int]. Use Beta(1,1) priors and update per-arm alpha/beta counts after each observed 0/1 reward. Return the sequence of arm choices.

Sample Answer

Approach: maintain Beta(α, β) for each arm (start α=1, β=1). At each trial sample θ_i ~ Beta(α_i, β_i) for all arms, pick arm with highest θ, observe binary reward from reward_fn(arm), update α or β, and record chosen arm.

python

from typing import List, Callable
import numpy as np

def thompson_sampling(arms: int, trials: int, reward_fn: Callable[[int], int]) -> List[int]:
    """
    Thompson Sampling for Bernoulli rewards.
    arms: number of arms (indexed 0..arms-1)
    trials: number of pulls to simulate
    reward_fn: function(arm_index) -> 0 or 1 (observed reward)
    Returns: list of chosen arm indices, length == trials
    """
    if arms <= 0 or trials <= 0:
        return []

    alphas = np.ones(arms, dtype=float)  # α_i = successes + 1
    betas = np.ones(arms, dtype=float)   # β_i = failures + 1
    choices: List[int] = []

    for t in range(trials):
        # Sample theta for each arm and pick the argmax
        thetas = np.random.beta(alphas, betas)
        chosen = int(np.argmax(thetas))
        choices.append(chosen)

        r = reward_fn(chosen)
        if r not in (0, 1):
            raise ValueError("reward_fn must return 0 or 1 for Bernoulli rewards")

        # Update posterior
        if r == 1:
            alphas[chosen] += 1
        else:
            betas[chosen] += 1

    return choices

Key points:- Uses Beta(1,1) uniform priors.- Time complexity O(trials * arms) due to sampling per arm each trial.- Space O(arms).Edge cases: validate arms/trials >0, ensure reward_fn returns 0/1. Alternative optimizations: use incremental sampling tricks or vectorized operations for many arms/trials, or maintain approximate sampling for very large arm counts.

EasyTechnical

48 practiced

Provide an intuitive explanation of the Upper Confidence Bound (UCB) algorithm for stochastic bandits. Describe how adding an optimism-based confidence bonus to empirical means encourages exploration and how the bonus shrinks as arms are sampled.

EasyTechnical

81 practiced

Explain the exploration–exploitation trade-off in multi-armed bandits. Give concrete production examples where exploration should be prioritized (e.g., cold-start product recommendations) and where exploitation dominates (e.g., revenue-critical checkout). How would you quantify and instrument the cost of exploration in production (business metrics, user impact, and monitoring)?

Sample Answer

Exploration–exploitation trade-off: you must choose between exploring less-certain actions to learn their value (exploration) and using the best-known action to maximize immediate reward (exploitation). In multi-armed bandits this is formalized by policies (e.g., epsilon-greedy, UCB, Thompson Sampling) that balance sampling uncertainty vs. expected reward; the theoretical cost of exploration is cumulative regret (difference between chosen rewards and the oracle).

When to prioritize exploration (production examples):- Cold-start recommendations: new users or new items need exploration to learn preferences and item quality; short-term loss is acceptable to avoid long-term poor personalization.- Launching a new feature or promotion: you must collect data on performance across segments before committing.- Long-tail content discovery: intentionally surface niche items to improve catalog knowledge and long-term engagement.

When exploitation dominates:- Revenue-critical checkout flows: any exploration that risks conversion or upsell hurts immediate revenue and user trust.- Safety-critical decisions (fraud detection, medical triage): choose proven policies; exploration is constrained or offline-only.- High-stakes retention moments (password resets, billing): minimize experimentation.

Quantifying & instrumenting cost of exploration:- Business metrics: track immediate revenue per session/user, conversion rate, average order value, churn/retention, lifetime value (LTV) delta vs. baseline.- Statistical regret: compute cumulative and per-period regret comparing bandit choices to an exploitation oracle or a well-tested control.- User impact: measure session length, NPS/CSAT, task success, and downstream engagement to detect negative UX effects; compute percent of users exposed to exploratory variants.- Monitoring & deployment controls: - Canary and exposure caps: limit fraction of traffic and per-user frequency for exploratory arms. - Dashboards: real-time KPI panels showing CTR, conversion, revenue, and confidence intervals per arm; cohort breakdown by device/region. - Alerts: threshold-based alerts on metric drops, high regret, or uplift reversals. - Logging: record arm assignments, context features, rewards, and timestamps to enable offline reanalysis and counterfactual evaluation (IPW, bootstrap). - Safety nets: auto-rollback to control when exploration causes statistically significant regressions; persist preferred arm once confidence exceeds threshold.- Evaluation: run offline simulations, regret bounds, and backtests; use A/B holdouts to validate long-term LTV impact.

Putting it together: pick a bandit algorithm aligned to business tolerance for short-term loss (Thompson Sampling for efficient exploration, UCB for principled optimism), limit exposure, monitor key business and user metrics in real time, and quantify exploration cost via regret and revenue/retention deltas so product teams can decide acceptable trade-offs.

EasyTechnical

54 practiced

Implement an epsilon-greedy multi-armed bandit policy in Python. Function signature: def epsilon_greedy(arms: int, trials: int, epsilon: float, reward_fn) -> List[int]. reward_fn(arm) returns a numeric reward. Maintain counts and estimated values; at each trial pick a random arm with probability epsilon otherwise pick argmax estimate. Return chosen-arm sequence.

Sample Answer

Approach: keep arrays for counts and estimated values (sample means). For each trial, with probability epsilon choose a random arm (explore), otherwise pick arm with highest estimate (exploit). After observing reward from reward_fn(arm) update the sample mean incrementally: new_mean = old_mean + (reward - old_mean) / count.

python

import random
from typing import List, Callable

def epsilon_greedy(arms: int, trials: int, epsilon: float, reward_fn: Callable[[int], float]) -> List[int]:
    """
    Run an epsilon-greedy multi-armed bandit policy.
    Returns the list of chosen arms (length == trials).
    """
    if arms <= 0 or trials <= 0:
        return []
    epsilon = max(0.0, min(1.0, epsilon))  # clamp epsilon to [0,1]

    counts = [0] * arms                 # times each arm was played
    estimates = [0.0] * arms            # estimated mean reward for each arm
    chosen = []

    for t in range(trials):
        # Decide explore vs exploit
        if random.random() < epsilon:
            arm = random.randrange(arms)  # explore
        else:
            # exploit: break ties by choosing the first max (deterministic)
            arm = max(range(arms), key=lambda i: estimates[i])

        reward = reward_fn(arm)
        counts[arm] += 1
        # Incremental update of sample mean
        n = counts[arm]
        estimates[arm] += (reward - estimates[arm]) / n

        chosen.append(arm)

    return chosen

Key points:- Uses incremental mean update (numerically stable, O(1) per update).- Time complexity O(trials * 1) and space O(arms).Edge cases:- Epsilon clamped to [0,1], handles arms/trials <= 0.Alternatives:- Use optimistic initial values to encourage exploration, or UCB/Thompson Sampling for better regret bounds.

Unlock Full Question Bank

Get access to hundreds of Specialized Topics: Bandits, Transformers, LLMs, Reward Modeling, RLHF interview questions and detailed answers.

Join thousands of developers preparing for their dream job.