Large language models and in Context learning Questions
Understand how large language models work: scaling laws, emergent abilities, and in-context learning. Discuss prompt engineering, few-shot learning, chain-of-thought prompting. Understand token probabilities and sampling strategies (temperature, top-p).
HardTechnical
33 practiced
Compare 8-bit post-training quantization, 4-bit asymmetric quantization, and weight-sharing quantization for transformer weights. Discuss expected impacts on GPU memory footprint, inference latency, numeric stability (e.g., softmax sensitivity), and sampling behavior (rare-token probability shifts).
Sample Answer
High-level summary:- 8-bit post-training quantization (PTQ): map 32-bit float weights to 8-bit integers, typically per-tensor or per-channel scales; simple, widely supported.- 4-bit asymmetric quantization: lower-bit (4-bit) quant with independent zero-point and scale (asymmetric), often per-channel and may use block-wise/grouped scales.- Weight-sharing quantization: cluster weight values to K centroids (e.g., k-means), store indices + codebook — extreme compression with non-uniform representation.GPU memory footprint:- 8-bit PTQ reduces weight memory ~4× vs FP32 (plus small scale vectors). Activations still often FP16/FP32, so total reduction depends on activation precision. Straightforward memory alignment and fast GPU loads.- 4-bit asymmetric ~8× reduction vs FP32 for weights but storage packing (nibbles) and added zero-point/scale per group introduce overhead; effective ~6–7× in practice.- Weight-sharing achieves highest compression (depends on K). With 256-centroid codebook and 8-bit indices you get similar to 8-bit; with 16-centroid and 4-bit indices you can beat 4-bit. But codebook lookup adds extra memory for centroids and indirection buffers.Inference latency:- 8-bit PTQ: minimal latency overhead; many GPUs/ libraries (cuBLASLt, TensorRT) support int8 GEMM, so throughput often increases.- 4-bit asymmetric: requires custom kernels for unpacking/scale application; can be faster than int8 if optimized (reduced memory bandwidth) but often less mature — latency wins only with specialized kernels or hardware.- Weight-sharing: adds indirection and codebook lookup per weight — GEMM becomes lookup+accumulate which is typically slower unless transformed into efficient matrix-matrix with pre-expanded tiles. Latency often worse unless heavily optimized.Numeric stability (softmax sensitivity, accumulation noise):- 8-bit PTQ: preserves numeric distribution reasonably well; softmax is sensitive to small logit shifts, but int8 noise is usually small if calibration per-channel is used. Accumulation uses higher-precision (int32) in GEMM, mitigating catastrophic drift.- 4-bit asymmetric: greater quantization noise; asymmetric scales reduce bias around zero but coarse resolution increases rounding errors. Softmax tail probabilities can shift more, especially when logits differ by small margins.- Weight-sharing: introduces structured, non-uniform error (cluster bias). This can systematically shift logits (not just random noise), causing consistent skew in softmax and possible collapse or inflation of certain token probabilities, especially for rare tokens whose logits rely on fine-grained weight values.Sampling behavior and rare-token probability shifts:- 8-bit PTQ: small, mostly random perturbations — rare-token probabilities may change slightly but relative ordering usually preserved; empirical calibration often keeps sampling stable.- 4-bit asymmetric: higher probability mass reallocation — rare tokens (low-probability tails) are most affected; you may observe increased repetition or pruning of low-probability candidates.- Weight-sharing: can cause pronounced shifts due to centroid bias: some regions of weight space map to same centroid, making many logits coarser and tilting probability mass toward more represented centroids — rare tokens can be heavily suppressed or spuriously boosted depending on clustering bias.Practical mitigations:- Per-channel/per-block scaling and bias correction reduce 8-bit/4-bit errors.- Fine-tuning (quant-aware or small calibration set) repairs softmax-sensitive layers (output projection, final LayerNorm).- Keeping LayerNorm and Embeddings at higher precision (FP16/FP32) preserves numeric stability and sampling fidelity.- For weight-sharing, use residual quantization or secondary fine-grained codebooks for critical layers.Recommendation:- Use 8-bit PTQ first for best latency vs accuracy trade-off. Move to 4-bit asymmetric with optimized kernels if memory pressure demands it and you can afford calibration/fine-tuning. Use weight-sharing only when extreme compression is required and you can retrain/fine-tune and invest in custom kernels to avoid sampling collapse.
MediumSystem Design
30 practiced
Design a prompt templating system for a multi-tenant LLM service that must safely insert: (1) system instructions, (2) retrieved documents, and (3) tenant-specific user data. Show a sample template and describe an escaping strategy, token-budget enforcement for a 16k model, and per-tenant isolation controls.
Sample Answer
Requirements:- Compose final prompt from: system instructions (platform-level), retrieved docs (variable size), tenant-specific user data (PII/metadata).- Multi-tenant safety: no cross-tenant leakage, enforce sanitization and token budgets for a 16k model (16,384 tokens).- Low latency, auditability, and defense against injection.High-level architecture:API Gateway → Prompt Templating Service → Token Budget Enforcer → Escaper/Sanitizer → LLM Adapter → Audit & Tenant Policy StoreTemplate (example):{ "system": "{{SYSTEM_INSTRUCTIONS}}", "context": "===DOCUMENTS_START===\n{{#each RETRIEVED_DOCS}}\n---DOC {{@index}}---\n{{escape_doc this}}\n{{/each}}\n===DOCUMENTS_END===", "user": "User: {{escape_user(TENANT_USER)}}\nTask: {{USER_PROMPT}}"}Escaping strategy:- Two-level escaping: 1. Structural escaping: replace template delimiter collisions (e.g., "{{", "}}", "===...===") in inserted text with safe tokens. 2. LLM-safe sanitization: remove control sequences (e.g., "System:", "Assistant:"), normalize whitespace, strip embedded prompt markers or instruction-like lines. Use deterministic canonicalization so audit logs can reverse-check what was removed.- Implement escape functions server-side (escape_doc, escape_user) — never trust client-side.Token-budget enforcement for 16k model:- Fixed allocations: e.g., reserve 1,024 tokens for system, 12,000 for documents+user, 2,360 for model response (tunable).- Pre-tokenize each part using the model tokenizer (byte-pair) before composing.- If RETRIEVED_DOCS exceed allocated document budget, apply: - Re-ranking + summarization (client or server-side) to compress content. - Aggressive truncation with "TRUNCATED" markers and provenance links.- Reject requests that cannot be safely compressed and return structured error with guidance.Per-tenant isolation controls:- Per-tenant policy store: allowed escape rules, max token budgets, redaction rules.- Namespacing: all temp files, caches, embeddings keyed by tenant ID; strong IAM and encryption-at-rest.- Runtime controls: per-tenant rate limits, concurrent request caps, and quarantine mode for suspicious activity.- Logging & audit: immutable, tenant-scoped logs for prompt inputs/outputs and escape actions; redact PII in logs per policy.- Access control: RBAC for tenant admins; strict service-to-service mutual TLS and signed requests.Trade-offs:- Pre-tokenization and summarization adds CPU cost but prevents OOM/overcharge.- Conservative sanitization may remove useful context; provide opt-in relaxations with stricter tenant-level review.This design ensures safe insertion, enforces token budgets for a 16k model, and maintains strong tenant isolation while remaining auditable and extensible.
HardTechnical
30 practiced
You advise a startup deciding between deploying a large off-the-shelf LLM versus a smaller distilled model augmented with RAG. Prepare a decision memo that compares accuracy, latency, cost per request for 1M monthly requests, engineering effort, and compliance/risk. Recommend a phased plan (pilot, metrics, MVP) with milestones and KPIs.
Sample Answer
Executive summaryYou must choose between (A) a large off-the-shelf LLM (hosted API) and (B) a smaller distilled model + RAG (self-hosted or managed infra + vector DB). Below I compare them on the requested axes, give rough cost/latency estimates for 1M monthly requests, note engineering/compliance trade-offs, and recommend a phased pilot → MVP plan with milestones and KPIs.Comparison (high-level)- Accuracy / factuality - Large OOTB LLM: Strong generalization, fewer prompt hacks needed, better open-domain reasoning. Higher risk of plausible hallucinations for domain-specific facts unless you provide context. - Distilled + RAG: Potentially higher factual accuracy on domain knowledge if retrieval quality is good; distilled model may be weaker at open reasoning unless tuned.- Latency (median / p95) - Large OOTB LLM (API): median 200–800 ms; p95 600–1500 ms (depends on model, network). - Distilled + RAG: inference median 50–200 ms + retrieval 30–150 ms → combined median 100–300 ms; p95 300–800 ms.- Cost per 1M monthly requests (example numbers; adjust to vendor/pricing) - Large API: assume $0.15–$0.50 per request => $150k–$500k / month. - Distilled + RAG: model hosting + vCPU/GPU + vector DB ops + storage: estimate $0.01–$0.03 per request => $10k–$30k / month (plus obs/maintenance).- Engineering effort - Large API: Low infra effort; effort focused on prompts, safety wrappers, billing, rate limits, caching. Quick to pilot (days–weeks). - Distilled + RAG: Higher up-front: model selection/distillation/fine-tuning, building ingestion pipelines, vector DB,Retriever, indexing, embeddings, monitoring. Ongoing ops (model updates, infra). 2–4x more engineering time than API.- Compliance / risk - Large API: Higher vendor/data exposure; may violate data residency or sensitive-data policies; check DPA and ability to opt-out of training. Simpler to secure at app-level but limited control. - Distilled + RAG: Easier to control data flow; can store vectors and docs in your VPC; better for strict compliance (HIPAA, finance) but requires your SOC/processes.Decision guidance- If time-to-market and lowest engineering lift matter and the domain is broad/general, prefer Large API.- If domain requires provable factuality, strict compliance, or cost sensitivity at scale → Distilled + RAG.Phased plan (Pilot → Metrics → MVP)Pilot (4–8 weeks)- Goal: validate accuracy, latency, cost assumptions on representative workload (10k–50k requests).- Tasks: - Set up two parallel experiments: 1) API path: integrate large LLM, design prompts, add caching, logging. 2) RAG path: select distilled model (e.g., Llama2-13B distill or faster), embedding model, vector DB (Pinecone/Weaviate/FAISS), pipeline for ingestion. - Instrument telemetry (latency, cost, errors, hallucination detection). - Run A/B on requests or split by use-case.- Milestones / KPIs: - p50/p95 latency < target (e.g., p95 < 800 ms) - Accuracy (human-evaluated F1 or relevance) difference quantified - Cost per request measured - Hallucination rate under threshold (e.g., <5% on domain queries)MVP (8–16 weeks after pilot)- Goal: production-ready single approach or hybrid.- Option A: API-first MVP - Add caching, request batching, prompt templates, safety filters, logging, rate-limits, CI for prompts. - KPIs: monthly cost ≤ budget; user satisfaction ≥ target; uptime 99.9%; hallucination rate acceptable.- Option B: RAG-first MVP - Harden retrieval (re-ranker, chunking strategy), fine-tune distilled model on domain prompts, deploy autoscaling infra, monitoring/alerting, document sync. - KPIs: cost per request target met; p95 latency target; retrieval precision@k; human-eval accuracy > baseline.- Hybrid approach: Use API for complex reasoning and RAG for factual lookups; route requests by intent classifier. This often yields highest accuracy vs cost trade-off but increases complexity.Operational considerations- Monitoring: telemetry for latency, cost, token usage, retrieval precision, hallucination detectors, user feedback loop.- Rollout: start with low-risk user segment; can A/B larger cohorts.- Security/compliance: data classification, encryption at rest/in transit, retention policies for logs/vectors, vendor contract review (DPA, training opt-out).- Cost controls: caching, batching, response-size limits, early-exit heuristics, tiered routing (cheap model for most queries, expensive model for complex ones).Recommendation (practical for a startup)1) Pilot both in-parallel for 4–8 weeks to gather real metrics and human-eval labels.2) If pilot shows API cost at 1M requests > 3–4× RAG cost and compliance is important, invest in RAG MVP. Otherwise, prefer API-first MVP to minimize time-to-market, then replace heavy-cost paths with RAG/hybrid where cost or factuality is an issue.Quick example KPI targets (for MVP decision)- Latency p95 ≤ 800 ms- Human-eval accuracy gap between chosen approach and gold ≤ 5%- Cost per request ≤ $0.03 (RAG) or budgeted cap for API- Hallucination rate ≤ 5% on domain queries- Production readiness: CI/CD, monitoring, incident playbookIf you want, I can convert this into an actionable Gantt-style milestone list with estimated engineer-weeks and vendor suggestions.
EasyTechnical
25 practiced
Explain how next-token probabilities are computed in transformer-based LLMs. Include discussion of logits, softmax normalization, temperature scaling, and what the resulting token probability distribution implies about model confidence. Describe how you would use token probabilities to detect out-of-distribution (OOD) inputs and surface monitoring alerts in production.
Sample Answer
Transformers compute next-token probabilities in three steps. First the model produces logits: the final hidden state for the next position is projected by a learned linear layer to a vector of raw scores (logits) — one score per vocabulary token. Second, logits are converted to probabilities with the softmax: p_i = exp(logit_i) / sum_j exp(logit_j). Softmax normalizes scores into a probability distribution over tokens. Temperature scaling adjusts confidence: divide logits by temperature T before softmax (logit_i / T). T>1 flattens (more uniform), T<1 sharpens (more peaked), enabling control over sampling diversity.What probabilities imply about confidence: high max probability and low entropy mean the model is confident; a flat distribution (low max, high entropy) indicates uncertainty. Note: raw probabilities can be miscalibrated — a token at 90% may not be correct 90% of the time.Using token probabilities for OOD detection and monitoring:- Per-token signals: monitor max token probability and distribution entropy per generation. Low max-prob or high entropy flags potential OOD or uncertainty.- Sequence-level: aggregate metrics (mean log-probability, min token prob, perplexity) across prompts/responses.- Statistical monitoring: track rolling baselines (mean, variance) and compute KL divergence or PSI between current and training/dev token-prob distributions; significant shifts imply input distribution drift.- Alerting: set thresholds (e.g., fraction of requests with max_prob < 0.2 exceeds X%) or anomaly detection on z-scores; trigger human review or fallback policy.- Improve reliability: calibrate model probs (temperature calibration, isotonic regression) using validation data; combine token-prob signals with other checks (embedding-distance to training set, input classifier) to reduce false positives.Edge considerations: token probs are sensitive to vocabulary and prompt length; use normalized measures (per-token cross-entropy) and tune thresholds per use-case.
MediumTechnical
23 practiced
Compare instruction tuning, full fine-tuning, LoRA (low-rank adapters), and prompt tuning for adapting LLMs to domain-specific customer support. Discuss trade-offs in compute cost, inference latency, storage for multi-tenant models, and ability to quickly A/B different behaviors.
Sample Answer
Brief definitions:- Full fine-tuning: update all model weights on domain data—gives maximum capacity to change behavior.- Instruction tuning: fine-tuning (often full or large-partial) on instruction-response pairs to align model for task formats.- LoRA (low-rank adapters): inject small, trainable low-rank matrices into transformer layers; keep base weights frozen.- Prompt tuning: learn a small continuous prompt (tokens/embeddings) prepended to inputs; base model frozen.Trade-offs (compute cost / training speed):- Full fine-tuning: high compute, long training; needs GPU memory proportional to model size and optimizer states.- Instruction tuning: similar to full if updating whole model; can be lighter if targeted. - LoRA: low compute and memory; trains orders of magnitude fewer params, fast iterations.- Prompt tuning: cheapest to train (tiny parameter count), very fast.Inference latency:- Full fine-tuning / instruction-tuned (weights baked in): no extra runtime overhead.- LoRA: small extra matmuls when applying adapters—negligible to low overhead if implemented well.- Prompt tuning: minimal overhead (longer prompt sequences slightly increase token processing cost).Storage / multi-tenant:- Full fine-tune/instruction-tune: separate full checkpoints per tenant → large storage per tenant (GBs).- LoRA: stores small adapter files (MBs) per tenant; composeable at load time.- Prompt tuning: extremely small per-tenant footprint (KB–MB).Ability to A/B and iterate:- Full fine-tune: slow to produce multiple variants; heavyweight to roll back.- Instruction tuning: similar to full unless you keep modular instruction datasets.- LoRA: excellent for A/B—swap adapters quickly, run few-shot or adapter ensembles.- Prompt tuning: excellent for rapid A/B and experimentation; instant swaps.Recommendation for customer support multi-tenant:- Use a frozen base LLM + LoRA adapters per tenant for balance: low training cost, small storage, near-native latency, easy A/B.- Use prompt tuning for ultra-fast experiments or low-resource tenants.- Reserve full/instruction tuning for platform-wide behavioral shifts where changes must be baked into base model.
Unlock Full Question Bank
Get access to hundreds of Large language models and in Context learning interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.