Retrieval Augmented Generation and Knowledge Integration Questions
Understand RAG systems: retrieval components, ranking strategies, and integration with LLMs. Discuss how RAG grounds outputs in external knowledge, reducing hallucinations. Address indexing strategies, retrieval latency, and quality trade-offs. For Staff-level, discuss designing RAG systems that scale to massive knowledge bases with minimal latency.
HardSystem Design
31 practiced
Design a distributed ANN index for 1B vectors with P95 query latency <100ms. Discuss sharding strategy, partition key selection, memory footprint per node, network IO patterns, and how you'd handle hot spots (popular vectors).
Sample Answer
Requirements & constraints:- 1 billion vectors, target P95 < 100ms for k-NN queries.- Assume vector dim = 768, typical for text embeddings. Don't store raw float32 per vector (3.07 TB) — must use compression + hierarchical index.High-level approach- Use a two-level hybrid: coarse-grained inverted file (IVF) to shard/query routing + local compressed PQ (Product Quantization) / OPQ + lightweight HNSW graph inside each shard for refinement. This balances recall, memory, and latency.Sharding strategy & partition-key- Shard by coarse-quantizer id (IVF centroids). Train C coarse centroids (e.g., C = 100k). Assign each vector to its nearest centroid -> centroid_id is partition key. This localizes relevant vectors and reduces cross-shard fanout.- Physical shards = S nodes. Map centroid_ids to nodes with a consistent hashing / range assignment (e.g., 100k centroids -> 1k centroids per node for S=100). Shard count chosen so each node can hold ~target memory footprint and CPU.- For multi-tenancy / failure, use consistent hash ring so shard ownership can move without massive reindexing.Query flow & network IO patterns1. Query pre-processing: compute query vector, OPQ rotation if used.2. Coarse search (on a dedicated lightweight routing layer or replicated centroid store): find top-m candidate centroids (m = 16–64). This is cheap (C ~100k centroids in RAM).3. Fan-out: send query to nodes owning those m centroids. Each node performs local PQ distance scans limited to vectors in selected inverted lists (or uses local HNSW to get local candidates).4. Each node returns top-k_local results (ids + compressed codes or re-ranked distances). Optionally return full vector for final exact re-rank if needed.5. Aggregator merges results and optionally asks 1–10 nodes for raw vectors for final exact L2/cosine on the CPU/GPU.Network IO estimates- Per-node response size small: return top 50 ids + 4–8 byte distance (50 * ~16 bytes = 800B). Main stateful IO is query request (query vector ~768 floats compressed by OPQ or sent as float32 = 3KB). Fanout m controls total network cost: for m=32, 32 * 3KB = 96KB outbound; responses ~32 * 1KB = 32KB inbound. With 100 QPS that’s ~12.8MB/s — well within 100Gbps NICs on many nodes, but account for peaks.Memory footprint per node (example sizing)- Use OPQ + PQ: 64-dimensional PQ with 8 subquantizers × 8-bit codes = 8 bytes per vector (common config gives good tradeoff). Additional per-vector metadata: 8 byte id + 2–4 bytes list pointer ≈ total 18–20 bytes/vector.- 1B vectors * 8 bytes = 8 GB (codes) + ids/metadata ~10–12 GB -> ~20 GB raw. Inverted list structures, centroid table, PQ codebooks, per-shard HNSW graph overhead add ~10–30 GB depending on the local index density.- Choose S such that per-node RAM including OS/serving overhead ≤ node RAM (e.g., 64–256 GB). Example: if S=100 nodes, each stores 10M vectors -> ~200 MB? Wait: 10M*20B = 200MB? Correctly: 10M * 20B = 200MB (check: 10M*20 = 200M bytes = 0.2GB). That seems small; more realistic: with PQ=8B: 10M*8B = 80MB codes; plus overhead maybe 2–4GB lists due to fragmentation/hnsw. Practical industry deployments use ~32–128 GB per node. So pick S=200 to keep per-node working set in 32–64GB RAM. Provide formula: per-node RAM ≈ N_total/S * bytes_per_vector + index_overhead (~10–30%).Latency considerations (getting P95 <100ms)- Keep fanout m small (16–64), minimize number of RPC hops, and parallelize local scans on multiple CPU cores or GPU.- Co-locate centroid routing to avoid extra network hop: keep centroid table replicated in frontends.- Use asynchronous RPC; timeout per node ~40–60ms to meet merge within 100ms.Handling hot spots (popular vectors / centroids)- Read replication: replicate inverted lists for hot centroids to multiple nodes and set up a small router that routes requests to least-loaded replica. Use dynamic replication factor based on request rate.- Query caching: cache top results for very frequent queries (exact or approximate) at the routing layer or using a fast LRU.- Load-aware routing: track per-centroid QPS and do consistent rebalancing (migrate centroid ranges) when thresholds exceeded.- Rate limiting + backpressure: protect background reindexing and writes from overwhelming serving.- Pre-warm: keep hot lists memory-resident, prefetch PQ codebooks and HNSW neighbor lists into cache.- Horizontal autoscaling: spin additional nodes, attach replicas automatically, use consistent hashing to minimize movement.Fault tolerance & consistency- Keep centroid->shard mapping in a config store; changes use rolling reassign with dual-writing during migration.- Use replication factor R (e.g., R=3) for inverted lists to survive node failures; asynchronous replication acceptable due to read-heavy nature.- Periodic background compaction/defragmentation to keep inverted lists efficient.Trade-offs & alternatives- HNSW global graph: faster single-node queries but memory-heavy (per-vector neighbor lists). Use hybrid: HNSW inside each shard for candidate refinement.- Flat brute-force on GPUs: feasible with heavy hardware (GPUs) but more costly; good for very low-latency SLA and small-scale shards.- More aggressive compression (4-bit PQ) reduces RAM but increases CPU for decode and can reduce recall.Summary (numbers to present in interview)- Shard by coarse centroids (IVF), S ~100–300 nodes depending on commodity RAM (target per-node 32–128GB).- PQ codes ~8 bytes/vector -> base storage ~8GB per 1B vectors; with metadata/index ~40–100GB total cluster overhead depending on S.- Fanout m=16–64 keeps network IO modest; per-query network ~<200KB typical.- Hotspot handling: dynamic replication of hot centroids, caching, load-aware routing, autoscaling.- Aim: local candidate generation + small fanout and efficient merge to meet P95 <100ms.
HardTechnical
30 practiced
An attacker crafts inputs to probe your RAG system and extract sensitive training documents via cleverly-constructed queries. How would you detect and mitigate extraction attacks and design logging/alerting to identify potential data leaks?
Sample Answer
Situation: A RAG (retrieval-augmented generation) system is exposed to users and an attacker crafts probes to extract sensitive training documents.Detection:- Canary documents: seed the index with unique, low-risk honeytokens (phrases, emails) that should never be returned to real users; any retrieval triggers immediate alert.- Query-pattern detection: monitor for “extraction” patterns — long sequences of progressive prefix/suffix prompts, repeated prompt-chaining, or systematic probing (e.g., "tell me everything about X", iterative n-gram enumeration).- Response-content analysis: compute overlap metrics (n-gram / longest common substring / cosine similarity) between responses and known sensitive-doc fingerprints. Flag high-overlap responses.- Statistical anomalies: track per-user entropy, token output length, repeated long outputs, and sudden bursts of similarity to private docs.- Behavioral baselines: model normal query distributions and use anomaly detection (isolation forest, autoencoder) to detect outliers.Mitigation:- Access controls & least privilege: limit which indexes / document collections are queryable per role.- Output filtering & redaction: apply sensitive-entity detectors (PII classifiers, regex rules) post-generation; block or redact matches.- Response-length and rate limiting: enforce per-session token caps, per-user query rate caps, and exponential backoff on suspicious behavior.- Retrieval hardening: return document snippets with provenance and context windows; avoid returning raw paragraphs. Apply chunking and overlap minimization so single retrievals don't give whole docs.- Canary-triggered throttling: on a canary hit, immediately throttle the session and require human review.- Differential privacy / private embeddings: train or fine-tune with DP and use embedding hashing/salt so extraction is harder.- Sanitize training data: remove/obfuscate extremely sensitive content ahead of ingestion.Logging & Alerting Design:- Logs to capture per-request: user_id (or session), IP, timestamp, prompt text, retrieval IDs & scores, returned snippets, model response, token counts, redaction hits, similarity scores to known sensitive docs, canary hits.- Enrich logs with contextual features: rolling query frequency, cumulative tokens returned, anomaly score, geolocation, auth level.- Alert rules (example thresholds): - Immediate HIGH alert: canary retrieval OR response similarity > 95% to sensitive doc. - MEDIUM alert: >3 high-length outputs (>1k tokens) within 1 hr from same user OR anomaly score above 0.99. - LOW alert: repeated enumeration-style queries (patterns matched >5 times).- Integrations & workflows: forward alerts to SIEM (Splunk/ELK), create automated incident tickets, and trigger session quarantine + human review playbook.- Forensics & replayability: store full immutable request/response traces and retrieval provenance to reproduce and assess leakage. Retain embeddings and retrieval logs for X days per policy.- Privacy-preserving monitoring: mask PII in logs, store sensitive artifacts under stricter access controls, and audit access to forensics.Example detection rule:- If (canary_hit == true) OR (similarity(sensitive_doc, response) > 0.9 AND token_overlap > 50) -> set severity=HIGH, throttle session, notify SOC, create ticket, preserve trace.Outcome & trade-offs:- These controls balance availability and security: aggressive filtering/rate-limiting reduces usability; canary + anomaly detection gives fast, high-precision alerts while redaction and provenance reduce risk. I would iterate thresholds using simulated attacks and record false-positive rates, tuning for acceptable business risk.
MediumTechnical
31 practiced
Explain approximate nearest neighbor (ANN) algorithm parameters that affect recall and latency. For HNSW, discuss efConstruction, M, and efSearch and how tuning them affects indexing time, memory, and query quality.
Sample Answer
Approximate nearest neighbor (ANN) performance is a trade-off between recall (accuracy of retrieved neighbors) and latency (query time); key parameters control graph density, search breadth, and index work at build-time vs query-time.General ANN parameters that affect recall vs latency:- Index structure density (more edges/trees → higher recall, more memory, slower build)- Search breadth/beam size (larger → higher recall, higher latency)- Quantization/compression level (more compression → lower memory, possible recall loss, faster I/O)- Parallelism and batch sizes (affect throughput/latency but not recall)HNSW-specific parameters:- M (max connections per node): controls graph degree. Higher M → denser graph, better connectivity → higher recall and faster queries (fewer hops) but increases memory per element and significantly increases index construction time. Typical M: 8–64; use higher M when recall/latency at query-time matter and memory/build-time budgets allow.- efConstruction (size of candidate list while inserting): larger efConstruction → more thorough neighborhood selection during build → higher final index quality (better recall) and slower indexing. It also increases temporary memory during build. Reasonable values: M*2..M*10; set higher if building once and want best quality.- efSearch (runtime expansion factor): controls how many candidates are explored per query. Increasing efSearch almost always raises recall but linearly increases query latency and CPU work. You can tune efSearch per query SLA: start near M and increase until marginal recall gain is small.Tuning strategy:- If build-time/resource-limited: prefer moderate M, higher efSearch at query-time.- If queries are latency-sensitive and index build can be expensive: increase M and efConstruction so efSearch can be small.- Measure recall vs qps/latency curve to pick knee point; consider mixed strategy (smaller index + adaptive efSearch per query).
MediumTechnical
52 practiced
For training a dense retriever, describe how in-batch negatives work and why they are efficient. Also explain pitfalls (e.g., false negatives) and mitigation strategies such as hard-negative mining and multi-stage training.
Sample Answer
In-batch negatives: When training a dense retriever (dual-encoder) we compute embeddings for a batch of query-document pairs (q_i, d_i). For each positive pair we treat the other documents d_j (j ≠ i) in the same batch as negatives. Using a softmax/InfoNCE loss (or contrastive loss) over the batch, the model learns to score the true document higher than in-batch negatives. This is efficient because the same forward pass yields O(batch_size^2) pairwise comparisons without extra encoder calls, fully utilizing GPU compute and memory for large effective negative counts.Why efficient:- No extra encoding of negatives — reuse batch embeddings.- Scales linearly in batch size for encoding cost but quadratically for comparisons.- Great signal early in training versus small explicit negative pools.Pitfalls:- False negatives: some in-batch negatives may actually be relevant to the query → noisy gradients and slower convergence.- Limited hardness: randomly sampled in-batch negatives might be too easy, reducing training signal.- Batch-size dependency: small batches reduce negative diversity.Mitigations:- Hard-negative mining: add curated hard negatives (e.g., from BM25 or previous model) alongside in-batch ones; this increases difficulty and improves discrimination.- Multi-stage training: start with in-batch negatives for broad learning (fast, stable), then fine-tune with hard negatives and longer contexts or cross-encoder re-ranking to refine hard distinctions.- Momentum or memory bank (e.g., MoCo) / cross-batch negatives: maintain a queue of recent embeddings to enlarge negative pool without extra encodes.- Loss & temperature tuning: lower temperature sharpens contrast but can amplify false-negative harm; combine contrastive loss with bootstrap or label smoothing.- Symmetric vs asymmetric training: training both query→doc and doc→query (or using hard negatives on both sides) improves robustness.Practical recipe:1. Pretrain/fine-tune with large batch in-batch negatives (InfoNCE).2. Add mined hard negatives (BM25 / cached ANN) and continue training with smaller learning rate.3. Optionally re-rank/further refine with cross-encoder supervision or distillation to fix remaining false-positive/negative errors.
MediumTechnical
28 practiced
Explain index freshness and document versioning in a RAG pipeline. For a knowledge base updated hourly, outline a strategy to incorporate updates with minimal disruption to queries and to support point-in-time retrieval.
Sample Answer
Index freshness = how recently the index reflects the source knowledge; document versioning = tracking immutable document ids + version identifiers so you can relate an index entry to a specific source state. In RAG pipelines these two concepts drive correctness (don’t retrieve stale facts) and reproducibility (answering “what did the KB say at time T?”).Strategy for an hourly-updated KB with minimal disruption and point-in-time retrieval:1. Source & metadata design- Assign each source document a stable doc_id and monotonically increasing version_id or timestamp on every update.- Store provenance metadata (source_uri, version_id, created_at).2. Nearline ingestion pipeline- Compute diffs each hour (added, changed, deleted). Only re-embed/patch affected documents rather than reindexing whole DB.- Use idempotent upserts keyed by (doc_id, version_id) into the vector store; keep old versions as archived entries (do not hard-delete).3. Atomic publish / read routing- Build indexes as write-once segments or snapshots (e.g., segment_vN). After ingest finishes, atomically switch a pointer/alias from snapshot_vN-1 → snapshot_vN. Queries use the alias so switch is instant and disruption-free.- While new snapshot builds, serve reads from previous snapshot.4. Point-in-time retrieval- Retain historical snapshots or keep versioned vectors with created_at. Query API accepts optional as_of timestamp; retrieval filters vectors where created_at <= as_of and, if necessary, picks the latest version per doc_id <= as_of.- For storage efficiency, compact old versions periodically into immutable archives.5. Consistency & validation- Run small canary queries against new snapshot before switching; validate embedding drift and top-k stability.- Maintain TTLs and retention policies for legal/space requirements.6. Practical notes- For large scale, use delta embedding workers, vector stores supporting aliases (Milvus/Pinecone/Weaviate), and metadata-index (searchable SQL/Elasticsearch) to resolve version filters quickly.- Ensure re-ranker and answer synthesis reference the provenance metadata so generated responses include “as-of” citations.This approach minimizes query downtime (atomic alias swap), reduces compute by patching deltas, and enables reproducible point-in-time retrieval by retaining versioned entries or snapshots.
Unlock Full Question Bank
Get access to hundreds of Retrieval Augmented Generation and Knowledge Integration interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.