Natural Language Processing (NLP) and Generative AI topics, including language modeling, transformer architectures, large language models, text generation, prompt engineering, evaluation, and deployment considerations for AI-powered systems.
MediumTechnical
45 practiced
Explain approaches for privacy-preserving ML in NLP inference: client-side inference, model distillation to smaller models that run locally, secure enclaves (TEEs), homomorphic encryption, and differential privacy during fine-tuning. For each approach discuss latency, accuracy, and practicality trade-offs for a customer data sensitive application.
Sample Answer
For a customer-data-sensitive NLP inference use case, here are practical approaches with latency, accuracy, and practicality trade-offs.Client-side inference (on-device):- Latency: Very low (no network round-trip), good for real-time UX.- Accuracy: Limited by model size and device compute; state-of-the-art models may not fit.- Practicality: High privacy (data never leaves device). Requires on-device optimized models, frameworks (TensorFlow Lite, ONNX), and heterogeneous hardware support; deployment complexity across OS versions.Model distillation to smaller local models:- Latency: Low; runs locally with small runtime.- Accuracy: Some degradation vs. teacher model but often acceptable if distilled carefully (task-specific fine-tuning, data augmentation).- Practicality: Good balance—reduces memory/compute while preserving privacy. Needs a private or anonymized dataset for distillation to avoid leaking sensitive examples.Trusted Execution Environments (TEEs / secure enclaves):- Latency: Moderate — added overhead for context switches and encrypted I/O.- Accuracy: Full model fidelity (server-grade models can run).- Practicality: Strong confidentiality for model and data; requires compatible hardware (Intel SGX, AMD SEV), attestation, and careful SDK integration. Scalability and cloud availability can be limiting.Homomorphic encryption (HE):- Latency: Very high — computationally expensive, especially for deep NN layers; often impractical for low-latency inference today.- Accuracy: Can retain model behavior but limited to operations supported efficiently; numeric techniques may reduce precision.- Practicality: Low for production real-time NLP; feasible for small models or offline/batched queries where privacy budget and latency permits. Integration complexity and tooling immaturity are concerns.Differential privacy (DP) during fine-tuning:- Latency: No runtime impact on inference.- Accuracy: Adds noise during training/finetuning; may reduce utility, especially with small sensitive datasets. Tuning clipping/noise is critical.- Practicality: Strong protection against membership inference in training data. Useful combined with other methods (e.g., distillation + DP). Requires expertise to set epsilon and measure trade-offs.Recommendation checklist:- If ultra-low latency and strong on-device privacy needed -> distillation + client-side inference.- If full model accuracy but strict server-side confidentiality -> TEEs.- For mathematical guarantees on training data leakage -> DP during fine-tuning (possibly combined with distillation).- Use HE selectively for high-sensitivity, low-throughput tasks or auditing.Consider hybrid strategies (distilled on-device models with DP finetuning, or TEEs for heavy models) to balance latency, accuracy, and operational practicality.
MediumTechnical
41 practiced
You must decide between two third-party LLMs for a knowledge assistant: a faster, cheaper model with slightly lower factual accuracy vs a slower, costlier model with better factuality. Discuss how you'd evaluate and choose (benchmarks, user experience metrics, latency SLOs, cost-per-query, fallback strategies) and how you might combine both to meet product goals.
Sample Answer
Clarify requirements first:- Define primary success criteria (e.g., factual accuracy 95% for critical answers, 99th‑percentile latency ≤600ms for instant UI, cost target $0.01/query).- Identify user personas and failure tolerance (support agent vs end-user-facing assistant).Evaluation plan / benchmarks:- Create a domain-specific factuality benchmark: curated Q/A set (1000 items) + adversarial hallucination probes and ambiguous queries. Measure exact-match, BLEU, and factuality metrics (FEVER score or LLM-specific fact-checker).- Latency benchmark: measure P50/P95/P99 end-to-end latencies including network + tokenization + model. Run under realistic concurrency to capture tail behavior.- Cost-per-query: compute average tokens * price, and total TCO at expected traffic (monthly).- UX metrics: task success rate, time-to-first-satisfactory-response, user-reported trust score in pilots.- Reliability: availability, error rates, and variance under load.Decision criteria and SLOs:- Set SLOs: factuality >= X% (depends on product risk), P95 latency <= target, cost per query <= budget. Weight these criteria (e.g., factuality 50%, latency 30%, cost 20%) and score models.- Run A/B tests with production traffic samples, measure business KPIs (conversion, support deflection).Fallback and hybrid strategies:- Use faster/cheaper model as default for low-risk interactions (chit-chat, navigation, short summaries).- Route high-risk or high-value queries (legal, medical, billing, low-confidence answers) to the higher-factuality model.- Confidence gating: use a lightweight verifier (cross-check via retrieval, secondary model, or rules) — when confidence below threshold, escalate to the slower model or return “I’m not sure” plus source citations.- Response caching for repeated queries to reduce cost and p95 tail.- Parallel speculative calls: fire both models in parallel, return faster answer if verified by a cheap verifier; else prefer factual model—costlier but reduces latency for many queries.- Dynamic routing: use query classifier (intent + risk + length) to choose model.Monitoring and continuous improvement:- Instrument feedback loop: user ratings, automated fact-checkers, and periodic human audits. Retrain classifier thresholds and update routing policies.- Track cost attribution, drift in factuality, and latency regressions; implement automated rollback.Trade-offs and final recommendation:- If product is user-facing with legal/financial stakes, prefer the higher-factuality model with optimized routing to control cost.- If latency and cost are primary with acceptable lower factuality, choose faster model but add retrieval+verifier and selective escalation.- Hybrid approach (default fast + selective escalation + caching + verification) usually offers best balance of cost, latency, and factuality while enabling iterative improvements.
EasyTechnical
33 practiced
Describe automatic evaluation metrics used for generated text (BLEU, ROUGE, METEOR, BERTScore). Explain what each focuses on, common pitfalls when applying them to open-ended generation (dialogue, creative writing), and when you must use human evaluation instead.
Sample Answer
Automatic metrics summarize overlap or similarity between generated text and references quickly, but each has different focus and limitations.BLEU- Focus: n-gram precision (typically up to 4-grams) with brevity penalty; used for MT.- Strengths: fast, language-agnostic, correlates with quality on tightly constrained tasks.- Pitfalls: rewards surface overlap, ignores paraphrase and meaning; poor for single references and open-ended outputs.ROUGE- Focus: recall-oriented n-gram overlap (ROUGE-N), longest common subsequence (ROUGE-L); popular for summarization.- Strengths: emphasizes coverage of reference content.- Pitfalls: like BLEU, sensitive to wording and reference count; higher recall but still misses semantic equivalence.METEOR- Focus: weighted harmonic mean of precision/recall with stemming, synonyms, and paraphrase matching.- Strengths: better correlation with human judgments than BLEU for some tasks.- Pitfalls: depends on synonym resources and language-specific tools; still limited for creative variation.BERTScore- Focus: contextual embedding similarity between tokens (precision/recall/F1) using pretrained transformers.- Strengths: captures semantic similarity and paraphrase better than surface metrics.- Pitfalls: can be misled by topical overlap, biased by encoder’s pretraining data, and sensitive to sentence length.Common pitfalls for open-ended generation (dialogue, creative writing)- Low correlation with human preference when many valid outputs exist- Penalizing novelty, style, or pragmatic appropriateness- Sensitivity to a single reference; score inflation with many references- Gaming by maximizing n-gram overlap without true qualityWhen to use human evaluation- Use humans if outputs require coherence, factual correctness, style, creativity, or pragmatic appropriateness (e.g., dialogue helpfulness, factuality in summarization, humor).- Human evaluation is necessary for final product decisions, qualitative error analysis, and measuring user preference or safety.Best practice- Combine automatic metrics (for fast iteration) with targeted human evaluations (for final assessment); pick metrics aligned with the specific quality dimension you care about (semantic faithfulness vs. surface match).
MediumTechnical
35 practiced
You must choose a vector database for a production semantic search service that will store 1M embeddings (dim 768), require 95th-percentile latency <50ms, support near-real-time inserts/deletes, and run on cloud-managed service with multi-region replication. Compare FAISS (self-hosted), Milvus, and a managed service like Pinecone. For each, discuss scalability, operational burden, index types, and recovery strategies.
Sample Answer
Situation: You need a production semantic-search vector DB for 1M embeddings (dim=768), p95 < 50ms, near‑real‑time (NRT) inserts/deletes, cloud-managed with multi‑region replication. Here’s a practical comparison of FAISS (self‑hosted), Milvus, and a managed service (e.g., Pinecone) across scalability, ops burden, index types, and recovery.FAISS (self-hosted)- Scalability: Excellent single‑node performance (CPU/GPU). For 1M@768, IVF+PQ on CPU or HNSW/IVF+PQ on GPU can meet p95 <50ms; multi‑node sharding requires custom orchestration.- Operational burden: High — you must deploy, autoscale, manage GPUs, networking, monitoring, and multi‑region replication yourself.- Index types: HNSW, IVF, PQ, OPQ (via Faiss). Very flexible and tunable for latency/recall tradeoffs.- Recovery strategies: Build automated index persistence, periodic snapshots of raw vectors + metadata, and scripted rebuilds; maintain warm standby nodes with preloaded indexes to meet RTO.Milvus (open-source, cloud options)- Scalability: Designed for distributed scale-out with dedicated components (proxy, index, datacoord); easier horizontal scaling for 1M and beyond. Can meet p95 <50ms when tuned and with GPU nodes.- Operational burden: Moderate — less than raw FAISS but requires cluster ops (Helm/K8s, storage, monitoring). Managed Milvus (Zilliz Cloud) reduces burden.- Index types: HNSW, IVF+PQ, ANNOY-like options; supports dynamic inserts/deletes and hybrid search (vector + scalar).- Recovery strategies: Built-in metadata persistence, incremental snapshots, and automatic segment recovery; still need backup policies and cross‑region replication setup.Managed (Pinecone or similar)- Scalability: Transparent autoscaling and multi‑region replication; 1M vectors trivial for service.- Operational burden: Low — provider handles nodes, replication, security, and upgrades. SLA and support available.- Index types: Exposes optimized indexes (HNSW-like, PQ internals abstracted). Less control over low‑level tuning but provides simple latency/recall knobs.- Recovery strategies: Provider-managed backups, point‑in‑time recovery, and cross‑region replication; you rely on SLA and export APIs for offline backups.Recommendation (AI Engineer view)- If you need lowest OPEX and fast time‑to‑prod with multi‑region replication: choose a managed service (Pinecone/Zilliz Cloud). It meets p95 requirements with minimal ops.- If you require maximum control, custom optimizations (novel quantization or custom GPU kernels), and can staff ops: self‑hosted FAISS with orchestration.- If you want an open‑source, distributed compromise: Milvus (self‑managed or managed) gives good scalability and NRT support with moderate ops.Operational checklist regardless of choice:- Load test p95 with real embeddings and queries, tune index params (ef/search_k, nprobe, PQ bits).- Implement ingest pipeline with backpressure, retention, and consistency model.- Automate snapshot + export of raw vectors/metadata and test restore drills across regions.
MediumTechnical
42 practiced
Describe strategies for incremental updates of a large ANN (approximate nearest neighbor) vector index to support frequent inserts and deletes while keeping query latency low. Discuss trade-offs between background reindexing, multiple index generations, write buffers, and hybrid exact+approx search approaches.
Sample Answer
High-level approach: treat updates as a write-heavy streaming problem layered on a read-optimized ANN structure. Combine short-term mutable storage with periodic consolidation into immutable, highly-optimized ANN shards so queries remain low-latency while inserts/deletes are supported.Strategies and components:- Write buffer (memory or small HNSW/IVF index): accept frequent inserts/deletes, indexed immediately for low-latency queries. Keep it small so queries against it are fast.- Multiple index generations (tiered indexes): maintain immutable large read-optimized index (Gen0, Gen1...). New writes go to buffer/GenN; background jobs merge smaller generations into larger ones (log-structured merge pattern).- Background reindexing/merge: run async compaction that rebuilds merged ANN with deleted items pruned. Use incremental merges to avoid full rebuilds. Schedule during low-load or use low-priority compute.- Hybrid exact+approx search: at query time, probe immutable ANN(s) for fast candidates, then re-rank or verify with an exact search on the write buffer (or a vector DB supporting exact dot-product). This keeps recall high for recent items and low latency for stable data.- Tombstones + lazy deletion: mark deletes in metadata, physically remove during merges to avoid costly in-place deletes.Trade-offs:- Write buffer size vs. query latency: larger buffers reduce merge frequency but increase hybrid verification cost and memory use.- Background reindexing cost vs. freshness: aggressive reindexing gives clean indexes faster (fewer false candidates) but consumes CPU/GPU and may momentarily affect latency if not isolated.- Multiple generations vs. lookup complexity: many generations increase query effort (must probe each), mitigated by compaction policy (size ratios) and bloom filters/timestamps to skip irrelevant shards.- Hybrid exact+approx accuracy vs. complexity: exact verification raises per-query cost but improves precision and handles recent updates reliably.Recommendations:- Use small in-memory HNSW or IVF for writes + immutable optimized FAISS/PQ shard for bulk reads.- Implement LSM-style compaction with size-tiered merges and tombstone pruning.- At query time: parallel probe immutable shards + write buffer; fuse results and re-rank with exact similarity on top-k.- Monitor metrics (query latency tail, recall, merge CPU) and tune buffer sizes, compaction schedule, and generation thresholds accordingly.
Unlock Full Question Bank
Get access to hundreds of NLP & Generative AI interview questions and detailed answers.
