Natural Language Processing Fundamentals Questions

Foundational knowledge of Natural Language Processing including how text is represented and processed, common tasks, model classes, and practical tooling. Core concepts include tokenization and subword segmentation, embedding representations and word vector methods such as Word2Vec and GloVe, attention mechanisms, sequence to sequence models, recurrent neural networks, and transformer based architectures. Common tasks to understand include text classification, sentiment analysis, named entity recognition, and machine translation. Candidates should be able to explain how pretrained transformer models are used conceptually, trade offs between model types, basic training and evaluation approaches for language tasks, and practical experience or familiarity with common libraries and toolkits used in the field.

MediumTechnical

17 practiced

Compare using pretrained GloVe embeddings vs contextual embeddings (BERT) for a small CPU-deployed classification task. Discuss memory and latency implications, adaptability to new domains, fine-tuning overhead, inference complexity, and expected accuracy trade-offs. Recommend a deployment approach given limited compute.

Sample Answer

High-level summary: GloVe (static embeddings) + a lightweight classifier is far cheaper in memory and CPU latency but less context-aware; BERT (contextual) yields better accuracy on complex language and domain shifts but has much higher inference and fine-tuning cost. For a small CPU-deployed classifier with limited compute, lean toward GloVe or a distilled/quantized contextual model depending on accuracy needs.

Comparison by dimension:

- Memory: - GloVe: embedding file sizes vary (e.g., common 300-d 400k-vocab ~450–500 MB in float32) but you can prune vocab to ~10–50 MB for task-specific tokens or load as float16/int8. Embedding + small dense classifier often <50–100 MB. - BERT: BERT-base ~110M params (~420 MB FP32); quantized/int8 ~110–200 MB; DistilBERT ~250 MB FP32, smaller with pruning/quantization.- Latency (CPU): - GloVe + logistic/linear/LightGBM: single sentence inference in single-digit ms to tens of ms depending on preprocessing. - BERT on CPU: typically tens to hundreds of ms (BERT-base ~300–1500 ms per sentence) unless distilled/optimized; batching helps throughput but increases latency.- Adaptability to new domains: - GloVe: static embeddings miss context; adapting requires retraining embeddings on domain corpora or heavy feature engineering. - BERT: contextual representations adapt better to domain language; fine-tuning yields strong domain transfer.- Fine-tuning overhead: - GloVe: train only classifier (fast, cheap), no catastrophic forgetting concerns. - BERT: fine-tuning costs GPU/time and is complex (hyperparameters, larger memory), though adapter modules or feature-based (frozen BERT + classifier) reduce cost.- Inference complexity: - GloVe: O(n) lookup + classifier computation; trivial CPU ops. - BERT: O(L * n * d^2) style transformer ops; heavy matrix multiplications, slower on CPU.- Expected accuracy trade-offs: - BERT > GloVe on contextual, ambiguous, long-range dependencies and when dataset is moderately sized. - For small datasets or simple lexical tasks, GloVe + good feature engineering can match or nearly match BERT.

Concrete deployment recommendations (limited compute):1. If strict latency/memory limits and acceptable accuracy: use pruned GloVe (task vocab only) + lightweight classifier (logistic/LightGBM/ small BiLSTM). This is simplest, cheapest, easiest to maintain.2. If you need better accuracy but have tight CPU: use a distilled and quantized model (DistilBERT/TinyBERT) exported to ONNX or TFLite with int8 quantization and optimized runtime (ONNX Runtime / Intel OpenVINO). Also consider freezing most layers and only using lightweight adapters.3. Hybrid: use GloVe baseline in production; profile errors and selectively route hard examples to an expensive contextual model (if budget allows).

Edge considerations: batch inference to amortize transformer cost, use integer quantization and CPU-optimized BLAS, monitor calibration drift—domain shift may require retraining or adapter tuning.

MediumTechnical

19 practiced

For an on-device sentiment classifier with strict latency and memory constraints, compare deploying a small LSTM-based model vs a distilled transformer (DistilBERT). Discuss parameter counts, quantization friendliness, runtime parallelism, expected accuracy trade-offs, and developer effort for optimization on CPU-only devices.

Sample Answer

Situation: You need an on-device sentiment classifier with strict latency and memory limits on CPU-only devices. Below is a practical comparison between a small LSTM and a distilled transformer (DistilBERT) across the requested axes.

Parameter counts- Small LSTM: typical tiny models (1–2 layers × 128–256 units) → ~0.5–3M parameters.- DistilBERT: ~6 layers, 768 hidden → ~66M parameters (fine-tuned head adds a few thousand). Even smaller distilled variants or TinyBERT can be 10–15M but usually still larger than compact LSTMs.

Quantization friendliness- LSTM: RNN ops (matmul + recurrent) quantize well for weights; per-channel int8 works. However, dynamic ranges in activations (gates) require careful calibration; support varies across runtimes.- DistilBERT: Transformer matmuls and layernorm quantize well; attention-heavy structure benefits from static quantization and is often supported by optimized backends (XNNPACK, FBGEMM). Larger memory for embedding tables can be quantized effectively.

Runtime parallelism- LSTM: More sequential due to time-step recurrence — limited parallelism across timesteps; can be batched across tokens but single-sentence latency suffers.- DistilBERT: Highly parallelizable across sequence positions (self-attention), so on multithreaded CPUs it exploits SIMD and multi-core better, yielding lower latency for short sequences if memory permits.

Expected accuracy trade-offs- LSTM: With small capacity, accuracy on nuanced language (negation, sarcasm) is lower; good for simple patterns; may lose 5–15% absolute vs transformer baselines depending on dataset.- DistilBERT: Retains much of BERT’s contextual understanding — better for complex linguistic phenomena, often 3–8% better than small LSTMs; diminishing returns when model is heavily compressed.

Developer effort for CPU optimization- LSTM: Easier to implement and tune; fewer framework dependencies. Optimizations: sequence packing, fused LSTM kernels, int8 quantization, pruning. Lower engineering cost.- DistilBERT: Higher initial effort (tokenizer, large embedding memory, transformer fusion). But benefits from mature tooling (ONNX Runtime, TensorFlow Lite with XNNPACK, Hugging Face Optimum). Optimizations: quantization-aware training, weight pruning, operator fusion, batching, custom kernels—requires more dev time but yields better accuracy/latency trade-offs on multi-core CPUs.

Recommendation- If memory < ~10MB and ultra-low engineering overhead needed → small LSTM.- If you can allocate ~20–100MB and invest in optimization, DistilBERT (or a smaller distilled variant) gives much better accuracy and benefits from CPU parallelism. Use int8 quantization, operator fusion, and an optimized runtime (TFLite/ONNX Runtime) for best results.

HardSystem Design

22 practiced

Design a Retrieval-Augmented Generation (RAG) system for knowledge-grounded QA over a company KB of 10M documents. Include choices for embedding model, ANN index (FAISS/ScaNN/Vector DB), index update strategy (near-real-time), re-ranker, generator model, caching, latency budgets, and measurements to ensure factuality and reduce hallucinations.

Sample Answer

Requirements & constraints:- 10M-doc company KB, low-latency QA (target P95 ≤ 2s), near-real-time updates (seconds–minutes), high factuality, explainable citations, support multi-tenancy and access controls.

High-level architecture:User → Query normalization → Embed query → ANN retrieve top-K → Re-ranker (cross-encoder) → Context construction + citation metadata → Generator (LLM) → Post-checks & caching → Response + sources.

Component choices and why:- Embedding model: use a high-quality semantic embedder (production: OpenAI text-embedding-3-large or a comparable open-source encoder distil/clone like sentence-transformers/all-mpnet-base-v2 for on-prem). Trade-off: cloud embeddings = better quality + managed; open-source = cost/control.- ANN index: FAISS with HNSW for low-latency vector search; combine with IVF-PQ for storage/compression if memory constrained. Alternatively use a managed vector DB (Milvus/Pinecone/Weaviate) for operational features (replication, RBAC, TTL).- Index update strategy: hybrid approach — write-through to primary vector DB for new/updated docs (near-real-time inserts), keep a small in-memory delta index (HNSW) for sub-second visibility; background merge job periodically compacts delta into main IVF index to maintain recall and storage efficiency.- Re-ranker: lightweight bi-encoder shortlist then cross-encoder (e.g., MonoT5 or t5-large fine-tuned for ranking) on top-20 to produce final top-5 — balances speed and precision.- Generator model: instruction-tuned LLM with grounding prompts (e.g., Llama 3 / GPT-4o with system prompt that forces citation and “answer only using provided sources”). Use retrieval-augmented prompt template with chunk provenance.- Caching: LRU query→(ranked doc ids, generated answer) cache for identical queries; snippet-level cache for popular docs; warm-up caching for embeddings of canonical queries.- Latency budgets (example P95 targets): embed 50–150ms, ANN retrieve 30–150ms, cross-encode rerank 50–200ms, generation 200–800ms → total aim P95 ≤ 2s.- Ops & monitoring: metrics for latency, QPS, index staleness, cache hit rate, and errors.

Measurements to ensure factuality / reduce hallucination:- Source grounding enforcement in prompts; require generator to highlight source ids for each claim.- Automatic checks: answer overlap vs top-k sources (token overlap, entailment via NLI model), answer provenance score (percentage of tokens covered by sources), and contradiction detection with an NLI model.- QA-based fidelity tests: generate synthetic questions from documents and check retrieval+generation R-precision/MRR and exact match.- Human-in-the-loop: sampled human evaluation for hallucination rate, precision@k, and utility.- Continuous feedback loop: label-driven reranker fine-tuning, calibrate generator temperature, add negative examples where generator hallucinates.- Safety: deny-listing, access controls, redaction for PII.

Trade-offs & notes:- If strict latency needed, favor smaller generator + stronger extractive answers; for higher fidelity, invest in better reranker + larger generator.- Plan for periodic re-embedding of docs after embedding model upgrades; use versioning and online A/B tests before full migration.

HardTechnical

16 practiced

Design an evaluation framework for abstractive summarization that goes beyond ROUGE to measure fluency, relevance, and factuality. Propose automated checks (QA-based factuality detection, entailment models), a human-eval protocol (rubrics, sampling, IAA), and how to combine automated signals into a monitoring dashboard to detect model regressions and hallucinations.

Sample Answer

Requirements & constraints:- Measure fluency, relevance, factuality for model outputs at scale; surface regressions/hallucinations; support triage and prioritization for engineers and product managers.

Automated checks (pipeline):1. Relevance/semantic overlap: - ROUGE + BERTScore + MoverScore for lexical/semantic overlap. - Embedding cosine (SBERT) to detect topic drift.2. Fluency/grammaticality: - Pretrained language-model perplexity (normalized by length). - Grammatical error classifier (fine-tuned RoBERTa). - Readability scores (Flesch–Kincaid) as a coarse signal.3. Factuality / hallucination detection: - QA-based faithfulness: generate Qs from summary (QG model), answer from source (QA model), compare answers (EM/F1); low match => potential hallucination. - NLI/entailment: entailment score from premise=source, hypothesis=summary (fine-tuned DeBERTa) to catch contradictions/unsupported claims. - Fact-check classifiers: DAE/FactCC-style model and QAFactEval for complementary signals.4. Calibration & uncertainty: - Keep model confidence scores; Bayesian/MC-dropout where possible. - Track distribution shifts of input/source (embedding drift).

Human-eval protocol:- Rubric with clear labels per summary: Fluency (1–4), Relevance (1–4), Factuality (Supported / Unsupported / Contradicted / Missing citations), Severity tags (minor, major).- Sampling: stratified sampling across model versions, source domains, confidence buckets, and automated-signal outliers (low entailment, low QA-match).- Annotator training: examples, gold anchors, calibration sessions.- IAA: compute Cohen’s kappa (or Krippendorff’s alpha) per label; target kappa > 0.6; adjudicate disagreements and update rubric.- Time-box: 3–5 human judgments per example for majority vote where high-stakes.

Combining signals & dashboard:- Compose a composite score per example: weighted aggregator (learned via logistic regression or small calibration network) taking normalized signals: QA-F1, entailment score, BERTScore, perplexity, confidence. Expose raw signals + composite.- Dashboard features: - Time-series of composite score and each signal by model version, dataset slice, and source domain. - Alerts: detect >X% drop in composite score or ≥Y increase in unsupported-factuality rate vs baseline. - Outlier explorer: list examples with contradictory signals (e.g., high BERTScore but low QA-F1) for human triage. - Heatmaps by token-level hallucination probability (from QA/NLI gradients) and corpus-level drift metrics.- Regression detection: - Automated daily jobs compute deltas with statistical tests (bootstrap CIs); flag regressions and create prioritized issues with representative failing examples. - Run A/B buckets and holdout test sets; require human-eval gate for releases where factuality metric crosses threshold.

Practicalities & trade-offs:- Combine complementary automated checks: no single metric suffices.- Human evaluation remains the gold standard; use active sampling to minimize labeling cost.- Continuously update calibration models and rubric as product/domain evolves.

This framework enables scalable monitoring, fast triage of hallucinations, and a human-in-the-loop safety gate for releases.

MediumTechnical

18 practiced

You're building a tokenizer for Chinese and Japanese text for NER. Compare character-level tokenization, lexicon-based word segmentation (e.g., Jieba/MeCab), and subword tokenization (SentencePiece). Discuss impacts on vocabulary size, alignment to entity spans, and handling of OOV terminology.

Sample Answer

Brief summary: for Chinese/Japanese NER the three options trade off vocabulary size, span alignment, and OOV robustness. Below I compare them and give practical guidance.

Character-level tokenization- Vocabulary size: very small (a few thousand to tens of thousands of characters).- Alignment to entity spans: excellent — every character maps directly so entity boundaries are easy to mark at character indices.- OOV handling: trivial — unseen words are composed of known characters so model can often generalize morphological/semantic cues.- Downsides: longer sequence lengths, less context per token (may need more model capacity to capture word-level semantics).

Lexicon-based word segmentation (Jieba/MeCab)- Vocabulary size: moderate to large (lexicon entries + inflected forms); depends on dictionary.- Alignment to entity spans: good when segmentation matches entity boundaries, but errors (under-/over-segmentation) can break entities and complicate labeling (requires convert-between-token-and-character-span logic).- OOV handling: poor for novel names/terminology unless you update lexicon; unknown words often split into suboptimal pieces.- Strength: tokens approximate semantic units, reducing sequence length and improving efficiency.

Subword tokenization (SentencePiece/BPE/Unigram)- Vocabulary size: controllable (e.g., 8k–32k); learns frequent character n-grams including whole words.- Alignment to entity spans: mixed — entities may map to multiple subword tokens; need careful span-to-subtoken mapping for label projection (use BIO on first subtoken or replicate labels).- OOV handling: strong — composes rare words from subword pieces, so novel names remain representable without unknown token.- Strength: balances vocabulary compactness and open-vocabulary capability; works well for transformer pretraining.

Recommendations for NER- If you must guarantee exact span alignment and prefer simpler label projection (and sequence length is acceptable): use character-level.- If you want semantic token units and have stable domain lexicons: use lexicon segmentation but augment dictionary and add fallback strategies.- For best generalization with transformer models and manageable vocab sizes: use SentencePiece with careful label projection (label first subtoken, or span-aware decoding) and consider hybrid approaches (character + subword embeddings or joint training).- Practical tip: evaluate on real entity spans, measure segmentation-induced span errors, and apply data augmentation (add synthetic OOV names) to test robustness.

Unlock Full Question Bank

Get access to hundreds of Natural Language Processing Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.