Natural Language Processing Questions

Approach: Use spaCy to batch-process texts with nlp.pipe for speed, preprocess with regex to remove URLs/emails, use the emoji library to demojize to textual tokens, then tokenize, lowercase, filter stopwords/punct/space, lemmatize, and optionally remove low-frequency tokens.

import re
from collections import Counter
import spacy
from emoji import demojize  # pip install emoji

URL_RE = re.compile(r'https?://\S+|www\.\S+')
EMAIL_RE = re.compile(r'\b[\w.-]+?@\w+?\.\w{2,4}\b')

def clean_tokenize(texts, nlp, min_freq=1):
    """
    texts: list[str]
    nlp: loaded spaCy model (e.g., spacy.load("en_core_web_sm"))
    min_freq: drop tokens with corpus frequency < min_freq
    returns: list[list[str]] tokenized cleaned docs
    """
    prepped = []
    for doc_text in texts:
        s = URL_RE.sub('', doc_text)
        s = EMAIL_RE.sub('', s)
        s = demojize(s)  # turns 😀 into :grinning_face:
        prepped.append(s.lower())

    docs = list(nlp.pipe(prepped, batch_size=128, disable=["ner"]))  # speed gain
    token_lists = []
    freq = Counter()

    for doc in docs:
        toks = []
        for t in doc:
            if t.is_punct or t.is_space:
                continue
            if t.like_num:
                continue
            if t.is_stop:
                continue
            lemma = t.lemma_.strip()
            if not lemma:
                continue
            toks.append(lemma)
            freq.update([lemma])
        token_lists.append(toks)

    if min_freq > 1:
        filtered = []
        for toks in token_lists:
            filtered.append([t for t in toks if freq[t] >= min_freq])
        return filtered
    return token_lists

Example:
Input: ["I love this app 😍! Visit https://example.com", "Email me at test@example.com — awful experience 😡"]
Output: [["love", ":smiling_face_with_heart_eyes:"], ["awful", ":pouting_face:"]]

Dependencies & versions (recommended):

• Python 3.8+
• spaCy >=3.2 (e.g., en_core_web_sm 3.2.0)
• emoji >=1.6.0

Performance considerations:

• Use nlp.pipe with disable=["ner"] when NER not needed to speed up processing.
• Increase batch_size for larger machines; monitor memory.
• For very large corpora, stream texts (generator) into nlp.pipe to avoid materializing all docs.
• If lemmatization speed is critical, consider spaCy's small models or spaCy-transformers trade-offs; also consider using a simpler stemmer (NLTK) for faster but less accurate normalization.

Teacher forcing is a training technique for seq2seq models where, at each decoder time step during training, the model receives the ground-truth previous token as input instead of its own previous prediction.

How it changes training dynamics:

• It simplifies learning by providing correct context each step, so the decoder learns to predict the next token conditioned on the true history. This speeds convergence and stabilizes gradients because errors don’t compound across steps during training.

Why it’s used:

• It reduces sequence-level credit assignment difficulty, enables faster training, and improves per-step prediction accuracy early in training.

Discrepancy between training and inference (exposure bias):

• At inference the model must feed its own sampled/predicted tokens back as inputs. Because it never experienced its own mistakes during training, small errors can compound, producing degraded or divergent sequences. This mismatch is called exposure bias.

Mitigation strategies:

• Scheduled sampling: mix ground-truth and model-generated tokens during training. Early epochs use mostly ground-truth inputs; gradually replace them with the model’s own predictions according to a schedule (linear, inverse sigmoid, or exponential). This exposes the model to its own errors and teaches recovery.
• Alternatives: sequence-level training (REINFORCE / minimum risk training) that optimizes sequence metrics, or Professor Forcing which uses adversarial loss to align hidden-state trajectories between training and sampling modes.

Key trade-off: more sampling during training reduces exposure bias but can increase training instability; schedules and hybrid losses balance robustness and learnability.

Approach: build a small regex-based text normalizer and pattern set for each theme (price, customer service, product bugs, missing features). Use case-insensitive patterns, allow common misspellings and punctuation-tolerant word boundaries. This is fast for 5k responses and interpretable.

import re
from collections import Counter

# compile patterns once (case-insensitive)
PATTERNS = {
    "price": re.compile(r"\b(pric|price|prce|cost|costly|expens|too expen|overpriced)\b", re.I),
    "customer_service": re.compile(r"\b(customer|cust\.? ?svc|custsvc|support|service|helpdesk|help desk|rep|representative)\b", re.I),
    "product_bugs": re.compile(r"\b(bug|bugs|crash|crashes|error|errors|fail(ed|ure)|glitch|issue|buggy|defect)\b", re.I),
    "missing_features": re.compile(r"\b(miss(ing)? feature|no feature|lack(s|ed) feature|need(s)? (feature|function)|want(s)? (feature|option)|not have|doesn'?t have)\b", re.I)
}

def preprocess(text):
    # basic normalization: lower, unify apostrophes, remove extra punctuation while preserving word boundaries
    t = text.lower()
    t = re.sub(r"[’‘`]", "'", t)
    t = re.sub(r"[\r\n]+", " ", t)
    # keep alphanumerics and common punctuation for token boundaries
    t = re.sub(r"[^a-z0-9\s\.\-']+", " ", t)
    return t

def extract_themes(responses):
    counts = Counter()
    matches = {k: [] for k in PATTERNS}
    for i, r in enumerate(responses):
        t = preprocess(r)
        for theme, pat in PATTERNS.items():
            if pat.search(t):
                counts[theme] += 1
                matches[theme].append((i, r))
    return counts, matches

# Sample input
sample = [
    "Too expensive — prce is high for what I get.",
    "Customer svc was rude and unhelpful.",
    "App keeps crashing, lots of bugs and errors.",
    "Missing features I need: reporting and export.",
    "Love it but support slow, and cost is high."
]

counts, _ = extract_themes(sample)
print(counts)  # expected: {'price':3? maybe 2, 'customer_service':2, 'product_bugs':1, 'missing_features':1}

Expected counts for sample (approximate): price:2 (lines 0,4), customer_service:2 (lines 1,4), product_bugs:1 (line 2), missing_features:1 (line 3).

Trade-offs:

• Regex pros: fast, transparent, easy to maintain; good when themes are well-defined and vocabulary limited.
• Regex cons: brittle to unseen paraphrases, long-tail misspellings, and context (e.g., "no more bugs" vs "still buggy"—both OK but sarcasm/hypotheticals can mislead).
• Fuzzy matching (edit distance) improves misspelling capture but increases false positives and compute cost; often use alongside regex for specific tokens.
• Supervised classification (fine-tuned model) handles varied phrasing and context better and can assign multiple labels, but needs labeled training data and is more complex to deploy and explain.

Recommendation: start with regex for initial triage, sample false positives/negatives, then iteratively add patterns or train a classifier if coverage/precision needs improvement.

A simple reproducible pipeline:

1. Approach: Clean text → TF-IDF (unigrams + bigrams, remove stopwords) → KMeans → choose k by silhouette/elbow → label clusters using top centroid features + exemplar responses.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import nltk
import re

def preprocess(text):
    text = text.lower()
    text = re.sub(r'\s+',' ', text)
    # optional: expand contractions, correct misspells, lemmatize
    return text

def vectorize(corpus):
    vec = TfidfVectorizer(ngram_range=(1,2),
                          max_df=0.6, min_df=5,
                          stop_words='english',
                          max_features=20000)
    X = vec.fit_transform(corpus)
    return X, vec

def choose_k(X, k_min=2, k_max=15):
    best_k, best_score = 2, -1
    for k in range(k_min, k_max+1):
        km = KMeans(n_clusters=k, random_state=42).fit(X)
        s = silhouette_score(X, km.labels_)
        if s > best_score:
            best_k, best_score = k, s
    return best_k

def label_clusters(km, vec, X, texts, top_n=10):
    terms = vec.get_feature_names_out()
    labels = {}
    centers = km.cluster_centers_
    for i in range(centers.shape[0]):
        top_terms = [terms[idx] for idx in centers[i].argsort()[-top_n:][::-1]]
        # exemplar responses closest to centroid
        reps_idx = ((X - centers[i]).power(2).sum(axis=1)).A1.argsort()[:3]
        examples = [texts[j] for j in reps_idx]
        labels[i] = {'top_terms': top_terms, 'examples': examples}
    return labels

Key choices & rationale:

• ngram_range (1,2) captures short phrases without exploding features.
• max_df/min_df reduce noise from extremely common/rare tokens.
• Choose k via silhouette (interpretable quality) and confirm with elbow on inertia.
• For stakeholders: present 3–5-word cluster labels made from top terms, 2–3 real example responses, cluster size and percent of total, and suggested business interpretation.
  Edge cases: short responses, multilingual text (detect language), heavy duplicates — deduplicate or weight accordingly.

Start with a reproducible pipeline outline and practical code snippets using gensim + scikit-learn for vectorization.

1. Preprocessing (reproducible)

• Tokenize, lower, remove stopwords, lemmatize, remove rare/common tokens, save random seed.
• Persist cleaned docs and the preprocessing pipeline (pickle/JSON).

Example:

import spacy, gensim
nlp = spacy.load("en_core_web_sm", disable=["parser","ner"])
def preprocess(text):
    doc = nlp(text.lower())
    return [tok.lemma_ for tok in doc if tok.is_alpha and not tok.is_stop]
clean_docs = [preprocess(d) for d in raw_docs]

2. Build dictionary & corpus (gensim)

• Filter extremes: no_below, no_above, keep_n

from gensim.corpora import Dictionary
dictionary = Dictionary(clean_docs)
dictionary.filter_extremes(no_below=5, no_above=0.5, keep_n=50000)
corpus = [dictionary.doc2bow(doc) for doc in clean_docs]

Key hyperparams: no_below, no_above, keep_n

3. Train LDA (gensim)

• Tune num_topics, passes, iterations, alpha, eta, chunksize, random_state

from gensim.models.ldamodel import LdaModel
model = LdaModel(corpus, id2word=dictionary, num_topics=25,
                 passes=10, iterations=400, alpha='asymmetric',
                 eta='auto', random_state=42, chunksize=1000)

Hyperparams to tune: num_topics (k), alpha (sparse vs symmetric), eta, passes, iterations, chunksize.

4. Selecting number of topics & evaluation

• Grid search over k (e.g., 5–50), compute coherence (c_v) and perplexity; prefer coherence for interpretability.

from gensim.models import CoherenceModel
coherence = CoherenceModel(model=model, texts=clean_docs, dictionary=dictionary, coherence='c_v').get_coherence()
perplex = model.log_perplexity(corpus)

Heuristics: choose k where coherence plateaus and topics remain semantically distinct; avoid very high coherence with duplicate topics. Use UMass/per-word perplexity only as secondary.

5. Topic labeling for stakeholders

• Extract top-n words per topic, compute representative documents (highest topic proportion), and create concise labels:
  • template: "<verb/noun phrase>: top words → example excerpt"
• Validate labels via small human-in-the-loop annotation (5–10 reviewers) and iterate.

6. Integrate into dashboard

• Precompute per-document topic distributions and top topic for each doc.

doc_topics = [model.get_document_topics(bow, minimum_probability=0.01) for bow in corpus]

• Store results in a table: doc_id, top_topic, topic_probs, top_words, exemplar_snippet.
• Expose via API to BI tools (Tableau/PowerBI) or build a simple Dash/Streamlit app showing:
  • Topic list with labels and top words
  • Topic prevalence over time
  • Filterable reviews by topic/sentiment
  • Representative documents and wordclouds

7. Additional best practices & heuristics

• Use bigrams/trigrams (gensim Phrases) to capture multiword concepts.
• Run stability checks: multiple seeds, measure topic similarity (Jaccard/JS divergence) across runs.
• Combine LDA with sentiment/metadata to prioritize actionable topics.
• Document pipeline, store models and dictionaries, version data, and provide a reproducibility script (requirements.txt, random seeds).

Time/Resource notes: 10k docs is small — iterate quickly; increase passes/iterations for stability when compute allows.