Natural Language Processing Questions

Direct answer

An embedding is a vector (a list of numbers) that represents a word so that similar words get similar vectors. A static embedding gives each word type exactly one vector, learned from co-occurrence statistics, meaning counts of which words appear near which others (Word2Vec, GloVe), so "bank" has the same vector in "river bank" and "bank loan". A contextual embedding (ELMo, BERT) computes a different vector for each occurrence of a token by running the whole sentence through a network, so the vector depends on the surrounding words. For a classification task on domain documents: choose static embeddings (or even TF-IDF) when labelled data is small, latency and memory are tight, and the label depends on which words appear rather than how they are arranged; choose a fine-tuned contextual encoder (fine-tuned: trained a little further on your labelled examples) when meaning depends on context or word order, you have enough labelled examples, and you can afford the compute. Decide with a baseline-first experiment, not by default.

The two kinds of embeddings

• Static: Word2Vec learns vectors by predicting the surrounding words from a centre word (skip-gram) or the centre word from its surroundings (CBOW, continuous bag of words); GloVe fits vectors to a global word co-occurrence count matrix. After training, the model is a lookup table of shape vocabulary size by dimension. A word not in the table has no vector (fastText-style models build vectors from character n-grams, short overlapping letter sequences, to soften this). To embed a document you pool the word vectors, which means combine them into one: a mean, or a TF-IDF weighted mean (each word's vector is multiplied by its TF-IDF weight, a number that is high for words frequent in this document but rare across documents, then the sum is divided by the total weight), and feed the result to a classifier.
• Contextual: ELMo reads the sentence with bidirectional LSTMs (recurrent networks that read the words one after another, once left to right and once right to left), BERT with a transformer encoder (a network whose layers let every token look at every other token). Both give one vector per token that depends on the whole sentence (BERT's tokens are subword pieces, fragments of words), and a sentence or document vector is produced by pooling (mean, or the vector of a special classification token that the model puts at the start of the input). For a task you either freeze the encoder and train a classifier on top (feature extraction) or fine-tune all the weights with a small learning rate (the step size of each weight update; a small one nudges the pretrained weights instead of overwriting them).

Worked example with hand-set vectors

The measure used throughout is cosine similarity, written cos. For two vectors $a$ and $b$ it is

the cosine of the angle between them: 1 means they point the same way, 0 means unrelated (at right angles), and -1 means opposite. Only the direction counts, not the length. Polysemy means one word with several unrelated senses, as "bank" has.

The program uses hand-set four-dimensional vectors (dimensions: finance, nature, person, action) so the mechanism is visible. The contextual vector is a stand-in: each word is mixed with the mean of the other words of its sentence. A real contextual encoder learns a far richer function; this only mimics "the vector depends on its neighbours". Because the mixing rule is written by hand, the 0.942 and 0.320 below follow from that rule; they illustrate the mechanism and are not a measurement of how any real encoder behaves.

import numpy as np

# Hand-set 4-d vectors; dimensions are [finance, nature, person, action]. Not trained: they make the mechanism visible.
E = {"bank":    np.array([0.5, 0.5, 0.0, 0.0]),     # one static vector must serve both senses
     "money":   np.array([1.0, 0.0, 0.0, 0.1]),
     "deposit": np.array([0.9, 0.0, 0.1, 0.5]),
     "river":   np.array([0.0, 1.0, 0.0, 0.0]),
     "fishing": np.array([0.0, 0.7, 0.2, 0.6]),
     "dog":     np.array([0.0, 0.3, 0.9, 0.0]),
     "man":     np.array([0.0, 0.0, 1.0, 0.0]),
     "bites":   np.array([0.0, 0.0, 0.2, 1.0])}
def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

# 1) static: the same vector for 'bank' in every sentence
print("static  cos(bank, money) =", round(cos(E["bank"], E["money"]), 3), "| cos(bank, river) =", round(cos(E["bank"], E["river"]), 3))

# 2) STAND-IN for a contextual encoder: mix each word with the mean of the other words in its sentence
def contextual(sentence, target, alpha=0.5):
    others = [E[w] for w in sentence if w != target]
    return alpha * E[target] + (1 - alpha) * np.mean(others, axis=0)

s_fin = ["deposit", "money", "bank"]
s_riv = ["fishing", "river", "bank"]
for name, s in (("finance sentence", s_fin), ("river sentence  ", s_riv)):
    v = contextual(s, "bank")
    print(f"context {name}: cos(bank_ctx, money) = {cos(v, E['money']):.3f} | cos(bank_ctx, river) = {cos(v, E['river']):.3f}")

# 3) a mean of static vectors ignores word order
a = np.mean([E[w] for w in ["dog", "bites", "man"]], axis=0)
b = np.mean([E[w] for w in ["man", "bites", "dog"]], axis=0)
print("mean-pooled static, 'dog bites man' vs 'man bites dog': cosine =", round(cos(a, b), 3))

# 3b) how a pooled document vector is built: plain mean, and a weighted mean with made-up illustrative weights
doc = ["deposit", "money", "bank"]
wts = {"deposit": 2.0, "money": 2.0, "bank": 1.0}
print("mean         :", np.round(np.mean([E[w] for w in doc], axis=0), 3))
print("weighted mean:", np.round(sum(wts[w] * E[w] for w in doc) / sum(wts.values()), 3))

# 4) memory of the lookup table: vocabulary x dimension x bytes per float
V, d = 100_000, 300
print("static table:", V * d * 4 / 1e6, "MB in float32 for", V, "words x", d, "dimensions")

Output:

static  cos(bank, money) = 0.704 | cos(bank, river) = 0.707
context finance sentence: cos(bank_ctx, money) = 0.942 | cos(bank_ctx, river) = 0.320
context river sentence  : cos(bank_ctx, money) = 0.358 | cos(bank_ctx, river) = 0.916
mean-pooled static, 'dog bites man' vs 'man bites dog': cosine = 1.0
mean         : [0.8   0.167 0.033 0.2  ]
weighted mean: [0.86 0.1  0.04 0.24]
static table: 120.0 MB in float32 for 100000 words x 300 dimensions

Reading the output:

• Polysemy: the static "bank" vector sits halfway between finance and nature (cosine 0.704 to "money", 0.707 to "river"), a blend that is not close to either sense. With context, "bank" in the finance sentence is close to "money" (0.942) and far from "river" (0.320), and in the river sentence the pattern reverses (0.358 and 0.916). This is the benefit that matters for domain text where common words carry specialised senses.
• Pooling by hand: the mean of deposit, money and bank adds the three vectors and divides by 3; in the finance dimension that is (0.9 + 1.0 + 0.5) / 3 = 0.8. With the made-up weights 2, 2 and 1 the weighted mean divides by 5 instead, giving (2 x 0.9 + 2 x 1.0 + 1 x 0.5) / 5 = 0.86, so the heavily weighted words pull the result toward themselves. Both vectors are printed above as a check.
• Order: a mean of static vectors for "dog bites man" and "man bites dog" is identical (cosine 1.0), since an average ignores order. Contextual encoders see order, which matters for negation, relations and anything where who did what is the label.
• Memory: a 100,000-word, 300-dimension static table is 120.0 MB in float32 (computed above), and a lookup costs almost nothing per token. A contextual encoder runs a network over every token of every document, so its serving cost depends on the number of layers and the sequence length, and its weights are usually far larger than a static table. Check the size and latency of the specific model you would deploy.

Trade-offs for domain-specific classification

Domain notes:

• Pretrained general-purpose vectors can miss domain terms (product codes, medical or legal terms) and carry the wrong sense for common words. Remedies: train or continue training static vectors on your own unlabelled domain text, or continue pretraining a contextual model on it, and compare against the general model.
• Frozen contextual features often beat static vectors with a linear classifier, but the gap depends on the task; fine-tuning the encoder gives the largest gain when you have labels. There is no result that holds for every dataset, so measure.

How I would decide

1. Build a TF-IDF plus logistic regression baseline first; it is quick and strong on many topic-style tasks.
2. Add pooled static embeddings (domain-trained if possible) and compare.
3. Fine-tune a contextual encoder and compare with a held-out set and per-class metrics.
4. Pick the cheapest option whose gain on the held-out set is larger than the noise (use several seeds or a bootstrap), then price in serving cost.
   What would flip the choice: a hard latency or memory budget (stay static), or errors dominated by negation and word order (go contextual).

Common mistakes

• Saying contextual embeddings are always better; on small, topic-driven datasets the simpler model can match them.
• Mean-pooling static vectors without TF-IDF weighting, so frequent uninformative words dominate the average.
• Using general-domain vectors for specialised text without checking what fraction of your tokens are unknown or fragmented.
• Comparing models trained with different splits or tuned on the test set.

Direct answer

Write the cleaning as a small pipeline applied to each review in a fixed order: replace URLs and emails with placeholder tokens (before anything else touches the punctuation), map emojis to named tokens, lowercase, expand negation contractions, split into word tokens, drop stop words but keep negations, and stem inflected forms. Return the token lists plus a Counter of all tokens, and apply the rare-term cut-off as a second step on the counts, not while counting. Cost is linear in the total number of characters, the frequency table needs memory proportional to the number of distinct terms (the token lists need memory proportional to the number of tokens unless streamed), and counting parallelizes because counts add.

Design decisions

• URLs and emails: replace each with <url> or <email> rather than deleting. The presence of a link or an address is a feature (spam, a request for contact), while the exact string is unique and would only bloat the vocabulary and leak identity. Do this first, because the later step that removes punctuation would break the pattern.
• Emojis: emotion-bearing for reviews, so keep them. A small mapping turns a few common emojis into named tokens; any other emoji becomes the generic <emoji>. A production list comes from a maintained library or the Unicode emoji data, not a handwritten range. The range used here covers common emoji blocks only.
• Stop words: remove common function words to shrink the vocabulary, but remove the negations from the stop list. The list here is scikit-learn 1.5.2's ENGLISH_STOP_WORDS minus not, no, nor, never, cannot, nothing, none, nobody, neither, nowhere and without, because "not slow" and "slow" are opposite reviews and the stock list would also silently delete "nothing works". Negations are also kept unstemmed. Review the whole list for your domain before trusting it.
• Contractions: "can't" becomes "can not" and "doesn't" becomes "does not" before tokenizing, so the negation survives as its own token instead of being cut off at the apostrophe. Typographic apostrophes (the curly ’ that phone keyboards insert) are first turned into the straight ' ; without that, "doesn’t" would split into doesn and t and the negation would be lost. The irregular forms (can't, cannot, won't) are rewritten next, then the general n't rule handles the rest. The order matters because the general rule alone would turn "can't" into "ca not" and "won't" into "wo not"; the trace at the end of the program shows this.
• Inflected forms: a stemmer cuts suffixes by rule so that "loved", "loves" and "loving" share one feature. NLTK's PorterStemmer works without downloaded data, but it is not a lemmatizer: it produces non-words ("terrible" becomes terribl, "battery" becomes batteri) and sometimes odd ones ("does" becomes doe). Lemmatization (looking each word up in a dictionary, with its part of speech, to return its base form such as "better" to "good"; it needs language resources) gives real words and is slower. For classification features a stem is fine; for display or for search matching, lemmas read better.
• Rare terms: min_count is applied after counting, to both the documents and the frequency table, so the caller can choose the cut-off without re-cleaning.
• Bad input: None or a non-string review becomes an empty token list instead of crashing the batch.

Code and example

How to read the four patterns before the code. URL matches http:// or https:// or www. followed by \S+, which means one or more non-whitespace characters, so it swallows the whole link up to the next space. EMAIL matches [\w.+-]+ (one or more letters, digits, underscores, dots, pluses or hyphens), then @, then a domain part [\w-]+ followed by one or more .something groups, so jo.k@mail.example.org matches as a whole. The (?<![\w.+-]) at its start means "the match may not begin in the middle of such a run"; it finds the same addresses in ordinary text (it can differ only on malformed text where one address runs straight into the next with no space, such as a@b.c-+@d.e, where the lookbehind skips the second address), and it stops the engine from retrying the pattern at every position of a very long unbroken run of letters, which would otherwise make that one pass slow (quadratic) on such a review. EMOJI is a character class covering two Unicode ranges where most emoji live. TOKEN is applied last, on lowercased text, and matches either a placeholder such as <url> (<[a-z_]+>) or a run of lowercase letters and digits ([a-z0-9]+), so punctuation is skipped and placeholders survive intact. Terms of art used below: a generator is a function that yields one item at a time instead of building a whole list; a shard is one slice of the data handled on its own; a worker process is a separate copy of the program running on one shard.

import re
from collections import Counter
from nltk.stem import PorterStemmer
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

URL = re.compile(r"(?:https?://|www\.)\S+")
EMAIL = re.compile(r"(?<![\w.+-])[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
EMOJI = re.compile("[\U0001F300-\U0001FAFF☀-➿]")
EMOJI_NAMES = {"\U0001F60D": "<emoji_love>", "\U0001F621": "<emoji_angry>", "\U0001F44D": "<emoji_thumbsup>"}
TOKEN = re.compile(r"<[a-z_]+>|[a-z0-9]+")
NEGATIONS = {"not", "no", "nor", "never", "cannot", "nothing", "none", "nobody", "neither", "nowhere", "without"}
STOP = frozenset(ENGLISH_STOP_WORDS) - NEGATIONS           # keep negations: "not good" must not become "good"
stem = PorterStemmer().stem

def clean_one(text):
    if not isinstance(text, str):                          # None, numbers etc. become an empty review
        return []
    t = URL.sub(" <url> ", text)
    t = EMAIL.sub(" <email> ", t)
    t = EMOJI.sub(lambda m: " " + EMOJI_NAMES.get(m.group(), "<emoji>") + " ", t)
    t = t.lower().replace("\u2019", "'")                   # curly apostrophe -> straight, so the n't rules see it
    t = re.sub(r"\b(can't|cannot)\b", "can not", t)
    t = re.sub(r"\bwon't\b", "will not", t)
    t = re.sub(r"n't\b", " not", t)
    out = []
    for tok in TOKEN.findall(t):
        if tok.startswith("<") or tok in NEGATIONS or tok not in STOP:
            out.append(tok if tok.startswith("<") or tok in NEGATIONS else stem(tok))
    return out

def preprocess(reviews, min_count=1):
    """Return (tokenised reviews, corpus term-frequency dict). Terms seen fewer than min_count times are dropped from both."""
    docs = [clean_one(r) for r in reviews]
    tf = Counter(tok for d in docs for tok in d)           # one pass: O(total tokens)
    if min_count > 1:
        docs = [[t for t in d if tf[t] >= min_count] for d in docs]
        tf = Counter({t: c for t, c in tf.items() if c >= min_count})
    return docs, dict(tf)

reviews = ["Loved it!! Works great, see https://shop.example.com/p/12 \U0001F44D",
           "Terrible. It doesn't work and I can't get a refund, mail me at jo.k@mail.example.org \U0001F621",
           "The battery lasts longer than the old one. Charging is not slow.",
           None,
           "Loved the screen, loved the sound, 5 stars \U0001F60D"]
docs, tf = preprocess(reviews)
for r, d in zip(reviews, docs):
    print(repr(r)[:60], "->", d)
print("distinct terms:", len(tf), "| top 5:", sorted(tf.items(), key=lambda kv: (-kv[1], kv[0]))[:5])
docs2, tf2 = preprocess(reviews, min_count=2)
print("min_count=2 keeps:", sorted(tf2.items(), key=lambda kv: (-kv[1], kv[0])))

# scaling: counts are additive, so shards can be counted independently and the Counters summed
shards = [reviews[:2], reviews[2:]]
merged = sum((Counter(preprocess(s)[1]) for s in shards), Counter())
assert dict(merged) == tf
print("sharded count equals single-pass count:", dict(merged) == tf)

# trace the contraction rewrites on one sentence
s = "i can't get a refund and it won't help"
print("generic n't rule alone:", re.sub(r"n't\b", " not", s))
step = re.sub(r"\bwon't\b", "will not", re.sub(r"\b(can't|cannot)\b", "can not", s))
print("special cases first   :", re.sub(r"n't\b", " not", step))
print("tokens kept           :", clean_one(s))
print("stop words in the text:", [w for w in "i can not get a refund and it will not help".split() if w in STOP])
print("curly apostrophes     :", clean_one("It doesn\u2019t work and I can\u2019t get a refund"))
print("other negations kept  :", clean_one("Nothing works, none of it, and it came without sound"))

Output:

'Loved it!! Works great, see https://shop.example.com/p/12 👍 -> ['love', 'work', 'great', '<url>', '<emoji_thumbsup>']
"Terrible. It doesn't work and I can't get a refund, mail me -> ['terribl', 'doe', 'not', 'work', 'not', 'refund', 'mail', '<email>', '<emoji_angry>']
'The battery lasts longer than the old one. Charging is not  -> ['batteri', 'last', 'longer', 'old', 'charg', 'not', 'slow']
None -> []
'Loved the screen, loved the sound, 5 stars 😍' -> ['love', 'screen', 'love', 'sound', '5', 'star', '<emoji_love>']
distinct terms: 23 | top 5: [('love', 3), ('not', 3), ('work', 2), ('5', 1), ('<email>', 1)]
min_count=2 keeps: [('love', 3), ('not', 3), ('work', 2)]
sharded count equals single-pass count: True
generic n't rule alone: i ca not get a refund and it wo not help
special cases first   : i can not get a refund and it will not help
tokens kept           : ['not', 'refund', 'not', 'help']
stop words in the text: ['i', 'can', 'get', 'a', 'and', 'it', 'will']
curly apostrophes     : ['doe', 'not', 'work', 'not', 'refund']
other negations kept  : ['nothing', 'work', 'none', 'came', 'without', 'sound']

Reading the output: the first review's URL became <url> and the thumbs-up emoji <emoji_thumbsup>. The second review, "doesn't work and I can't get a refund", keeps two not tokens, one from each contraction, and its email became <email>. The third keeps "not slow" as not, slow, with "battery" stemmed to batteri. The None review became an empty list. There are 23 distinct terms in this five-review corpus; love (3) and not (3) lead the table, and min_count=2 keeps only love, not and work. Why does "can" vanish from "I can't get a refund" while not stays? After the rewrite the text reads "i can not get a refund"; can is on the stop list (a common function word) and not was deliberately taken off it, so only not is kept. That is intended, not a bug: the pair not, refund carries the complaint. The last two lines show the two protections: with typographic apostrophes "doesn’t" and "can’t" still become not, and nothing, none and without survive the stop list. The "sharded count" line confirms that counting two shards separately and adding the Counters gives the same table as one pass (the assert fails if they differ).

Complexity and scaling

Let $N$ be the total characters, $T$ the total tokens and $V$ the number of distinct terms. $O(N)$ reads "the work grows in proportion to $N$".

• Time: each regex pass is linear in the review length (the email pattern is anchored with the lookbehind described above so that a long unbroken run of characters does not make it quadratic), so cleaning is $O(N)$; counting is $O(T)$ dictionary updates; the min_count filter is $O(T + V)$.
• Memory: the frequency table is $O(V)$. The token lists above hold all $T$ tokens at once, which does not scale to a very large corpus: process reviews as a generator and write token lists to disk or feed them straight into the next stage.
• Parallelism: counts are additive, so run the cleaning and counting on shards in worker processes and sum the Counters (the program checks that this equals the single-pass table).
• Apply min_count only to the merged table. Pruning each shard first is wrong, because a term can be below the threshold in every shard and above it overall.
• If $V$ grows too large (the tail of rare terms is long in real text), hash tokens into a fixed number of buckets (the hashing trick: a hash function turns each token into a bucket number, so no vocabulary is stored, as in scikit-learn's HashingVectorizer), or use a probabilistic counter (a small sketch structure that estimates counts in fixed memory); the cost is collisions, where two different tokens share a bucket, and losing the ability to list terms.
• Compile regexes once at module level, as above, and keep the preprocessing function identical in training and serving.

Pitfalls

• Lowercasing before extracting URLs or emails is harmless, but removing punctuation before extracting them breaks both.
• Stemming and stop-word lists are language specific; check the language before applying English rules.
• Do not remove negations, and think before removing numbers (ratings like "5 stars" carry signal).
• The token pattern here keeps only letters and digits; accented letters would be split or dropped, so use Unicode-aware patterns (for example \w) for non-English reviews.

Direct answer

Turn each response into a TF-IDF vector, compress it with a truncated SVD, cluster with k-means using fixed seeds, choose the number of themes by combining silhouette, seed stability and whether the clusters make sense to a reader, then name each cluster from its highest-weighted terms and its most central responses. The whole chain is deterministic once the cleaning rules, library versions and seeds are pinned, which is what "quick and reproducible" means here.

The pipeline in order

1. Clean the text. Lowercase, strip URLs and punctuation, drop an explicit stop-word list (very common words such as "the") and words that appear in fewer than 5 responses (min_df=5). Keep the rules in code so a rerun gives the same vocabulary.
2. TF-IDF. TF-IDF (term frequency times inverse document frequency) gives each word in a response the weight $\text{tf}(w,d) \times \log\frac{N}{\text{df}(w)}$, where $\text{tf}(w,d)$ is how often word $w$ occurs in response $d$, $N$ is the number of responses and $\text{df}(w)$ is the number of responses containing $w$. Words that appear everywhere get low weight, words concentrated in a few responses get high weight. scikit-learn's TfidfVectorizer uses a smoothed variant of this formula and, with sublinear_tf=True, replaces tf by $1+\log(\text{tf})$. The result is a sparse matrix (a table that stores only its non-zero entries, since most words are absent from most responses) of shape (2,000 responses, vocabulary size). The next section works the formula on three tiny responses.
3. Compress. Truncated SVD (singular value decomposition, which keeps only the few directions along which the word-weight vectors vary most, so words that tend to occur together end up sharing a direction) turns each sparse row into a dense vector of, say, 20 numbers, and Normalizer rescales each vector to length 1 so that k-means compares direction (topic mix), not response length. This step is optional for 2,000 short responses but makes k-means faster and less noisy.
4. Cluster. k-means puts each vector in the nearest of k centres, recomputes the centres as the cluster means and repeats; it minimises inertia (the sum of squared distances to the assigned centre) and ends in a local optimum (a solution no small change improves, though a better one may exist elsewhere), so use n_init=10 restarts (ten complete runs from different random starting centres, keeping the one with the lowest inertia) and a fixed random_state.
5. Choose k, label, review (below).

Choosing the number of themes

No single number decides it. Check three things:

• Inertia curve (elbow). Inertia always falls as k grows, so look for where the drop flattens. On text the elbow is usually soft.
• Silhouette. For each response, $s = (b-a)/\max(a,b)$ where $a$ is its mean distance to its own cluster and $b$ its mean distance to the nearest other cluster. It lies in $[-1, 1]$ and the mean over responses is the score. Higher means tighter, better-separated clusters.
• Stability. Cluster with several different seeds and compare the labelings with the adjusted Rand index (ARI: it looks at every pair of responses and checks whether the two labelings both put the pair together or both keep it apart, corrected for chance; 1 means identical groupings, about 0 means chance agreement, and the cluster numbers themselves do not matter). A k whose themes change between seeds is not a dependable set of themes.

Then apply the stakeholder test: read the top terms and examples of each cluster at k-1, k and k+1, and prefer the smallest k whose clusters have distinct, nameable meanings. A product team can act on 4 to 8 themes; 30 is a taxonomy nobody reads.

Three formulas on tiny numbers

The program below works TF-IDF on three short responses, one silhouette value on four points, and the adjusted Rand index on six labels. The TF-IDF lines use the plain formula from step 2; scikit-learn's default adds smoothing, so the last line compares its weight for one word.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import adjusted_rand_score, silhouette_score

# TF-IDF by hand on three tiny responses.
docs = ["price too high", "price fee high high", "slow support"]
N = len(docs)
vocab = sorted({w for d in docs for w in d.split()})
df = {w: sum(w in d.split() for d in docs) for w in vocab}
print("df:", df)
for i, d in enumerate(docs):
    words = d.split()
    row = {w: words.count(w) * np.log(N / df[w]) for w in sorted(set(words))}
    print(f"response {i} ('{d}'): " + ", ".join(f"{w} {v:.3f}" for w, v in row.items()))
# scikit-learn's default is smoothed: idf = ln((1 + N) / (1 + df)) + 1, then each row is scaled to length 1.
tv = TfidfVectorizer(norm=None)
M = tv.fit_transform(docs).toarray()
w = list(tv.get_feature_names_out()).index("price")
print("scikit-learn weight of 'price' in response 0 (smoothed idf, no length scaling):", round(M[0, w], 3),
      "= 1 * (ln(4/3) + 1) =", round(np.log(4 / 3) + 1, 3))

# Silhouette by hand: four points, two obvious groups.
pts = np.array([[0, 0], [0, 1], [4, 0], [4, 1]], dtype=float)
lab = np.array([0, 0, 1, 1])
d = lambda i, j: float(np.linalg.norm(pts[i] - pts[j]))
a, b = d(0, 1), (d(0, 2) + d(0, 3)) / 2
print(f"point 0: a = {a:.3f} (distance to its cluster mate), b = {b:.3f} (mean distance to the other cluster), s = {(b - a) / max(a, b):.3f}")
print("scikit-learn mean silhouette over the four points:", round(silhouette_score(pts, lab), 3))

# Adjusted Rand index on six responses: label names do not matter, groupings do.
truth = [0, 0, 0, 1, 1, 1]
print("same grouping, names swapped:", adjusted_rand_score(truth, [1, 1, 1, 0, 0, 0]))
print("one response moved:", round(adjusted_rand_score(truth, [0, 0, 1, 1, 1, 1]), 3))
print("unrelated grouping:", round(adjusted_rand_score(truth, [0, 1, 0, 1, 0, 1]), 3))

Output:

df: {'fee': 1, 'high': 2, 'price': 2, 'slow': 1, 'support': 1, 'too': 1}
response 0 ('price too high'): high 0.405, price 0.405, too 1.099
response 1 ('price fee high high'): fee 1.099, high 0.811, price 0.405
response 2 ('slow support'): slow 1.099, support 1.099
scikit-learn weight of 'price' in response 0 (smoothed idf, no length scaling): 1.288 = 1 * (ln(4/3) + 1) = 1.288
point 0: a = 1.000 (distance to its cluster mate), b = 4.062 (mean distance to the other cluster), s = 0.754
scikit-learn mean silhouette over the four points: 0.754
same grouping, names swapped: 1.0
one response moved: 0.324
unrelated grouping: -0.111

Reading it: "price" appears in 2 of 3 responses, so its weight per occurrence is $\ln(3/2) = 0.405$, while "too", which appears in one response, gets $\ln 3 = 1.099$. In response 1 "high" occurs twice and its weight doubles to 0.811. For the silhouette point, $a = 1$ (its cluster mate is 1 away), $b = 4.062$ (the mean distance to the two points of the other cluster), so $s = (4.062 - 1)/4.062 = 0.754$. For the ARI, swapping the cluster names keeps 1.0, moving one response out of six drops it to 0.324, and an unrelated grouping falls below 0. So the 0.995 reported below means the clusters and the hidden themes treat almost every pair of responses the same way.

Worked run

The data below are 2,000 synthetic responses generated in the code from 4 hidden themes (pricing, support, app, delivery), a stand-in for real survey text, so the final line can compare against the truth. Real responses are messier, with lower silhouette values and fuzzier clusters.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import Normalizer
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, adjusted_rand_score

# Synthetic survey: 2,000 short responses drawn from 4 hidden themes (a stand-in for real data).
rng = np.random.default_rng(0)
themes = {
    "pricing":  "price expensive cost cheaper fee subscription billing charge overpriced discount".split(),
    "support":  "support agent helpful response ticket waiting slow chat representative reply".split(),
    "app":      "app crash bug slow loading login update screen freezes mobile".split(),
    "delivery": "delivery late package shipping arrived damaged courier tracking delay box".split(),
}
filler = "the is very really always never my was it when and to for with often just so".split()
names = list(themes)
truth, texts = [], []
for _ in range(2000):
    t = int(rng.integers(4))
    n_theme = int(rng.integers(2, 5))
    words = list(rng.choice(themes[names[t]], n_theme)) + list(rng.choice(filler, int(rng.integers(3, 6))))
    if rng.random() < 0.15:                      # 15% also mention a second theme
        words += list(rng.choice(themes[names[int(rng.integers(4))]], 1))
    rng.shuffle(words)
    truth.append(t)
    texts.append(" ".join(words))
truth = np.array(truth)

# Pipeline: TF-IDF -> truncated SVD (dense, 20 dims) -> unit length -> k-means.
tfidf = TfidfVectorizer(stop_words=filler, min_df=5, sublinear_tf=True)
X_sparse = tfidf.fit_transform(texts)
X = Normalizer().fit_transform(TruncatedSVD(20, random_state=0).fit_transform(X_sparse))
print("matrix", X_sparse.shape, "vocabulary", len(tfidf.vocabulary_))

print("k  inertia  silhouette  seed-stability(ARI)")
for k in range(2, 9):
    labs = [KMeans(k, n_init=10, random_state=s).fit_predict(X) for s in range(5)]
    km = KMeans(k, n_init=10, random_state=0).fit(X)
    stab = np.mean([adjusted_rand_score(labs[i], labs[j]) for i in range(5) for j in range(i + 1, 5)])
    print(f"{k}  {km.inertia_:7.1f}  {silhouette_score(X, km.labels_):10.3f}  {stab:.3f}")

# Label the chosen k=4: top TF-IDF terms per cluster and the response nearest its centre.
km = KMeans(4, n_init=10, random_state=0).fit(X)
terms = np.array(tfidf.get_feature_names_out())
for c in range(4):
    idx = np.where(km.labels_ == c)[0]
    top = terms[np.asarray(X_sparse[idx].mean(axis=0)).ravel().argsort()[::-1][:4]]
    nearest = idx[np.argmin(((X[idx] - km.cluster_centers_[c]) ** 2).sum(1))]
    print(f"cluster {c}: n={len(idx)} terms={list(top)} example='{texts[nearest]}'")
print("agreement with hidden themes (ARI):", round(adjusted_rand_score(truth, km.labels_), 3))

Output:

matrix (2000, 39) vocabulary 39
k  inertia  silhouette  seed-stability(ARI)
2   1556.0       0.133  1.000
3   1349.9       0.194  1.000
4   1171.3       0.241  1.000
5   1121.0       0.217  0.882
6   1067.7       0.198  0.788
7   1018.3       0.179  0.627
8    975.2       0.174  0.689
cluster 0: n=492 terms=['late', 'shipping', 'damaged', 'arrived'] example='delivery often shipping delay box and really'
cluster 1: n=477 terms=['fee', 'discount', 'charge', 'expensive'] example='was subscription when the is charge just fee expensive'
cluster 2: n=486 terms=['login', 'bug', 'loading', 'app'] example='update just was when loading bug was often'
cluster 3: n=545 terms=['support', 'ticket', 'representative', 'reply'] example='helpful slow representative to never reply my for'
agreement with hidden themes (ARI): 0.995

The example responses above read like word salad because the generator draws theme words and filler words independently, with no grammar. A real response is ordinary prose, such as "Billing charged me twice and nobody replied for a week", which mixes two themes in one sentence and gives fuzzier clusters.

Reading it: on this draw the silhouette is highest at k=4 (0.241), the clusters at k=4 are identical across the 5 seeds (ARI 1.000), and stability falls to 0.882 at k=5 and lower beyond, so k=4 is the choice. The silhouette peak is the steadier signal: on other random draws from the same generator it also peaks at k=4, while the stability pattern varies (a smaller k can be unstable on one draw, and k=5 can stay stable on another), so stability screens out unreliable values of k and does not choose one alone. The inertia keeps falling, which is why inertia alone cannot pick k. The last line shows the 4 clusters match the 4 hidden themes (ARI 0.995); that check exists only because the data are synthetic.

Labeling for business stakeholders

• Show the 4 to 5 highest-weighted terms per cluster (the cluster's mean TF-IDF) and the 3 to 5 responses nearest to its centre. Here cluster 0 reads as delivery problems ("late, shipping, damaged, arrived"), cluster 1 as pricing and fees, cluster 2 as app bugs and loading, cluster 3 as support responsiveness.
• A person writes the label in plain business language (for example "Late or damaged deliveries"), after reading about 20 random members of the cluster, not only the central ones, to check it is coherent.
• Report each theme's size as a share of responses (here 492, 477, 486 and 545 of 2,000) and keep a "needs review" bucket for responses far from every centre.
• Save the vectorizer vocabulary, SVD, centres, seeds and library versions so the same responses always get the same theme.

Pitfalls

• k-means gives each response one theme, but a response mentioning price and delivery belongs to two; if that is common, use a topic model (topic models give each response a mix of themes) or multi-label classification instead.
• Silhouette on text is low in absolute terms (0.24 above even on clean data); compare values across k, not against a "good" threshold.
• Rerunning with different cleaning rules shifts clusters, so changes to preprocessing are versioned like code.

Direct answer

Pick the metric from the shape of the output, not from the model. Classification uses precision, recall and F1 (with accuracy only when classes are balanced). Tagging is scored on whole entities, not single tags. Translation uses BLEU (the share of the output's word sequences of length 1 to 4 that appear in a reference translation, combined by a geometric mean and cut down by a brevity penalty when the output is shorter than the reference) at corpus level, with a human or learned judgment as a cross-check. Summarization uses ROUGE (n-gram recall against reference summaries) plus a faithfulness check. Language modeling uses perplexity. Corpus-level scoring matters whenever the metric is a ratio of counts that is unstable on one sentence (BLEU above all). The usual pitfalls are listed at the end; those on skewed classes, tagging granularity and perplexity carry a number from the run, and the others are practices.

The task families and their metrics

Precision is the share of predicted positives that are right, recall is the share of real positives found, and $F_1 = 2PR/(P+R)$ is their harmonic mean (it is low when either is low). Macro averaging computes F1 per class and averages the classes equally. Micro averaging pools all decisions first, so large classes dominate.

Terms in the table, in plain words: ROUGE-1, ROUGE-2 and ROUGE-L count the overlap of single words, word pairs and the longest shared word sequence with a reference summary; chrF scores overlapping character sequences; a learned metric is a trained model that rates an output; calibration means that a stated confidence matches how often the model is right; faithfulness means every claim in a summary is supported by the source; negative log-likelihood (NLL) is minus the natural log of the probability the model gave the correct token; a bootstrap re-scores many resamples of the test set, drawn with replacement, to see how much a score moves by chance. If time is short, the three to be ready to defend are entity-level F1, corpus-level BLEU and perplexity, plus why accuracy fails on skewed classes.

The metric does not depend on whether the model is a convolutional network (CNN, which scans short windows of tokens), a recurrent network (RNN, which reads tokens one at a time while carrying a memory) or a Transformer (which lets every token look at every other). All three produce the same kind of output, so a tagger is scored with entity F1 whichever one it is. The model family shows up in the production metrics: latency, throughput, memory and cost per request, which are measured on the serving hardware and are what decide whether a small CNN or a larger Transformer is allowed into the product.

A run that shows the numbers

The program below uses scikit-learn 1.5.2 and NLTK 3.9.1 (NLTK is the Natural Language Toolkit library). BLEU here is NLTK's, with 4-gram uniform weights and its smoothing method 1 for sentence scores (smoothing adds a small amount, 0.1, to any n-gram match count that is zero, so one missing order does not zero the whole score).

import math
from collections import Counter
import numpy as np
from sklearn.metrics import accuracy_score, f1_score
from nltk.translate.bleu_score import corpus_bleu, sentence_bleu, SmoothingFunction

# 1. Classification: 95 of 100 messages are "ok", 5 are "spam"; a model that always says "ok".
y_true = ["ok"] * 95 + ["spam"] * 5
y_pred = ["ok"] * 100
print("accuracy", accuracy_score(y_true, y_pred))
print("macro-F1", round(f1_score(y_true, y_pred, average="macro", zero_division=0), 3))

# 2. Tagging: entity-level versus token-level (BIO tags).
gold = ["B-ORG", "I-ORG", "I-ORG", "O", "B-LOC", "I-LOC"]
pred = ["B-ORG", "I-ORG", "O", "O", "B-LOC", "O"]
def spans(tags):
    out, start = set(), None
    for i, t in enumerate(tags + ["O"]):
        if t == "O" or t.startswith("B-"):
            if start is not None:
                out.add((start, i, tags[start][2:]))
                start = None
            if t != "O":
                start = i
    return out
tok_ok = sum(g == p for g, p in zip(gold, pred))
print("token accuracy", round(tok_ok / len(gold), 3))
g, p = spans(gold), spans(pred)
print("entity exact matches", len(g & p), "of", len(g), "gold entities")

# 3. Translation: corpus-level BLEU versus the mean of sentence-level BLEU (4-gram, uniform weights).
refs = [[["the", "cat", "sat", "on", "the", "mat", "today"]],
        [["he", "did", "not", "go", "to", "the", "market", "at", "all"]],
        [["ok"]]]
hyps = [["the", "cat", "sat", "on", "the", "mat", "today"],
        ["he", "did", "not", "go", "to", "the", "shop", "at", "all"],
        ["fine"]]
sm = SmoothingFunction().method1
sent = [sentence_bleu(r, h, smoothing_function=sm) for r, h in zip(refs, hyps)]
print("sentence BLEU", [round(s, 3) for s in sent], "mean", round(float(np.mean(sent)), 3))
print("corpus BLEU", round(corpus_bleu(refs, hyps), 3))

# 4. Summarization: ROUGE-1 recall and precision (unigram overlap, counts clipped).
ref = "the council approved the new budget on monday".split()
cand = "council approved budget".split()
overlap = sum((Counter(ref) & Counter(cand)).values())
print("ROUGE-1 recall", round(overlap / len(ref), 3), "precision", round(overlap / len(cand), 3))

# 5. Language modelling: perplexity = exp(mean negative log-likelihood per token).
nll_word = [2.0, 1.0, 3.0, 2.0]                 # 4 word tokens, natural-log NLL each
total = sum(nll_word)
print("perplexity per word", round(math.exp(total / 4), 3))
print("same text split into 10 subword tokens", round(math.exp(total / 10), 3))

Output:

accuracy 0.95
macro-F1 0.487
token accuracy 0.667
entity exact matches 0 of 2 gold entities
sentence BLEU [1.0, 0.661, 0] mean 0.554
corpus BLEU 0.747
ROUGE-1 recall 0.375 precision 1.0
perplexity per word 7.389
same text split into 10 subword tokens 2.226

How to read each block:

• Classification. A model that always answers "ok" scores 0.95 accuracy yet 0.487 macro-F1, because it finds none of the 5 spam messages. With imbalanced classes report per-class precision and recall.
• Tagging. Per-token accuracy is 0.667, which sounds respectable because the O tags are easy. Both predicted entities have a wrong boundary (the organization stops one token early, the location drops its last token), so zero of the 2 gold entities match exactly. Entity-level F1 is 0, and that is the number a user of the extractor feels.
• Translation. Corpus-level BLEU pools the n-gram match counts and the lengths over all sentences before taking the geometric mean, so it is 0.747. The mean of the three sentence-level scores is 0.554, and the one-word pair with no shared word scores 0 by itself. Sentence-level BLEU is therefore noisy and is a poor ranking signal for single outputs. Corpus-level scoring is the right unit for comparing systems, and it is also not the average of sentence scores.
• Summarization. The candidate "council approved budget" has ROUGE-1 precision 1.0 and recall 0.375, because it is short and every word is in the reference. Always report recall and precision (or F1) together, or the shortest output wins.
• Language modeling. The same four words with the same total negative log-likelihood have perplexity 7.389 per word and 2.226 per subword when the text is split into 10 subword tokens. The model did not improve; the denominator changed. The four NLL values are made-up per-word figures for illustration, and the 10-subword split is an assumed tokenization of the same text: a real tokenizer decides where the tokens end.

BLEU by hand on the same three pairs

The program below counts, for each sentence, the matching n-grams out of the hypothesis n-grams for n = 1 to 4, adds the counts over the three sentences, and rebuilds the corpus number from the pooled counts and the lengths.

import math
from collections import Counter
from nltk.translate.bleu_score import corpus_bleu

refs = [["the", "cat", "sat", "on", "the", "mat", "today"],
        ["he", "did", "not", "go", "to", "the", "market", "at", "all"],
        ["ok"]]
hyps = [["the", "cat", "sat", "on", "the", "mat", "today"],
        ["he", "did", "not", "go", "to", "the", "shop", "at", "all"],
        ["fine"]]

def grams(t, n): return Counter(tuple(t[i:i + n]) for i in range(len(t) - n + 1))

match, total = [0] * 4, [0] * 4
for r, h in zip(refs, hyps):
    row = []
    for n in range(1, 5):
        m = sum((grams(h, n) & grams(r, n)).values())      # clipped matches
        t = sum(grams(h, n).values())                      # n-grams in the hypothesis
        match[n - 1] += m; total[n - 1] += t; row.append(f"{m}/{t}")
    print("matches/total for n=1..4:", ", ".join(row))
print("pooled matches/total:", ", ".join(f"{m}/{t}" for m, t in zip(match, total)))

c, r = sum(map(len, hyps)), sum(map(len, refs))
bp = 1.0 if c > r else math.exp(1 - r / c)
print(f"hypothesis length {c}, reference length {r}, brevity penalty {bp:.3f}")
# NLTK counts at least 1 n-gram per sentence in each denominator, so the one-word sentence adds 1 to the n = 2, 3, 4 denominators.
p_exact = [m / t for m, t in zip(match, total)]
p_nltk = [match[0] / total[0]] + [match[i] / (total[i] + 1) for i in (1, 2, 3)]
for name, ps in (("exact pooled counts", p_exact), ("NLTK denominators", p_nltk)):
    gm = math.exp(sum(math.log(x) for x in ps) / 4)
    print(f"{name}: precisions {[round(x, 4) for x in ps]}, geometric mean {gm:.4f}, BLEU {bp * gm:.4f}")
print("nltk corpus_bleu:", round(corpus_bleu([[x] for x in refs], hyps), 4))

Output:

matches/total for n=1..4: 7/7, 6/6, 5/5, 4/4
matches/total for n=1..4: 8/9, 6/8, 4/7, 3/6
matches/total for n=1..4: 0/1, 0/0, 0/0, 0/0
pooled matches/total: 15/17, 12/14, 9/12, 7/10
hypothesis length 17, reference length 17, brevity penalty 1.000
exact pooled counts: precisions [0.8824, 0.8571, 0.75, 0.7], geometric mean 0.7938, BLEU 0.7938
NLTK denominators: precisions [0.8824, 0.8, 0.6923, 0.6364], geometric mean 0.7468, BLEU 0.7468
nltk corpus_bleu: 0.7468

Reading it: sentence 1 matches everything (7/7, 6/6, 5/5, 4/4), so its BLEU is 1.0. Sentence 2 matches 8/9, 6/8, 4/7 and 3/6, whose geometric mean is the 0.661 shown above. Sentence 3 shares no word, so its unigram precision is 0 and the score is 0. Pooling the counts gives 15/17, 12/14, 9/12 and 7/10, and since the hypotheses and references both have 17 words the brevity penalty is 1. NLTK counts at least one n-gram in each sentence's denominator, and the one-word sentence has no 2-, 3- or 4-grams, so those denominators become 15, 13 and 11 instead of 14, 12 and 10. That gives precisions 0.8824, 0.8, 0.6923 and 0.6364, a geometric mean of 0.7468, and the 0.747 above. Another implementation without that convention would print 0.7938 on the same text, which is one more reason to report the tool and its settings.

When corpus-level versus sentence-level scoring matters

• Use corpus level for BLEU and for any ratio of counts: the numerator and denominator are summed over the test set, which is stable and is what published BLEU numbers mean.
• Use sentence level to find the worst outputs for error analysis, or when a metric is a per-example score by design (an exact match, a per-example F1 that you then average).
• Averaging per-example F1 gives a different number from pooling counts and then computing F1 (macro versus micro in disguise). Say which one is reported.

Pitfalls when interpreting the numbers

1. Accuracy on skewed classes hides a missed minority class (0.95 accuracy against 0.487 macro-F1 above).
2. Token-level versus entity-level tagging scores differ a lot; quote the entity-level one and the matching rule (exact versus overlap).
3. BLEU depends on tokenization and references. Scores from different tokenizers or a different number of references are not comparable; report with a standardized tool and its settings (for example SacreBLEU, a library that fixes the tokenization and settings so scores are comparable) and do not compare to numbers from other setups.
4. ROUGE rewards overlap, not truth. A fluent summary with an invented claim can score well, so add a faithfulness check or human review.
5. Perplexity is not comparable across tokenizers or vocabularies, as the 7.389 versus 2.226 above shows.
6. One test set with no uncertainty. A difference of a point can be noise; resample the test set (bootstrap) to get a confidence interval, and check the test data is not in the training data.

Follow-through for production

Pair the offline metric with an online one that matches the product: for example rate of corrections by users for an extractor, or the share of tickets routed correctly for an intent classifier, plus the latency and cost numbers of the chosen model family.

Direct answer

Start with a classical model: TF-IDF features over word and character n-grams (short runs of characters, which tolerate typos) feeding a logistic regression, trained with class weights for the rare intents. It is fast enough to meet almost any latency budget, strong on short messages, and a baseline you must beat anyway. Move to a fine-tuned transformer (a pretrained encoder such as BERT adapted to your labels) only if error analysis shows confusions that depend on meaning rather than surface words and the transformer's measured p95 latency (95th percentile: 95 of 100 requests are faster) fits the budget on the serving hardware. A common shipped shape is both: the linear model answers when it is confident and the transformer handles the uncertain rest.

Noisy text and typos

• Normalize lightly: Unicode normalization (NFKC, which maps look-alike forms such as full-width letters and ligatures to their plain equivalents), lowercase, collapse repeated characters ("pleeease" to "pleease"), replace digit strings such as order ids with a placeholder that keeps only their shape (for example order 48213 and order 77120 both become order <DDDDD>), so they neither flood the vocabulary nor vanish. Avoid aggressive spell-correction, which rewrites product names and ids.
• Make the features typo-tolerant: character n-grams share most of their pieces between "password" and "pasword", while a word feature for "pasword" is a different, probably unseen, vocabulary entry. Subword tokenization in a transformer gives a similar effect.
• Train on noise you expect: augment training messages with typos generated the way your users make them (dropped and doubled letters, keyboard neighbours), and test on a typo-heavy set.

Labels

• Start from what exists (agent-chosen ticket categories), then treat it as noisy: those labels were chosen for routing, not for modelling.
• Design the taxonomy so intents are mutually exclusive, add "other" and "needs a human" classes, and write a definition and examples for each.
• Label a sample twice, measure agreement, fix ambiguous definitions, then use cross-validated errors to find mislabeled examples: train on most of the data, predict the held-out part, repeat until every message has been predicted by a model that did not see it, and send the messages where that prediction confidently disagrees with the given label to a person for review.
• For rare intents do not wait for random sampling: search for likely examples (keywords, similar messages to a seed example) and label those; keep a separate random set labeled for unbiased evaluation.

Skewed intent frequencies

Report macro-F1 (F1 per intent averaged with equal weight) and per-intent recall, not accuracy, because a model that ignores the rare intent still scores high accuracy. During training use class weights (class_weight="balanced" in scikit-learn scales each class's loss by the inverse of its frequency) or oversampling (repeating examples of the rare intents so the model sees them as often as the common ones). At decision time tune the threshold per intent on validation data against the cost of the miss, and send low-confidence messages to a person or a fallback. Evaluate on both the natural distribution (what production will see) and a balanced set (to expose weak classes).

A run: typos and skew

The messages below are generated in the code from 16 template phrases over 4 intents with a 400/300/250/25 training distribution (a stand-in for real messages), so absolute scores are far higher than real data would give; the comparison between rows is what the run shows. The model is scikit-learn 1.5.2 logistic regression. Each row is evaluated on the same ten balanced test sets (100 messages per intent) with typos injected: every letter has a one-in-fifteen chance of being dropped or a one-in-fifteen chance of being doubled, so roughly 13% of letters are damaged in total (the first printed line checks the two rates). No transformer is trained in this run, so the comparison with a transformer rests on counted operations and on the decision rule in the last sections, not on a measured accuracy gap.

import random
import numpy as np
import torch
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, recall_score
from sklearn.pipeline import make_pipeline

torch.set_num_threads(1)

# Toy support intents with a skewed distribution (stand-in for real labeled messages).
phrases = {
    "reset_password": ["i forgot my password", "cannot log in to my account", "reset my password please", "password reset link not working"],
    "billing":        ["why was i charged twice", "i need a refund for my invoice", "update my payment method", "question about my bill"],
    "cancel":         ["i want to cancel my subscription", "please close my account", "how do i cancel", "stop my plan"],
    "shipping":       ["where is my order", "track my package", "my delivery is late", "order has not arrived"],
}
counts = {"reset_password": 400, "billing": 300, "shipping": 250, "cancel": 25}   # cancel is rare

def typo(s, rng, rate=0.2, tally=None):
    out = []
    for ch in s:
        r = rng.random()
        if tally is not None and ch != " ":
            tally["letters"] += 1
        if ch != " " and r < rate / 3:
            if tally is not None:
                tally["dropped"] += 1
            continue                                   # drop a letter
        out.append(ch)
        if ch != " " and rate / 3 <= r < 2 * rate / 3:
            out.append(ch)                             # double a letter
            if tally is not None:
                tally["doubled"] += 1
    return "".join(out)

def make(n_per, noisy, rng):
    X, y = [], []
    for lab, n in n_per.items():
        for _ in range(n):
            s = rng.choice(phrases[lab])
            X.append(typo(s, rng) if noisy else s)
            y.append(lab)
    return X, y

rng0 = random.Random(0)
Xtr, ytr = make(counts, False, rng0)
Xaug, yaug = make(counts, True, rng0)              # typo-augmented copy of the same label distribution
tests = [make({k: 100 for k in phrases}, True, random.Random(100 + s)) for s in range(10)]   # 10 typo-heavy balanced test sets

def run(name, analyzer, ngram, class_weight, augment):
    X = Xtr + Xaug if augment else Xtr
    y = ytr + yaug if augment else ytr
    clf = make_pipeline(TfidfVectorizer(analyzer=analyzer, ngram_range=ngram),
                        LogisticRegression(max_iter=1000, class_weight=class_weight)).fit(X, y)
    f1s, rec = [], []
    for Xte, yte in tests:
        pred = clf.predict(Xte)
        f1s.append(f1_score(yte, pred, average="macro"))
        rec.append(recall_score(yte, pred, labels=["cancel"], average=None)[0])
    print(f"{name:40s} macro-F1 mean={np.mean(f1s):.3f} min={np.min(f1s):.3f}  recall(cancel) mean={np.mean(rec):.3f}")

tally = {"letters": 0, "dropped": 0, "doubled": 0}
typo(" ".join(p for v in phrases.values() for p in v) * 200, random.Random(7), tally=tally)
print(f"corruption check: {tally['dropped'] / tally['letters']:.3f} of letters dropped, {tally['doubled'] / tally['letters']:.3f} doubled")

run("word n-grams, plain loss", "word", (1, 1), None, False)
run("word n-grams, class_weight=balanced", "word", (1, 1), "balanced", False)
run("word n-grams, balanced + typo-augmented", "word", (1, 1), "balanced", True)
run("char 2-4-grams, balanced", "char_wb", (2, 4), "balanced", False)

# Confidence routing: the linear model answers when sure, the rest would go to a heavier model (not trained here).
clf = make_pipeline(TfidfVectorizer(analyzer="word"), LogisticRegression(max_iter=1000, class_weight="balanced")).fit(Xtr + Xaug, ytr + yaug)
Xte, yte = tests[0]
proba = clf.predict_proba(Xte)
conf = proba.max(axis=1)
pred = clf.classes_[proba.argmax(axis=1)]
hit = pred == np.array(yte)
for thr in (0.5, 0.7):
    sure = conf >= thr
    print(f"threshold {thr}: linear model answers {sure.mean():.3f} of messages, accuracy on those {hit[sure].mean():.3f}; forwarded {1 - sure.mean():.3f}, linear accuracy on those {hit[~sure].mean():.3f}")
k = int(np.argmin(conf))
print("least confident message:", repr(Xte[k]), "| true", yte[k], "| predicted", pred[k], "| confidence", round(float(conf[k]), 2))

# Latency by counted operations, not wall-clock time.
d_model, d_ff = 384, 1536
layer = torch.nn.TransformerEncoderLayer(d_model=d_model, nhead=6, dim_feedforward=d_ff, batch_first=True)
per_layer = sum(p.numel() for p in layer.parameters())
attention = 4 * d_model * d_model + 4 * d_model        # query, key, value, output projections with biases
feed_forward = 2 * d_model * d_ff + d_ff + d_model     # expand to 1536 and project back, with biases
norms = 2 * 2 * d_model                                # two layer norms, each a scale and a shift
print("per layer:", attention, "attention +", feed_forward, "feed-forward +", norms, "norms =", attention + feed_forward + norms, "| torch counts", per_layer)
print("6-layer encoder body (d=384) parameters =", 6 * per_layer)
print("approx multiply-adds for a 16-token message =", 16 * 6 * per_layer)
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)).fit(Xtr)
nnz = vec.transform(["i forgot my password"]).nnz
print("char-ngram linear model parameters =", len(vec.vocabulary_) * 4 + 4)
print("linear multiply-adds for 'i forgot my password' =", nnz, "non-zero features x 4 classes =", nnz * 4)
print("transformer / linear multiply-add ratio =", round(16 * 6 * per_layer / (nnz * 4)))

Output:

corruption check: 0.065 of letters dropped, 0.067 doubled
word n-grams, plain loss                 macro-F1 mean=0.872 min=0.843  recall(cancel) mean=0.738
word n-grams, class_weight=balanced      macro-F1 mean=0.882 min=0.848  recall(cancel) mean=0.779
word n-grams, balanced + typo-augmented  macro-F1 mean=0.960 min=0.949  recall(cancel) mean=0.856
char 2-4-grams, balanced                 macro-F1 mean=1.000 min=0.997  recall(cancel) mean=0.999
threshold 0.5: linear model answers 0.885 of messages, accuracy on those 0.992; forwarded 0.115, linear accuracy on those 0.652
threshold 0.7: linear model answers 0.733 of messages, accuracy on those 0.993; forwarded 0.267, linear accuracy on those 0.841
least confident message: 'pleasse clos yy accoount' | true cancel | predicted reset_password | confidence 0.3
per layer: 591360 attention + 1181568 feed-forward + 1536 norms = 1774464 | torch counts 1774464
6-layer encoder body (d=384) parameters = 10646784
approx multiply-adds for a 16-token message = 170348544
char-ngram linear model parameters = 2028
linear multiply-adds for 'i forgot my password' = 50 non-zero features x 4 classes = 200
transformer / linear multiply-add ratio = 851743

• Class weights raise the recall of the rare cancel intent from 0.738 to 0.779, and macro-F1 from 0.872 to 0.882.
• Routing by confidence, shown on the first of the ten test sets with the typo-augmented word model: at a top-probability threshold of 0.5 the linear model answers 88.5% of messages with 0.992 accuracy, and the 11.5% it is unsure of would be forwarded to the heavier model or a person; the linear model's own accuracy on that forwarded slice is only 0.652, which is why forwarding it is worth the cost. At 0.7 it answers 73.3% (accuracy 0.993) and forwards 26.7%. The least confident message in the set, pleasse clos yy accoount, is a cancel request the model assigns to the wrong intent with confidence 0.3. The right threshold is where the forwarded share fits the heavy model's capacity and latency budget; the heavy model itself is not trained here.
• Typo-augmented training lifts word features to 0.960 macro-F1 and 0.856 cancel recall, because the model has now seen corrupted spellings. The training copy and the test sets here are corrupted by the same function, so this gain is an upper estimate: against real misspellings or a different noise source, measure it again before trusting it.
• Character n-grams reach 0.999 cancel recall and 1.000 mean macro-F1 with no augmentation, which is partly an artefact of the tiny phrase set. Real data have far more phrasing variety, so expect a smaller gap and measure it on your own test set.
• The worst of the ten test sets (the min column) tracks the mean closely for every row, so the ordering is not an accident of one noisy test set.

Latency: counted, not timed

A multiply-add is one multiplication plus one addition, the basic unit of work in a model. A weight matrix with $p$ weights costs $p$ multiply-adds for every token that passes through it, which is why a 16-token message costs about 16 times the parameter count. The ratio compares two unlike things (a sparse linear model and a dense encoder), leaves out the cost of building the TF-IDF features from the text (a pass over the message that grows with its length), and is an order of magnitude, not a prediction of milliseconds. The linear model costs one multiply-add per non-zero feature per class: for "i forgot my password" that is 50 non-zero character n-gram features times 4 classes, 200 multiply-adds, and the model holds 2,028 parameters here. The example transformer body (6 layers, hidden size 384) has 10,646,784 parameters. Each layer holds the attention projections (query, key, value and output matrices), a feed-forward block (expand to 1,536 units and project back) and two layer norms, which the run adds up to the same per-layer count that torch reports; six layers give the total; a 16-token message costs about 16 times that, 170,348,544 multiply-adds (ignoring the attention-score term and embeddings), which is about 851,743 times the linear model's work on the same kind of message. A real linear model has more features than this toy, so the ratio will differ. Whether that is acceptable is decided by timing both on the real serving hardware at the real batch size and comparing p95 to the budget. A distilled encoder (a smaller model trained to imitate a larger one) or quantization (storing the weights in fewer bits, such as 8-bit integers) reduces the gap but not to zero.

Choosing, and what flips it

Switch from the baseline when error analysis shows that most remaining errors are two intents with similar words but different meanings, and the transformer fixes them on a held-out set.

Pitfalls

• Reporting accuracy on skewed data; tuning thresholds on the test set; trusting augmented-typo scores that use the same corruption function for training and test (use a different noise source in the test, or real misspellings).
• Treating the model's confidence as a probability without checking calibration (do predicted 0.9 confidences come out right about 90% of the time?); a routing threshold is only as good as that check.