Natural Language Processing Questions

Direct answer

A WordPiece tokenizer (a subword scheme that splits a word into pieces from a fixed vocabulary, marking every piece after the first with ##) usually normalizes the text before it splits it. Normalizing means rewriting characters into a standard form; the form used below is Unicode NFKD, which decomposes each character into a base character plus combining marks (separate accent characters that attach to the letter before them) and also replaces compatibility forms such as the ligature fi (two letters drawn as one glyph) with plain letters. So a token's text is a slice of the normalized string, not of your original string. The robust way to recover original character spans is to normalize with an offset map: for every character of the normalized string, remember which original character(s) produced it. Then walk the token list left to right with a cursor into the normalized string, find each token there, and translate its normalized start and end into original start and end through the map. Entity labels predicted per token are then lifted to character spans by merging ## pieces into words, reading the label of each word's first piece, and joining B-/I- runs. NER here means named entity recognition (tagging spans such as persons, locations, organizations), and BIO tagging marks the Beginning of an entity, the Inside of one, or Outside any entity.

The mechanism, in order

1. Normalize with offsets. Each original character goes through the same steps the tokenizer applies (here: decompose with Unicode NFKD, drop combining marks, lowercase, turn any whitespace run into one space). Each output character records starts[j] and ends[j], the half-open range of the original string it came from (start is included and end is not, as in Python slicing: text[0:4] covers indices 0, 1, 2 and 3, so a span of 0-4 has length 4). Positions count code points, the individual entries of the Unicode table; a letter with a separate accent is two code points. One original character can produce several output characters (the ligature fi becomes fi: both map to the same original index), and a dropped combining mark is folded into the span of the character before it, so Zoé typed as e plus a separate accent still gets a span that covers the accent.
2. Walk the tokens with a cursor. For a token that does not start with ##, skip spaces, then require the normalized text to start with the token at the cursor. For a ## piece, the match must begin exactly at the cursor (no skipping). Anything else means the token list does not belong to this text, so raise instead of guessing.
3. Special tokens. [UNK] (the placeholder for a word the vocabulary cannot build) has no text, so it takes the whole pre-tokenized unit at the cursor. A pre-tokenized unit is what the first splitting step produces before any piece lookup: here a run of word characters, or a single punctuation or emoji character.
4. Translate. The token covers normalized positions cur .. cur+n-1, so its original span is (starts[cur], ends[cur+n-1]).

A small example first, by hand, then the full program. Take Ze followed by a separate accent U+0301, a space, the ligature U+FB01 and x: original indices 0 to 5. Normalization produces ze fix (6 characters). The accent is dropped but folded into the span of the e before it, so e maps to original[1:3]. The ligature yields two output characters, f and i, that both map to original[4:5]. For tokens ze, fi, ##x the cursor starts at 0: ze covers normalized 0..1, so its span is (starts[0], ends[1]) = (0, 3). The cursor moves to 2, which is a space, so for fi (not a continuation) it skips to 3 and covers 3..4, giving (4, 5). ##x must start exactly at the cursor, 5, and gives (5, 6). The first lines of the program output print exactly these steps.

The tokenizer in the full program is a toy greedy longest-match WordPiece with a 15-entry vocabulary, a stand-in for a real one; the alignment function only sees the original string and the token strings. In practice many tokenizer libraries can return offsets directly, and you should use that when it is available; the function below is what you write when you must reconstruct them, or when you want to understand what such a field contains.

import re
import unicodedata

def normalize_with_offsets(text):
    """Return (normalized, starts, ends): normalized[j] came from text[starts[j]:ends[j]]."""
    out, starts, ends = [], [], []
    for i, ch in enumerate(text):
        if ch.isspace():
            if out and out[-1] != " ":
                out.append(" "); starts.append(i); ends.append(i + 1)
            continue
        produced = False
        for d in unicodedata.normalize("NFKD", ch):
            if unicodedata.category(d) == "Mn":
                continue
            for low in d.lower():
                out.append(low); starts.append(i); ends.append(i + 1); produced = True
        if not produced and ends:
            ends[-1] = i + 1  # a dropped combining mark stays inside the previous span
    if out and out[-1] == " ":
        out.pop(); starts.pop(); ends.pop()
    return "".join(out), starts, ends

PRE = re.compile(r"\w+|[^\w\s]")

VOCAB = {"[UNK]", "zoe", "flew", "to", "istanbul", "with", "nv", "##id", "##ia",
         "final", "'", "s", "-", "cafe", "bar"}

def wordpiece(norm):
    """Toy greedy longest-match WordPiece over the normalized text. Returns tokens and reference spans."""
    toks, spans = [], []
    for m in PRE.finditer(norm):
        w, start, pieces = m.group(), 0, []
        while start < len(w):
            end = len(w)
            while end > start:
                p = ("##" if start else "") + w[start:end]
                if p in VOCAB:
                    break
                end -= 1
            if end == start:
                pieces = None; break
            pieces.append((p, m.start() + start, m.start() + end)); start = end
        if pieces is None:
            toks.append("[UNK]"); spans.append((m.start(), m.end()))
        else:
            for p, a, b in pieces:
                toks.append(p); spans.append((a, b))
    return toks, spans

def align(original, tokens, verbose=False):
    """Map each token to a (start, end) span in `original` (end exclusive)."""
    norm, starts, ends = normalize_with_offsets(original)
    cur, result = 0, []
    for tok in tokens:
        if tok == "[UNK]":
            while norm[cur] == " ":
                cur += 1
            n = len(PRE.match(norm, cur).group())
        else:
            cont = tok.startswith("##")
            piece = tok[2:] if cont else tok
            if not cont:
                while norm[cur] == " ":
                    cur += 1
            if not norm.startswith(piece, cur):
                raise ValueError(f"token {tok!r} does not match text at {cur}: {norm[cur:cur+8]!r}")
            n = len(piece)
        result.append((starts[cur], ends[cur + n - 1]))
        if verbose:
            print(f"  token {tok!r}: cursor {cur}, covers normalized {cur}..{cur + n - 1}, original span {result[-1]}")
        cur += n
    return result

def entities(original, tokens, spans, labels):
    """Word = first token plus following '##' tokens; the word takes its first piece's label."""
    words = []
    for t, (a, b), lab in zip(tokens, spans, labels):
        if t.startswith("##") and words:
            words[-1][1] = b
        else:
            words.append([a, b, lab])
    ents, cur = [], None
    for a, b, lab in words:
        if lab.startswith("B-") or (lab.startswith("I-") and (cur is None or cur[2] != lab[2:])):
            cur = [a, b, lab[2:]]; ents.append(cur)
        elif lab.startswith("I-"):
            cur[1] = b
        else:
            cur = None
    return [(original[a:b], t, a, b) for a, b, t in ents]

def utf16_offset(text, i):
    return len(text[:i].encode("utf-16-le")) // 2

# A tiny hand-traceable case: 'e' followed by a separate accent (U+0301), a space, the ligature U+FB01, then 'x'.
tiny = "Ze\u0301 \ufb01x"
n2, s2, e2 = normalize_with_offsets(tiny)
print("tiny original  :", ascii(tiny), "has", len(tiny), "characters, indices 0..5")
for j, ch in enumerate(n2):
    print(f"  normalized[{j}] = {ch!r} came from original[{s2[j]}:{e2[j]}] = {ascii(tiny[s2[j]:e2[j]])}")
print("tiny spans:")
print(align(tiny, ["ze", "fi", "##x"], verbose=True))

text = "Zoé flew  to İstanbul \U0001F600 with Nvidia final"
norm, starts, ends = normalize_with_offsets(text)
tokens, ref_spans = wordpiece(norm)
spans = align(text, tokens)
print("normalized:", repr(norm))
print("tokens:", tokens)
for t, (a, b) in zip(tokens, spans):
    print(f"{t:10s} {a:2d}-{b:2d} {text[a:b]!r}")

# Reference check: spans in normalized space, mapped through idx by the tokenizer itself.
ref = [(starts[a], ends[b - 1]) for a, b in ref_spans]
print("matches tokenizer's own spans:", ref == spans)
# Robustness check: a token list that does not belong to this text must be rejected.
try:
    align(text, ["zoe", "flew", "from"])
except ValueError as e:
    print("rejected a token list that does not belong to the text")

# Illustrative per-token predictions (hand-set stand-in for a model); only the first piece of a word is read.
labels = ["B-PER", "O", "O", "B-LOC", "O", "O", "B-ORG", "O", "O", "O"]
for e in entities(text, tokens, spans, labels):
    print(e, "utf16 start:", utf16_offset(text, e[2]))

Output:

tiny original  : 'Ze\u0301 \ufb01x' has 6 characters, indices 0..5
  normalized[0] = 'z' came from original[0:1] = 'Z'
  normalized[1] = 'e' came from original[1:3] = 'e\u0301'
  normalized[2] = ' ' came from original[3:4] = ' '
  normalized[3] = 'f' came from original[4:5] = '\ufb01'
  normalized[4] = 'i' came from original[4:5] = '\ufb01'
  normalized[5] = 'x' came from original[5:6] = 'x'
tiny spans:
  token 'ze': cursor 0, covers normalized 0..1, original span (0, 3)
  token 'fi': cursor 3, covers normalized 3..4, original span (4, 5)
  token '##x': cursor 5, covers normalized 5..5, original span (5, 6)
[(0, 3), (4, 5), (5, 6)]
normalized: 'zoe flew to istanbul 😀 with nvidia final'
tokens: ['zoe', 'flew', 'to', 'istanbul', '[UNK]', 'with', 'nv', '##id', '##ia', 'final']
zoe         0- 4 'Zoé'
flew        5- 9 'flew'
to         11-13 'to'
istanbul   14-22 'İstanbul'
[UNK]      23-24 '😀'
with       25-29 'with'
nv         30-32 'Nv'
##id       32-34 'id'
##ia       34-36 'ia'
final      37-41 'final'
matches tokenizer's own spans: True
rejected a token list that does not belong to the text
('Zoé', 'PER', 0, 4) utf16 start: 0
('İstanbul', 'LOC', 14, 22) utf16 start: 14
('Nvidia', 'ORG', 30, 36) utf16 start: 31

Reading the output: Zoe plus its separate accent gets span 0-4 and the substring Zoé; the double space and the dotted capital İ (which lowercases to more than one character when lowercased naively) are handled by the offset map; the emoji is [UNK] and still gets exactly its own span; Nvidia is three pieces (nv, ##id, ##ia) with contiguous spans 30-32, 32-34, 34-36; the ligature final is four original characters but its normalized text final is five, and the span still ends at the right original index. The comparison against the tokenizer's own spans prints True, and the deliberately wrong token list is rejected.

From token predictions to entity spans

A model predicts one label per token, but training usually labels only the first piece of each word (the other pieces are ignored in the loss), so at inference you read the first piece of each word:

1. Group tokens into words: a token starting with ## extends the previous word's end.
2. A word takes the label of its first piece.
3. A B-X starts an entity; an I-X continues the open entity of the same type; an I-X with no matching open entity is treated as a new entity (a common repair); O closes the open entity.
4. The entity's character span is the start of its first word and the end of its last word, and its text is original[start:end].

In the run, the hand-set labels (an illustrative stand-in for model output) give Zoé as PER at 0-4, İstanbul as LOC at 14-22 and Nvidia as ORG at 30-36. Because the substring is taken from the original text, the user sees İstanbul with its original capital and dot, not the lowercased token.

Normalization pitfalls

Complexity and edge cases

Normalization is one pass, and the walk is one pass over the tokens with a cursor that never moves backward, so the whole alignment is linear in the length of the text plus the number of tokens, with two integer arrays of the normalized length as extra memory. Edge cases to cover in tests: empty string and empty token list, text with leading or trailing whitespace, a token list containing [UNK], text that ends right after a ## piece, and a mismatch (which must raise, as the run shows). Do not use str.find(token) from the start of the string: a repeated word such as the second to in "to be or not to be" would map to the first occurrence. The cursor is what makes repeated tokens map correctly.

Running the code

python 3.12, standard library only
python align_wordpiece.py

Direct answer

A Transformer encoder (a network that reads all tokens of a text at once and outputs one vector per token, shape tokens by hidden size) gives the classifier a sequence of vectors, but a classifier needs one vector per document. Three standard ways to get it: take the vector of a special first token (CLS, a marker token the tokenizer adds at the start of every text), average the token vectors, or take the element-wise maximum. A fourth, learned attention pooling, weights tokens by a small trainable scorer. When the label depends on the overall content spread through the text (topic, tone), default to masked mean pooling as lengths vary and text gets noisier, because its scale does not depend on how many padding tokens share the batch. When the label is triggered by one short passage hidden in a long text, a mean dilutes that passage as the text grows (the worked example shows exactly this), so switch to max or attention pooling, or to chunk-then-aggregate. For documents longer than the model's window, split into overlapping chunks, classify or embed each chunk, and aggregate the chunk results with a rule chosen from what the label means (mean for overall tone, max for "any chunk contains it").

The pooling options

Notation: for one document, $H$ is the matrix of final-layer token vectors with $n$ real tokens, so $H$ has shape $n \times d$ where $d$ is the hidden size; $m_i$ is 1 for a real token and 0 for padding (padding is the filler added so that all texts in a batch have equal length).

Attention pooling in symbols: with a trainable scorer vector $w$ of size $d$, the score of token $i$ is $h_i \cdot w$, the weights are $a_i = \text{softmax}(h_i \cdot w)$ taken over the real tokens only, and the pooled vector is $\sum_i a_i h_i$. For three tokens $h_1 = (1, 0)$, $h_2 = (0, 1)$, $h_3 = (4, 0)$ and $w = (1, 0)$, the scores are 1, 0, 4, the weights are 0.0466, 0.0171, 0.9362, and the pooled vector is $(3.79, 0.02)$, close to the strong token, whereas the plain mean is $(1.67, 0.33)$. The program prints this example.

The masked mean is

where $L$ is the padded length, $h_i$ is the vector of token $i$ and $m_i$ the mask above. The mask appears twice: in the numerator so padding vectors add nothing, and in the denominator so the divisor is the real token count and not $L$. For max pooling the padded positions are set to negative infinity before taking the maximum, so they can never win.

Worked example

Real encoder outputs need a trained model, so the program builds a stand-in: each token vector has 4 numbers of Gaussian noise (mean 0, standard deviation 1), and "evidence" tokens get +3.0 added to coordinate 0, a crude model of a trained encoder where tokens that matter for the label stand out along some direction. Only coordinate 0 is printed. The numbers show how each pooling rule behaves when length, padding and noise change; they are not measurements on a trained model. A first-token vector is not shown because what it holds is created by fine-tuning; on random stand-in data it is just one more token. The masked mean is checked against an independent PyTorch computation.

How to read the helpers: make_doc returns an array of n_tokens rows and 4 columns and adds the +3.0 to a random choice of n_signal rows; pad appends zero rows up to length L and returns a mask with 1 for real rows; mean_pool multiplies each row by its mask value and divides the column sums by the real count; max_pool replaces padded rows by minus infinity so they can never win; attn_pool scores each row with the scorer vector, applies the softmax over real rows only, and returns the weighted sum.

import numpy as np
import torch
torch.set_num_threads(1)

rng = np.random.default_rng(0)
d = 4
signal = np.array([3.0, 0.0, 0.0, 0.0])

def make_doc(n_tokens, n_signal, noise_spike=0.0):
    """Hand-built stand-in for encoder outputs: n_tokens vectors of size d, shape [n_tokens, d]."""
    H = rng.normal(0, 1.0, size=(n_tokens, d))
    if n_signal:
        idx = rng.choice(n_tokens, size=n_signal, replace=False)
        H[idx] += signal
    if noise_spike:
        H[rng.integers(n_tokens), 0] += noise_spike
    return H

def pad(H, L):
    out = np.zeros((L, d)); out[:len(H)] = H
    mask = np.zeros(L); mask[:len(H)] = 1
    return out, mask

def mean_pool(H, mask):
    return (H * mask[:, None]).sum(0) / mask.sum()

def max_pool(H, mask):
    return np.where(mask[:, None] == 1, H, -np.inf).max(0)

H10 = make_doc(10, 3)
for L in (10, 64):
    Hp, m = pad(H10, L)
    print(f"padded to {L:2d}: masked mean[0]={mean_pool(Hp, m)[0]:.3f}  unmasked mean[0]={Hp.mean(0)[0]:.3f}  masked max[0]={max_pool(Hp, m)[0]:.3f}")
Hp, m = pad(H10, 64)
t, tm = torch.tensor(Hp), torch.tensor(m)
ref = (t * tm[:, None]).sum(0) / tm.sum()
print("masked mean matches torch:", bool(np.allclose(mean_pool(Hp, m), ref.numpy())))

print("3 evidence tokens, average of 200 documents per length")
for n in (16, 128, 512):
    ms, xs = [], []
    for _ in range(200):
        H = make_doc(n, 3); m1 = np.ones(n)
        ms.append(mean_pool(H, m1)[0]); xs.append(max_pool(H, m1)[0])
    print(f"length {n:3d}: mean[0]={np.mean(ms):.3f}  max[0]={np.mean(xs):.3f}")

print("no evidence, one junk token with a spike of 8.0, average of 200 documents per length")
for n in (16, 512):
    ms, xs = [], []
    for _ in range(200):
        H = make_doc(n, 0, noise_spike=8.0); m1 = np.ones(n)
        ms.append(mean_pool(H, m1)[0]); xs.append(max_pool(H, m1)[0])
    print(f"length {n:3d}: mean[0]={np.mean(ms):.3f}  max[0]={np.mean(xs):.3f}")

def chunks(n, content=510, overlap=128):
    stride = content - overlap
    out, s = [], 0
    while True:
        out.append((s, min(s + content, n)))
        if s + content >= n: break
        s += stride
    return out
print("chunks for 2000 tokens, 510 content tokens, overlap 128:", len(chunks(2000)), "| no overlap:", len(chunks(2000, 510, 0)))
print("windows:", chunks(2000))

# ---- Dense evidence, attention pooling, noisy-OR ----
def attn_pool(H, mask, w):
    """Attention pooling with a scorer vector w: score each token, softmax over real tokens, weighted sum."""
    s = np.where(mask == 1, H @ w, -np.inf)
    a = np.exp(s - s.max()); a = a / a.sum()
    return a @ H

print("dense evidence: a quarter of the tokens carry the +3.0, average of 200 documents per length")
for n in (16, 128, 512):
    ms, xs = [], []
    for _ in range(200):
        H = make_doc(n, n // 4); m1 = np.ones(n)
        ms.append(mean_pool(H, m1)[0]); xs.append(max_pool(H, m1)[0])
    print(f"length {n:3d}: mean[0]={np.mean(ms):.3f}  max[0]={np.mean(xs):.3f}")

H3 = np.array([[1.0, 0.0], [0.0, 1.0], [4.0, 0.0]])
sc = H3 @ np.array([1.0, 0.0])
wt = np.exp(sc) / np.exp(sc).sum()
print("3-token attention example: scores", sc, "weights", np.round(wt, 4), "pooled", np.round(wt @ H3, 4), "plain mean", np.round(H3.mean(0), 4))

w_att = np.array([2.0, 0.0, 0.0, 0.0])   # hand-set scorer; training would learn a vector like this
print("3 evidence tokens, attention pooling with the hand-set scorer, average of 200 documents per length")
for n in (16, 128, 512):
    ats = []
    for _ in range(200):
        H = make_doc(n, 3); m1 = np.ones(n)
        ats.append(attn_pool(H, m1, w_att)[0])
    print(f"length {n:3d}: attention[0]={np.mean(ats):.3f}")

p_chunks = np.array([0.1, 0.2, 0.7])
print("chunk probabilities", p_chunks, "| mean", round(p_chunks.mean(), 4), "| max", p_chunks.max(), "| noisy-OR", round(1 - np.prod(1 - p_chunks), 4))

Output:

padded to 10: masked mean[0]=0.452  unmasked mean[0]=0.452  masked max[0]=2.871
padded to 64: masked mean[0]=0.452  unmasked mean[0]=0.071  masked max[0]=2.871
masked mean matches torch: True
3 evidence tokens, average of 200 documents per length
length  16: mean[0]=0.589  max[0]=3.977
length 128: mean[0]=0.075  max[0]=3.874
length 512: mean[0]=0.016  max[0]=3.936
no evidence, one junk token with a spike of 8.0, average of 200 documents per length
length  16: mean[0]=0.496  max[0]=8.009
length 512: mean[0]=0.019  max[0]=7.907
chunks for 2000 tokens, 510 content tokens, overlap 128: 5 | no overlap: 4
windows: [(0, 510), (382, 892), (764, 1274), (1146, 1656), (1528, 2000)]
dense evidence: a quarter of the tokens carry the +3.0, average of 200 documents per length
length  16: mean[0]=0.763  max[0]=4.061
length 128: mean[0]=0.753  max[0]=5.088
length 512: mean[0]=0.747  max[0]=5.550
3-token attention example: scores [1. 0. 4.] weights [0.0466 0.0171 0.9362] pooled [3.7916 0.0171] plain mean [1.6667 0.3333]
3 evidence tokens, attention pooling with the hand-set scorer, average of 200 documents per length
length  16: attention[0]=3.629
length 128: attention[0]=3.385
length 512: attention[0]=2.927
chunk probabilities [0.1 0.2 0.7] | mean 0.3333 | max 0.7 | noisy-OR 0.784

Reading the output:

1. Padding. The same 10-token document gives a masked mean of 0.452 whether it is padded to 10 or to 64. Averaging over all 64 rows (a common bug) gives 0.071, which is 0.452 times 10/64: the vector's scale now depends on what else is in the batch. The masked max is unaffected (2.871 both times). This is also why the mask must be passed to the pooling layer, not only to the encoder.
2. Length variance. With 3 evidence tokens, the expected mean on coordinate 0 is 3 tokens times 3.0 divided by n, which is 0.5625 at n = 16, about 0.07 at 128 and about 0.018 at 512; the printed averages (0.589, 0.075, 0.016) follow that shape. The same evidence yields a signal about 37 times weaker at 512 tokens than at 16 (0.589 against 0.016), so a classifier head trained mostly on short documents sees long documents as near-neutral. Max pooling stays between 3.87 and 3.98 at every length.
3. Noise. One junk token with a spike of 8.0 and no evidence at all drives the max to about 8.0 at both lengths, which is larger than the genuine evidence value (about 3.9). The mean barely moves for long documents (0.019 at 512) but is moved by a spike at 16 tokens (0.496). So mean is robust to a stray token only when the document is long, and max is never robust to one.
4. Window arithmetic. A 2,000-token document with a 512-token window, 2 positions reserved for special tokens (for BERT-style models a start marker CLS and an end marker SEP, leaving 510 content tokens) and 128 tokens of overlap needs 5 windows; without overlap it needs 4. Overlap costs one extra window here and keeps a phrase that straddles a boundary intact in at least one window.
5. Dense evidence (the bridge to the mean recommendation). In point 2 the number of evidence tokens was fixed at 3, so the longer the document, the smaller their share, and the mean shrinks. When evidence is instead a constant share of the text, here a quarter of the tokens, the mean stays near 0.75 at every length (0.763, 0.753, 0.747), which is the expected 0.25 times 3.0. That is the situation of overall-tone or topic labels, where every part of a longer text keeps carrying the label, and it is why masked mean is the default for them. The max drifts upward with length (4.06 to 5.55) because the number of evidence tokens grows with the text (a quarter of 16, 128 and 512 tokens is 4, 32 and 128 of them), and the largest of more draws centred near 3.0 is larger, so max pooling has a length-dependent scale of its own.
6. Attention pooling. With a hand-set scorer that looks at coordinate 0 (a trained scorer would learn something like it), the pooled coordinate is 3.63, 3.39 and 2.93 at lengths 16, 128 and 512 for the sparse 3-evidence case, against 0.589, 0.075 and 0.016 for the mean: attention puts a large share of the weight on the few evidence tokens, so the pooled value stays far above the mean's and fades much more slowly with length. It still drifts down as more noise tokens compete for weight, and by 512 tokens the evidence tokens no longer hold most of it. It only works if the scorer has been trained to find the evidence. The 3-token example printed above shows the mechanism: weight 0.9362 on the strong token.
7. Noisy-OR. To combine chunk probabilities when the label means "any chunk contains it", noisy-OR treats each chunk as an independent chance to fire: the document probability is $1 - \prod_k (1 - p_k)$. For chunk probabilities 0.1, 0.2 and 0.7 it is $1 - 0.9 \times 0.8 \times 0.3 = 0.784$, above the max (0.7) and far above the mean (0.3333). Many weak chunks add up under noisy-OR, so on long documents it can over-fire and its threshold should be tuned on dev data.

When the mean advice holds and when it fails. Masked mean is the right default when the label is carried by the text as a whole (point 5) and the classifier head is fine-tuned on a realistic mix of lengths, so it learns the scale it will see; a head trained on a mix of lengths can learn to read the different scales. It fails when the evidence is a few tokens in a long text (point 2): the mean of 3 evidence tokens at 512 tokens is about 0.018, while the mean of the 509 noise tokens alone fluctuates by roughly 1 divided by the square root of 512, about 0.044, so the signal sits below the noise and a classifier head can recover little of it. There, use max, attention pooling or chunk-then-max, and check the dev metric by length bucket so the failure is visible. The shrinking-mean result is therefore a warning about sparse evidence, not a contradiction of the default.

Long documents: the max-length limit and chunk-then-aggregate

Encoders with learned absolute position embeddings (a table holding one learned vector per position, up to a fixed maximum, so positions past the table do not exist) have a fixed maximum input length (for many BERT-style models 512 tokens; check the model's configuration for the value of yours). Anything longer cannot be fed in one pass. Options, in the order to try them:

1. Truncate to the first window (or the first and last parts, since introductions and conclusions often carry the label). It is the cheapest baseline: measure how much accuracy it loses on your longest bucket before building more.
2. Chunk, then aggregate. Slide a window with overlap, run the encoder on each window, pool inside each window with masked mean (or take its first-token vector), and get either a chunk embedding or a chunk logit (the raw class score before it is turned into a probability by a softmax). Combine across chunks:
   • mean of chunk logits when the label is the overall tone of the document;
   • max (or noisy-OR, see point 7 above) of chunk scores when the label means "any part contains it";
   • a learned attention pooling over chunk embeddings (the same score, softmax and weighted sum as above, with chunks in place of tokens) when you have enough labelled documents to train it.
     Training chunk classifiers on document labels adds label noise (a chunk of a positive document may be neutral); a multiple-instance setup handles that more honestly: the document is a "bag" of chunks, only the bag has a label, and the bag's score is computed from the chunk outputs (a max, a mean or noisy-OR) so the model is trained on the bag label directly.
3. A model with a longer window if budget and latency allow. Cost grows with length for any of these options, and a chunked pipeline multiplies the number of encoder passes by the number of chunks.

Always mask padding inside every chunk; the last chunk is usually shorter than the others.

What pushes toward each strategy

Validate whichever you pick by slicing the dev metric by length bucket and by noise level; one overall accuracy hides the long-document failure shown in the output above.

Common mistakes

• Pooling over padding positions, or dividing by the padded length.
• Truncating silently and never checking how many documents exceed the window.
• Max pooling on text that contains markup or boilerplate.
• Using the first-token vector from a checkpoint (the saved weights of a pretrained model) and a head (the small classifier layer on top) that were never fine-tuned together, then concluding the model is weak.
• Chunking without overlap, so phrases that straddle a boundary are cut.

Running the code

# Python 3.12, numpy 2.1.3, torch 2.5.1 (CPU)
OMP_NUM_THREADS=1 python pool.py

The idea

If every sentence embedding is scaled to length 1, the cosine similarity between a query and a sentence is just their dot product. So the whole corpus can be scored with one matrix-vector product: an [N, D] matrix of embeddings times a [D] query vector gives N scores. The top-k is then a selection problem: np.argpartition finds the k best in linear time without sorting all N scores, and only those k are sorted. At 100k sentences this is exact and fast enough; at a hundred times that, the cost and the memory both grow 100x, and the usual answers are to store the vectors in fewer bytes (quantisation) and to score only part of the corpus (an approximate index).

Code: exact top-k

Terms used in the code, in plain words. Recall@10 is the fraction of the true top 10 (found by exact search) that an approximate method also returned: 1.0 means it lost nothing, 0.9 means it missed one of ten. k-means is a clustering method that repeatedly assigns each vector to the nearest of a chosen number of centroids (a centroid is the average vector of one cluster) and moves each centroid to the average of its members. In the cell index, nlist is how many such clusters ("cells") the corpus is split into, and nprobe is how many of the nearest cells are searched for each query; searching only those cells is what saves work. MiniBatchKMeans is a faster k-means that learns from small random batches. The synthetic vectors have 64 dimensions only so the program runs quickly; the memory and cost arithmetic uses 384, a typical sentence-embedding width, because that is the size a real system would store. The two numbers never mix: recall comes from the 64-d data, bytes and multiply-adds come from the 384 budget.

The program below builds a synthetic stand-in for sentence embeddings (200 topic centres plus noise, 64 dimensions, unit length), checks the fast selection against a full sort, then measures what quantisation and a simple coarse-cell index (the idea behind IVF, an "inverted file" index) give up. A real system would use a trained sentence encoder instead of random topic vectors; the cost arithmetic below is for a 384-dimension width, an example width, and the recall numbers come from the synthetic data.

import numpy as np
from sklearn.cluster import MiniBatchKMeans

rng = np.random.default_rng(0)
N, D, K = 100_000, 64, 10

# Synthetic stand-in for sentence embeddings: 200 topic centres plus noise
centres = rng.normal(size=(200, D))
topic = rng.integers(0, 200, size=N)
emb = (centres[topic] + 0.8 * rng.normal(size=(N, D))).astype(np.float32)
emb /= np.linalg.norm(emb, axis=1, keepdims=True)           # unit length, so dot product = cosine
queries = emb[rng.choice(N, 50, replace=False)] + 0.05 * rng.normal(size=(50, D)).astype(np.float32)
queries /= np.linalg.norm(queries, axis=1, keepdims=True)


def top_k(emb, q, k):
    scores = emb @ q                                          # [N]
    k = min(k, len(scores))
    idx = np.argpartition(-scores, k - 1)[:k]                 # O(N) selection of the k best, unordered
    return idx[np.argsort(-scores[idx], kind="stable")]       # sort only those k


q = queries[0]
fast = top_k(emb, q, K)
slow = np.argsort(-(emb @ q), kind="stable")[:K]              # full sort reference
print("matches full sort:", np.array_equal(fast, slow))
print("top-3 ids and cosines:", fast[:3], np.round((emb @ q)[fast[:3]], 4))

# Memory arithmetic for N sentences of dimension D
def mb(n, bytes_per_vec):
    return n * bytes_per_vec / 1e6

DIM = 384                                                     # a common sentence-embedding width, used for the budget only
for n in (N, 100 * N):
    print(f"N={n:>10,} dim={DIM}  float32 {mb(n, DIM*4):8.1f} MB  float16 {mb(n, DIM*2):8.1f} MB  int8 {mb(n, DIM):8.1f} MB")

for n in (N, 100 * N):
    print(f"exact search at N={n:>10,}: {n * DIM / 1e6:10.1f} million multiply-adds per query")

# int8 scalar quantisation: scale each vector so its largest entry maps to 127, the biggest value an int8 holds
scale = np.abs(emb).max(axis=1, keepdims=True) / 127.0
emb_i8 = np.round(emb / scale).astype(np.int8)
def recall(approx, exact):
    return len(set(approx) & set(exact)) / len(exact)
r_i8 = []
for q in queries:
    exact = top_k(emb, q, K)
    s = (emb_i8.astype(np.float32) * scale) @ q
    r_i8.append(recall(np.argpartition(-s, K - 1)[:K], exact))
print("int8 recall@10 (mean over 50 queries):", round(float(np.mean(r_i8)), 4))

# IVF idea: coarse k-means, scan only the nprobe nearest cells
nlist = 256
km = MiniBatchKMeans(n_clusters=nlist, random_state=0, n_init=1, batch_size=4096).fit(emb)
cells = [np.where(km.labels_ == c)[0] for c in range(nlist)]
cent = km.cluster_centers_.astype(np.float32)
for nprobe in (1, 4, 16):
    rec, scanned = [], []
    for q in queries:
        near = np.argsort(-(cent @ q))[:nprobe]
        cand = np.concatenate([cells[c] for c in near])
        s = emb[cand] @ q
        approx = cand[np.argsort(-s)[:K]]
        rec.append(recall(approx, top_k(emb, q, K)))
        scanned.append(len(cand) / N)
    print(f"nprobe={nprobe:>2}  recall@10={np.mean(rec):.3f}  fraction scanned={np.mean(scanned):.3f}")

# Two tiny hand-checkable examples
v = np.array([0.5, -0.2, 0.1])
sc = np.abs(v).max() / 127.0
codes = np.round(v / sc).astype(np.int8)
print("int8 codes:", codes, "| scale:", round(float(sc), 6), "| decoded:", np.round(codes * sc, 4))
tiny = np.array([0.1, 0.9, 0.3, 0.7, 0.5])
print("argpartition, k=2:", np.sort(np.argpartition(-tiny, 1)[:2]), "| full sort order:", np.argsort(-tiny))

Output:

matches full sort: True
top-3 ids and cosines: [71429 29778 66022] [0.9396 0.7573 0.7373]
N=   100,000 dim=384  float32    153.6 MB  float16     76.8 MB  int8     38.4 MB
N=10,000,000 dim=384  float32  15360.0 MB  float16   7680.0 MB  int8   3840.0 MB
exact search at N=   100,000:       38.4 million multiply-adds per query
exact search at N=10,000,000:     3840.0 million multiply-adds per query
int8 recall@10 (mean over 50 queries): 0.986
nprobe= 1  recall@10=0.924  fraction scanned=0.005
nprobe= 4  recall@10=1.000  fraction scanned=0.015
nprobe=16  recall@10=1.000  fraction scanned=0.062
int8 codes: [127 -51  25] | scale: 0.003937 | decoded: [ 0.5    -0.2008  0.0984]
argpartition, k=2: [1 3] | full sort order: [1 3 4 2 0]

Reading the numbers

• top_k agrees with a full argsort on the first query. Cost: N*D multiply-adds for the scoring, O(N) for argpartition, O(k log k) for the final sort (O(N) reads "work proportional to N"). Selection is cheaper than sorting because it only has to split the scores into the best k and the rest, not order the rest: sorting all N costs about N log N, which for N = 100,000 is roughly 17 times more comparisons than a single pass. The score vector costs N floats, so for many queries at once, score in chunks of queries (a [Q, N] score matrix is Q*N floats).
• Memory for the vectors alone at 384 dimensions: 153.6 MB in float32 for 100k sentences, 15,360 MB (about 15 GB) for 10 million. Exact scoring at 10 million is 3,840 million multiply-adds per query.
• Halving the bytes per number (float16) halves memory; int8 with a per-vector scale cuts it 4x. How the scale works: an int8 holds whole numbers from -128 to 127. Dividing a vector by max abs / 127 stretches it so its largest entry lands exactly on 127 and every other entry lands proportionally inside the range; rounding then keeps whole numbers only. Multiplying the codes back by the scale recovers the vector up to a rounding error of at most half a step. The last two output lines show this on [0.5, -0.2, 0.1]: the scale is 0.5 / 127 = 0.003937, the codes are 127, -51 and 25, and decoding gives back 0.5, -0.2008 and 0.0984 (small rounding errors, which is the loss). The last line also shows argpartition on five scores picking positions 1 and 3 (scores 0.9 and 0.7) without ordering the other three. On this synthetic data int8 kept 0.986 of the true top-10 on average over 50 queries.
• The coarse-cell index clusters the corpus into 256 cells with k-means, and at query time scans only the nprobe cells whose centroids are closest to the query. Scanning 0.5% of the corpus found 0.924 of the true top-10, and 1.5% found all of them on this data.

The recall figures are specific to this easy, well-clustered synthetic corpus whose queries are perturbed corpus points. Real embeddings are less cleanly clustered and real queries are further from the corpus, so measure recall on your own queries before choosing settings.

Scaling to 100x: the plan to commit to

1. Store fewer bytes. Move from float32 to float16 or int8 (2x to 4x), or to product quantisation (splitting the vector into pieces and replacing each piece by a code from a small codebook), which can reach tens of times. Cost: some recall, recoverable below.
2. Score less of the corpus. Use an approximate nearest-neighbour (ANN) index: IVF as above, or a graph index such as HNSW (hierarchical navigable small world, a layered neighbour graph walked greedily toward the query). Both are approximate: they trade recall for speed through a knob (nprobe for IVF, the search width for HNSW).
3. Re-rank. Fetch, for example, the top 100 from the compressed or approximate stage, then rescore those 100 with the full-precision vectors and keep 10. That recovers most of the lost accuracy at the cost of 100 full-precision reads.
4. Shard if the index no longer fits in one machine's memory: split the corpus across machines, search every shard, merge the per-shard top-k.
5. Use a library (a vector database or an ANN library) rather than hand-rolling the index in production; keep the exact search as the reference you measure recall against.

What would change the plan: if the corpus stays near 100k, keep exact search (no recall loss, nothing to tune). If the corpus changes constantly, a graph index handles inserts more gracefully than an IVF index whose cells drift away from the data; IVF needs periodic re-clustering.

Common mistakes

• Using Euclidean or raw dot product on vectors that were not normalised, so long vectors win regardless of meaning.
• Sorting all N scores.
• Stating a recall or latency win without a recall measurement against exact search on real queries.
• Normalising the corpus but not the query (this does not change the ranking for one query, but changes the score values and any similarity threshold).

Direct answer

Commit to one multilingual model as the default, reached in this order: first measure each language separately so you know where the gap is; then start from a pretrained multilingual encoder whose tokenizer covers your languages fairly; then transfer from English, add a small amount of real target-language data per language, and balance the training mix with temperature sampling; and ship language by language as each clears its own quality bar. Move a language to its own model or adapter only when it fails the bar because of interference (languages sharing one model's capacity degrading each other), a script or tokenizer problem the shared model cannot absorb, or a latency target. The ordering is by effort and expected impact, and each step is checked by the per-language evaluation set from step one.

Prioritized plan (effort and impact)

Tokenizer choice, with a measurement

A subword tokenizer trained mostly on English splits other languages into many more pieces per word. That raises sequence length (latency and cost), dilutes meaning across pieces and usually hurts accuracy. The measure is fertility, tokens per word on held-out text per language. The program below trains a tiny byte-pair-encoding (BPE) tokenizer on a handful of hand-written English, Spanish and Swahili sentences (the Spanish and Swahili lists translate the English list sentence for sentence, in the same order, so each row of the three lists says the same thing), once on English only and once on all three with equal sentences per language, and measures fertility on held-out sentences. BPE works like this: start with every word split into single characters plus an end-of-word marker </w> (so ab</w> can be told apart from a word that merely continues with b). Count how often each adjacent pair of symbols occurs across the training words (weighted by word frequency), merge the most frequent pair into one new symbol everywhere, and repeat for a fixed number of merges (here 120); the ordered list of merges is the tokenizer. To encode a new word, split it into characters and apply the merges in order. Fertility then counts the resulting pieces per word (in this toy the end marker counts as a piece of its own whenever no merge has absorbed it into the last letters), so a tokenizer that has learned a language's common letter groups cuts its words into few pieces and one that has not falls back to near-single characters. Because the merges are learned from all three languages' text, letter groups common in Spanish and Swahili get merged too, which is why the three-language tokenizer gives fewer pieces per Spanish or Swahili word than the English-only one. The two lines printed after the fertility table show this on real words: historia (Spanish for "history", from the held-out sentence "they study history at the university") is cut into 7 pieces by the English-only tokenizer and 6 by the three-language one, and mkulima (Swahili for "farmer") into 6 and 5, each count including the end marker; the first merges are frequent word endings and letter pairs in the pooled text: a</w> comes mostly from Spanish and Swahili words, while e</w> and s</w> come mostly from English and Spanish ones. It is a toy of 8 sentences per language; real tokenizers train on vast corpora, but the direction is what matters: the vocabulary budget goes to the languages in the training text.

from collections import Counter

# the Spanish and Swahili lists translate the English list, sentence for sentence
train = {
    "en": ["the children play football at school", "i like to read many books",
           "my mother cooks the food in the kitchen", "we go to the market tomorrow morning",
           "the teacher teaches the students in class", "good morning my friend",
           "the dog sleeps under the big tree", "she writes a long letter to her sister"],
    "es": ["los niños juegan al fútbol en la escuela", "me gusta leer muchos libros",
           "mi madre cocina la comida en la cocina", "vamos al mercado mañana por la mañana",
           "el profesor enseña a los estudiantes en clase", "buenos días mi amigo",
           "el perro duerme debajo del árbol grande", "ella escribe una carta larga a su hermana"],
    "sw": ["watoto wanacheza mpira shuleni", "ninapenda kusoma vitabu vingi",
           "mama anapika chakula jikoni", "tunaenda sokoni kesho asubuhi",
           "mwalimu anafundisha wanafunzi darasani", "habari za asubuhi rafiki yangu",
           "mbwa analala chini ya mti mkubwa", "yeye anaandika barua ndefu kwa dada yake"],
}
heldout = {
    "en": ["the farmer walks to the river every evening", "they study history at the university"],
    "es": ["el agricultor camina hacia el río cada tarde", "ellos estudian historia en la universidad"],
    "sw": ["mkulima anatembea kuelekea mtoni kila jioni", "wanasoma historia chuoni kikuu"],
}

def train_bpe(sentences, n_merges):
    words = Counter(w for s in sentences for w in s.split())
    vocab = {tuple(w) + ("</w>",): c for w, c in words.items()}
    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for sym, c in vocab.items():
            for a, b in zip(sym, sym[1:]):
                pairs[(a, b)] += c
        if not pairs:
            break
        best = max(pairs, key=lambda p: (pairs[p], p))      # ties go to the alphabetically larger pair
        merges.append(best)
        new_vocab = {}
        for sym, c in vocab.items():
            out, i = [], 0
            while i < len(sym):
                if i < len(sym) - 1 and (sym[i], sym[i + 1]) == best:
                    out.append(sym[i] + sym[i + 1]); i += 2
                else:
                    out.append(sym[i]); i += 1
            new_vocab[tuple(out)] = c
        vocab = new_vocab
    return merges

def encode(word, merges):
    sym = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(sym) - 1:
            if sym[i] == a and sym[i + 1] == b:
                sym[i:i + 2] = [a + b]
            else:
                i += 1
    return sym

def fertility(sentences, merges):
    words = [w for s in sentences for w in s.split()]
    return sum(len(encode(w, merges)) for w in words) / len(words)

n_merges = 120
tok_en = train_bpe(train["en"], n_merges)
tok_all = train_bpe([s for lang in train for s in train[lang]], n_merges)   # equal sentences per language
print(f"BPE with {n_merges} merges, tokens per word on held-out sentences")
print(f"{'':>4} {'English-only':>13} {'three languages':>16}")
for lang in ("en", "es", "sw"):
    print(f"{lang:>4} {fertility(heldout[lang], tok_en):>13.2f} {fertility(heldout[lang], tok_all):>16.2f}")

# What the merges look like, and how one held-out word per language is cut up by each tokenizer
print("first 6 merges, three-language tokenizer:", [a + b for a, b in tok_all[:6]])
for word in ("historia", "mkulima"):
    print(f"{word}: English-only -> {encode(word, tok_en)}   three languages -> {encode(word, tok_all)}")

# Temperature sampling over languages: p_i proportional to n_i ** alpha
counts = {"en": 5_000_000, "es": 800_000, "sw": 20_000}
total = sum(counts.values())
print("\nlanguage examples:", counts)
print(f"{'alpha':>5} " + " ".join(f"{'p_' + l:>8}" for l in counts) + f" {'sw passes over its data per 5M training examples':>50}")
for alpha in (1.0, 0.7, 0.5, 0.3):
    w = {l: n ** alpha for l, n in counts.items()}
    z = sum(w.values())
    p = {l: w[l] / z for l in counts}
    sw_epochs = 5_000_000 * p["sw"] / counts["sw"]
    print(f"{alpha:>5} " + " ".join(f"{p[l]:>8.4f}" for l in counts) + f" {sw_epochs:>50.1f}")

Output:

BPE with 120 merges, tokens per word on held-out sentences
      English-only  three languages
  en          3.57             3.14
  es          5.00             3.93
  sw          6.00             4.30
first 6 merges, three-language tokenizer: ['a</w>', 'e</w>', 's</w>', 'an', 'i</w>', 'en']
historia: English-only -> ['h', 'i', 's', 'to', 'ri', 'a', '</w>']   three languages -> ['h', 'i', 'st', 'or', 'i', 'a</w>']
mkulima: English-only -> ['m', 'k', 'u', 'li', 'ma', '</w>']   three languages -> ['m', 'ku', 'li', 'm', 'a</w>']

language examples: {'en': 5000000, 'es': 800000, 'sw': 20000}
alpha     p_en     p_es     p_sw   sw passes over its data per 5M training examples
  1.0   0.8591   0.1375   0.0034                                                0.9
  0.7   0.7703   0.2136   0.0161                                                4.0
  0.5   0.6834   0.2734   0.0432                                               10.8
  0.3   0.5656   0.3264   0.1079                                               27.0

In this toy the English-only tokenizer needs 5.00 tokens per Spanish word and 6.00 per Swahili word, against 3.57 for English; training on the three languages lowers those to 3.93 and 4.30 (English 3.14). With a pretrained model you usually cannot retrain the tokenizer, so you measure fertility per language and choose the model whose tokenizer is fair for your languages. Training a new tokenizer means pretraining or at least continued pretraining of the model.

Transfer and data balancing

Zero-shot transfer (train on English labels, evaluate on the other languages) gives a baseline and tells you which languages need more. It works because a multilingual encoder maps similar meanings in different languages to nearby representations, so what the classifier learns from English labels partly carries over; this is positive transfer, and it is usually larger between related languages. Then add native examples; even a few hundred correctly labeled examples per language, mixed with English, can be worth more than a large amount of noisy translated data, so measure both on the step-1 sets.

For the training mix, sample language $i$ with probability proportional to $n_i^{\alpha}$, where $n_i$ is the number of examples in language $i$ and $\alpha$ between 0 and 1 flattens the distribution ($\alpha = 1$ keeps natural proportions, $\alpha = 0$ is uniform). The output's second table uses 5,000,000, 800,000 and 20,000 examples. At $\alpha = 1$ Swahili is 0.34% of draws and its data is seen 0.9 times in a 5,000,000-example run; at $\alpha = 0.3$ it is 10.79% and seen 27.0 times. The last column is computed as 5,000,000 x p_sw / 20,000, the number of Swahili examples drawn divided by the number of Swahili examples that exist. For $\alpha = 0.5$: the weights are $\sqrt{5{,}000{,}000} = 2236.1$, $\sqrt{800{,}000} = 894.4$ and $\sqrt{20{,}000} = 141.4$, so $p_{sw} = 141.4 / 3271.9 = 0.0432$ and 5,000,000 x 0.0432 / 20,000 = 10.8 passes. Upsampling that far risks memorizing the small set, so choose $\alpha$ on the per-language validation sets, and combine it with augmentation or added native data rather than repeating the same examples.

One multilingual model or several

• One model: one deployment, one regression suite, positive transfer between related languages, easy addition of a new language by fine-tuning. Risks: capacity shared across languages (a language can be hurt by the others), the largest model's latency for every language.
• Several models: each can use a language-specific tokenizer and a smaller or tuned architecture; interference is gone. Costs: N deployments to monitor and update, N datasets to maintain, no transfer between related languages.
• Recommendation: start with one model, keep the per-language report, and split off a language when its gap persists after steps 2 to 5, when its script (writing system) or morphology (how words are built from stems, prefixes and suffixes; heavily inflected or compounding languages need more pieces per word) is poorly served by the shared tokenizer, or when its latency or cost budget differs. Language-family adapters (small extra layers trained per language or language group while the shared model stays frozen) are a middle path, and are only worth the extra machinery if one model fails for a diagnosed reason.
• What flips it: a language that needs a different tokenizer, a regulatory requirement for separate deployments, or a latency budget one model cannot meet.

Trade-offs and mistakes

• Reporting one average over languages hides the failing ones; report per language with intervals.
• Evaluating on machine-translated test sets can mislead because translated text differs from what real users write.
• Assuming language ID is solved; short or code-mixed inputs (one sentence mixing two languages) are routed wrongly and then fail in the wrong model.
• Ignoring maintenance: every extra model multiplies retraining and monitoring work.

Direct answer

I would normalize with Unicode NFC plus a short explicit mapping list, train one shared subword model (SentencePiece, a library that learns a vocabulary straight from raw text, with the BPE algorithm, which repeatedly merges the most frequent adjacent pair, or the unigram algorithm, which prunes a large candidate vocabulary down to the pieces that explain the text best) on text sampled across the ten languages with a temperature that up-weights the small ones, give it a byte fallback (a character never seen in training is spelled out as its raw bytes) so no character is ever unknown, and tell the model the target language with a tag token. The compute consequences come from three numbers: the vocabulary size (it sets the embedding table, which holds one learned vector per vocabulary entry, and the output layer, which scores every vocabulary entry at each position), the tokens-per-word ratio per language (it sets sequence length, and attention cost grows with the square of length), and how many times the training schedule repeats the small languages. I would start with one shared vocabulary of 64,000 pieces and a sampling temperature of 5, then adjust both from the per-language results below. Each number is computed.

1. Normalization: decide what to rewrite, because the target side is rewritten too

Every character has a code point, the number Unicode assigns to it (written U+0041 for A), and one visible letter can be one code point or several. Unicode therefore defines normalization forms that rewrite equivalent spellings into one. NFC composes characters (e plus a combining accent, two code points, becomes the single é). NFKC does that and also maps "compatibility" characters, such as ligatures and fullwidth letters, to plain equivalents. Normalization on the source side only affects the input; on the target side it changes the text the system outputs. The program prints code point counts and the NFKC result for six examples.

import unicodedata
import numpy as np

# 1. What normalization forms do to text from different scripts (code point counts).
samples = {
    "fullwidth Latin": "ＮＡＴＯ",           # fullwidth N A T O
    "fi ligature": "fish",
    "Arabic lam-alef form": "ﻻ",
    "Devanagari qa (precomposed)": "क़",
    "superscript two": "x²",
    "Japanese half-width katakana": "ｶﾅ",
}
print(f"{'sample':30s} {'raw':>3s} {'NFC':>3s} {'NFKC':>4s}  NFKC result")
for name, s in samples.items():
    nfc, nfkc = unicodedata.normalize("NFC", s), unicodedata.normalize("NFKC", s)
    shown = " ".join(f"U+{ord(c):04X}" for c in nfkc)
    print(f"{name:30s} {len(s):3d} {len(nfc):3d} {len(nfkc):4d}  {shown}")
print("NFKC maps x^2 to 'x2':", unicodedata.normalize("NFKC", "x²") == "x2")

# 2. Temperature sampling over languages. Corpus sizes are INVENTED for illustration (millions of sentence pairs).
langs = ["en", "ru", "zh", "ar", "ja", "hi", "ko", "el", "th", "sw"]
size = np.array([200, 80, 60, 20, 18, 5, 4, 2, 1, 0.3])
budget = 300.0  # millions of sentences seen in training (invented)
print("\nlang  size   T=1 share  T=5 share  epochs@T=1  epochs@T=5  seen@T=1  seen@T=5")
p1 = size / size.sum()
p5 = size ** (1 / 5); p5 /= p5.sum()
for l, n, a, b in zip(langs, size, p1, p5):
    print(f"{l:4s} {n:6.1f}  {a:8.3f}  {b:9.3f}  {a * budget / n:10.2f}  {b * budget / n:10.2f}  {a * budget:8.1f}  {b * budget:8.1f}")
# The same formula by hand on two languages of size 100 and 1 (million pairs).
two = np.array([100.0, 1.0])
for T in (1, 5):
    w = two ** (1 / T)
    print(f"two languages 100 and 1, T={T}: weights {w[0]:.3f} and {w[1]:.3f}, shares {w[0] / w.sum():.3f} and {w[1] / w.sum():.3f}")

# 3. Parameter and compute budget of a shared output layer, d_model = 512.
d = 512
print("\nvocab size V   embedding params V*d   output-layer FLOPs per target token 2*V*d")
for V in (32000, 64000, 250000):
    print(f"{V:12d} {V * d:20,d} {2 * V * d:24,d}")
print("10 separate 32000 vocabs, one per language:", f"{10 * 32000 * d:,d}", "embedding params")

# 4. Sequence-length inflation: tokens per word differs by script. Attention scores grow with length squared.
for f_a, f_b in [(1.3, 2.6), (1.3, 3.9)]:
    print(f"fertility {f_a} vs {f_b}: length ratio {f_b / f_a:.1f}x, attention-score count ratio {(f_b / f_a) ** 2:.1f}x")

Output:

sample                         raw NFC NFKC  NFKC result
fullwidth Latin                  4   4    4  U+004E U+0041 U+0054 U+004F
fi ligature                      3   3    4  U+0066 U+0069 U+0073 U+0068
Arabic lam-alef form             1   1    2  U+0644 U+0627
Devanagari qa (precomposed)      1   2    2  U+0915 U+093C
superscript two                  2   2    2  U+0078 U+0032
Japanese half-width katakana     2   2    2  U+30AB U+30CA
NFKC maps x^2 to 'x2': True

lang  size   T=1 share  T=5 share  epochs@T=1  epochs@T=5  seen@T=1  seen@T=5
en    200.0     0.512      0.172        0.77        0.26     153.7      51.5
ru     80.0     0.205      0.143        0.77        0.54      61.5      42.9
zh     60.0     0.154      0.135        0.77        0.68      46.1      40.5
ar     20.0     0.051      0.108        0.77        1.63      15.4      32.5
ja     18.0     0.046      0.106        0.77        1.77      13.8      31.8
hi      5.0     0.013      0.082        0.77        4.93       3.8      24.6
ko      4.0     0.010      0.079        0.77        5.89       3.1      23.6
el      2.0     0.005      0.068        0.77       10.26       1.5      20.5
th      1.0     0.003      0.060        0.77       17.86       0.8      17.9
sw      0.3     0.001      0.047        0.77       46.81       0.2      14.0
two languages 100 and 1, T=1: weights 100.000 and 1.000, shares 0.990 and 0.010
two languages 100 and 1, T=5: weights 2.512 and 1.000, shares 0.715 and 0.285

vocab size V   embedding params V*d   output-layer FLOPs per target token 2*V*d
       32000           16,384,000               32,768,000
       64000           32,768,000               65,536,000
      250000          128,000,000              256,000,000
10 separate 32000 vocabs, one per language: 163,840,000 embedding params
fertility 1.3 vs 2.6: length ratio 2.0x, attention-score count ratio 4.0x
fertility 1.3 vs 3.9: length ratio 3.0x, attention-score count ratio 9.0x

Reading the first table (a nukta is a dot-below mark that Devanagari adds to a letter to write another sound, and a composition exclusion is a character that Unicode lists as never being recombined into one code point by NFC): fullwidth Latin letters map to ordinary Latin (4 code points either way); the fi ligature expands from 3 to 4 code points; the Arabic lam-alef presentation form (U+FEFB) becomes the two letters U+0644 U+0627; the Devanagari letter QA (U+0958) is decomposed to KA plus nukta even by NFC (it is a composition exclusion), so a precomposed and a decomposed spelling of the same letter become identical only after normalizing; half-width katakana become full-width; and NFKC turns x² into x2, a meaning change that is harmless for a search index but wrong in a translation someone will read.

Decisions:

• Source side: NFKC, then collapse whitespace, strip control and zero-width characters (keeping the zero-width joiner and the zero-width non-joiner, invisible characters that ask neighbouring letters to join or to stay apart, because Indic conjuncts and the spelling of some Arabic-script languages such as Persian depend on them), and drop language-specific noise such as Arabic tatweel (the elongation mark).
• Target side: NFC plus a short explicit list (non-breaking space, fullwidth Latin, ligatures). Blanket NFKC would silently rewrite superscripts and symbols in the output.
• Case: keep it. Lower-casing destroys information the target needs, and a shared vocabulary can hold both forms.
• Do not strip diacritics for translation: in many Latin-script languages an accent changes the word.
• Script-specific quirks (Indic nukta forms, Arabic alef variants) are handled by NFC and a per-language list reviewed by a speaker of the language.
• Data filtering belongs here too: language identification, length ratio between source and target, and removing duplicates, because noisy pairs hurt low-resource languages the most.

2. Tokenization and vocabulary: one shared subword model, no word-level pre-splitting

• Chinese, Japanese and Thai have no spaces, so a rule such as "split on whitespace first" fails for them. SentencePiece-style tokenizers operate on the raw character stream and treat whitespace as a symbol, so one procedure covers all ten languages.
• Vocabulary size trades parameters against length: a larger vocabulary means shorter sequences but more embedding and output parameters. The printed table shows the cost of each size.
• Training data for the vocabulary must be sampled with the same temperature idea as the model (next section); sampling in proportion to size would give the small languages almost no merges, and their words would then be cut into characters.
• Byte fallback represents any unseen character as its bytes, removing [UNK] at a cost of several tokens per rare character.
• Language tag: prepend a target-language token (for example <2ar>) to the source so one model can translate into any of the ten.
• Shared vs separate vocabularies: with a shared vocabulary, the Latin-script languages share pieces and names copy across languages; ten separate 32000-entry vocabularies cost 163,840,000 embedding parameters (printed below), five times the 32,768,000 of one shared 64,000-entry vocabulary, and block that sharing. I commit to shared.

3. Sampling across languages: temperature

If language $i$ has $n_i$ sentence pairs, sampling probability with temperature $T$ is

where $T=1$ is proportional to size and larger $T$ flattens the distribution toward uniform. Raising each size to the power $1/T$ shrinks the gaps between them: with two languages of 100 and 1 million pairs, $T=1$ gives shares 0.990 and 0.010, while $T=5$ gives weights $100^{0.2}=2.512$ and $1^{0.2}=1$, so shares 0.715 and 0.285 (the last two lines of the second printed block show these). A language 100 times larger now gets only 2.5 times the weight.

The program's second table uses invented corpus sizes (in millions of sentence pairs, for illustration) and an invented training budget of 300 million sentences seen. An epoch is one pass over a language's data, so "epochs" is the number of times each of its sentences is used. At $T=1$ every language is passed over 0.77 times, but that equal pass count hides very unequal absolute amounts: English supplies 51% of the batches (153.7 million sentences seen) while Swahili gets 0.3 million pairs times 0.77, about 0.2 million sentences seen, about 670 times fewer, so the smallest languages are barely learned. At $T=5$ the shares move toward even (Swahili sees 14.0 million sentences), but its 0.3 million pairs would be repeated about 47 times, and Thai about 18 times; repeating the same pairs that often invites memorization, so in practice you pick an intermediate temperature and watch the development loss per language. Start at 5 because it gives the smallest languages a usable share, then try lower values such as 2 or 3 if the small languages overfit; treat the exact $T$ as a tuned hyperparameter, not a known constant.

4. Compute implications

• Embedding and output layer: with hidden size 512, a 32000-entry vocabulary has 16,384,000 embedding parameters, a 64000 vocabulary 32,768,000, a 250000 vocabulary 128,000,000. The output layer needs about $2Vd$ floating-point operations (FLOPs, counting each multiply and each add) per generated token (a matrix of size $d \times V$ times a vector), which at $V=250000$ is 256,000,000 per token, about eight times the 32000 case. Ten separate 32000-entry vocabularies would cost 163,840,000 embedding parameters, five times the 32,768,000 of one shared 64000-entry vocabulary.
• Sequence length: fertility (tokens per word) differs by script. With illustrative fertilities of 1.3 and 2.6 the second language's sentences are 2.0 times longer, and the attention score count (which grows with length squared) is 4.0 times larger; at 1.3 versus 3.9 it is 3.0 times longer and 9.0 times the scores. Measure fertility per language on held-out text and watch the languages whose fertility is far above the others: they pay more compute per sentence and are truncated first at a maximum length.
• Padding waste: batch by length bucket per language.

5. How to evaluate the choices

Report per language, never an average only. Use a character-based metric such as chrF (a character n-gram F-score) next to BLEU (a score for how many word sequences in the output also appear in a reference translation), since BLEU depends on how a language is tokenized for scoring; keep the scoring tokenization fixed across experiments. Track fertility, the byte-fallback rate and round-trip fidelity (turning text into token ids and back returns the normalized text exactly) per language as health checks of the preprocessing itself.

Running the code

pip install numpy      # numpy 2.1.3
# save the program above to a file and run it with python