Natural Language Processing Questions

The idea. Skip-gram learns word vectors by asking: given a centre word, which words appear around it? The straightforward way to answer is a softmax (a function that turns a list of scores into probabilities that sum to 1) over the entire vocabulary, and that is too expensive because the sum has one term per vocabulary word. Negative sampling replaces the one huge "which of the V words?" question with a handful of cheap yes/no questions: is this the real context word, or is it a randomly drawn fake?

The formulation. Slide a window of radius $m$ over the text. For each position with centre word $w$ and each context word $c$ within the window, the model should give high probability to $c$ given $w$. Each word has two vectors: $v_w$ when it is the centre and $u_c$ when it is a context. The full-softmax version defines

where $V$ is the vocabulary size. The denominator is the problem: computing it needs a dot product with all $V$ context vectors, and the gradient (the direction in which to adjust parameters) touches all $V$ of them, for every training pair.

What negative sampling does. For each real pair $(w,c)$, draw $k$ random words from a noise distribution and treat them as fake contexts. Train a logistic classifier: push $u_c \cdot v_w$ up for the real pair and $u_n \cdot v_w$ down for each fake $n$:

where $\sigma(x) = 1/(1+e^{-x})$ is the sigmoid function, which squeezes any score into a probability between 0 and 1 (0.5 at score 0, close to 1 for a large positive score, close to 0 for a large negative one). The first term is small when the real pair scores high, and each fake's term $-\log \sigma(-x)$ is small when the fake scores low. Unlike the softmax, a sigmoid looks at one score on its own, with no sum over the vocabulary. The noise distribution is typically word frequency raised to the power 0.75, which draws rare words more often than their raw frequency would. This is a different objective, not the same one computed faster: the outputs are no longer a normalised distribution over the vocabulary. What you keep is the aim that matters, vectors in which words sharing contexts end up close.

The same idea on four scores, by hand. Take one centre word and a four-word vocabulary. Its dot products with the context vectors are 2.0 for the real context word and -1.0, 0.0 and -2.0 for three sampled fakes. The softmax needs all four scores to normalise: it computes $e^{2.0}+e^{-1.0}+e^{0.0}+e^{-2.0}$ and divides $e^{2.0}$ by that total. Negative sampling gives each score its own sigmoid and adds up the four log terms, with no shared total. A loss is a penalty to be minimised: 0 would mean a perfect score.

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

# Scores u.v for one centre word: index 0 is the real context word, the rest are 3 sampled fakes.
scores = [2.0, -1.0, 0.0, -2.0]

# Full softmax over a 4-word vocabulary: one probability for the real word, normalised by all 4 terms.
denominator = sum(math.exp(s) for s in scores)
p_real = math.exp(scores[0]) / denominator
print("softmax: denominator", round(denominator, 4), "| p(real) =", round(p_real, 4), "| loss =", round(-math.log(p_real), 4))

# Negative sampling: each score gets its own sigmoid, no shared denominator.
pos_term = -math.log(sigmoid(scores[0]))
print("real pair: sigmoid(2.0) =", round(sigmoid(2.0), 4), "-> term", round(pos_term, 4))
neg_terms = []
for s in scores[1:]:
    flipped = 0.0 - s
    t = -math.log(sigmoid(flipped))
    neg_terms.append(t)
    print(f"fake with score {s:+.1f}: sigmoid({flipped:+.1f}) = {sigmoid(flipped):.4f} -> term {t:.4f}")
print("negative-sampling loss =", round(pos_term + sum(neg_terms), 4))

Output:

softmax: denominator 8.8923 | p(real) = 0.831 | loss = 0.1852
real pair: sigmoid(2.0) = 0.8808 -> term 0.1269
fake with score -1.0: sigmoid(+1.0) = 0.7311 -> term 0.3133
fake with score +0.0: sigmoid(+0.0) = 0.5000 -> term 0.6931
fake with score -2.0: sigmoid(+2.0) = 0.8808 -> term 0.1269
negative-sampling loss = 1.2603

Reading it: the softmax gives the real word probability 0.831, a loss of 0.1852. Negative sampling scores the real pair with sigmoid(2.0) = 0.8808, a term of 0.1269. A fake with score 0.0 is a coin flip (sigmoid 0.5000), so it costs the most (0.6931); a fake with score -2.0 is pushed down well already and costs 0.1269. The total is 1.2603. The two losses are different quantities (one is a single negative log probability, the other a sum of four binary terms), so only their behaviour compares: the softmax denominator needed every vocabulary score, while each sigmoid term needed only its own score. With a real vocabulary that means only the 1 + k scored words (the real context word and the sampled fakes) instead of all V; in this four-word toy the two sets coincide, so the toy shows the structure of the two losses but not the saving. This is also why the 23.4593 and 15.6233 in the next program should not be set against each other: they come from random vectors and different formulas, and only the row counts and multiply-adds carry the message.

Measured cost, with the program below. It uses a vocabulary of 1,000 words, 50-dimensional vectors, $k=5$, takes one real pair, computes both losses with automatic differentiation (autograd: software that applies the chain rule to the code you wrote and returns the gradient, meaning how much the loss changes if each number is nudged) and counts how many of the context-vector rows receive a nonzero gradient. It also prints the multiply-adds of the output layer, counted as vocabulary size times dimension for the softmax and $(1+k)$ times dimension for negative sampling. The frequency profile is made up for the demonstration.

import torch
import torch.nn.functional as F
torch.set_num_threads(1)
torch.manual_seed(0)

V, d, k = 1000, 50, 5
v = torch.randn(d)                       # centre-word vector
U = torch.randn(V, d, requires_grad=True)  # output (context) vectors, one row per vocabulary word
centre_context = 17                      # index of the true context word

# Full softmax: score every word, normalise over the whole vocabulary
U.grad = None
loss_full = -F.log_softmax(U @ v, dim=0)[centre_context]
loss_full.backward()
rows_full = int((U.grad.abs().sum(1) > 0).sum())
print("full softmax loss", round(loss_full.item(), 4), "| output rows with nonzero gradient:", rows_full)

# Negative sampling: 1 positive + k sampled negatives, each a sigmoid
counts = torch.arange(1, V + 1, dtype=torch.float32).flip(0)   # a made-up frequency profile
noise = counts ** 0.75; noise = noise / noise.sum()
raw = counts / counts.sum()
print("most frequent word: raw share", f"{raw[0].item():.6f}", "-> noise share", f"{noise[0].item():.6f}")
print("rarest word:        raw share", f"{raw[-1].item():.6f}", "-> noise share", f"{noise[-1].item():.6f}")
neg = torch.multinomial(noise, k, replacement=False)
U.grad = None
loss_ns = -F.logsigmoid(U[centre_context] @ v) - F.logsigmoid(-(U[neg] @ v)).sum()
loss_ns.backward()
rows_ns = int((U.grad.abs().sum(1) > 0).sum())
print("negative sampling loss", round(loss_ns.item(), 4), "| output rows with nonzero gradient:", rows_ns, "(1 positive +", k, "negatives)")

# Multiply-adds for the output layer of one training pair
for V_real in (1000, 100000, 1000000):
    print(f"V={V_real}: full softmax {V_real * d} multiply-adds, negative sampling {(1 + k) * d} -> ratio {V_real / (1 + k):.1f}")

Output:

full softmax loss 23.4593 | output rows with nonzero gradient: 1000
most frequent word: raw share 0.001998 -> noise share 0.001748
rarest word:        raw share 0.000002 -> noise share 0.000010
negative sampling loss 15.6233 | output rows with nonzero gradient: 6 (1 positive + 5 negatives)
V=1000: full softmax 50000 multiply-adds, negative sampling 300 -> ratio 166.7
V=100000: full softmax 5000000 multiply-adds, negative sampling 300 -> ratio 16666.7
V=1000000: full softmax 50000000 multiply-adds, negative sampling 300 -> ratio 166666.7

Reading the output

• Full softmax updates all 1,000 context rows for one training pair. Negative sampling updates 6: the 1 real context word and the 5 sampled negatives. (The centre vector is also updated in both cases; the table counts the output side, where the savings are.)
• Cost per pair grows with $V$ for the softmax (50,000 multiply-adds at $V=1000$, 5,000,000 at 100,000 and 50,000,000 at 1,000,000) but stays at 300 for negative sampling, because it depends only on $k$ and the dimension. The ratio is $V/(1+k)$: about 166.7 at 1,000 words and 166,666.7 at a million.
• The 0.75 power moves probability mass toward rare words: the rarest word goes from a share of 0.000002 to 0.000010, while the most frequent drops from 0.001998 to 0.001748.

Trade-offs. The number of negatives $k$ is a hyperparameter: small $k$ is faster but gives a noisier signal; a handful is common. Negative samples can accidentally include a true context word; with a large vocabulary this is rare and ignored. Alternatives that also avoid the full softmax are the hierarchical softmax (a binary tree with the words as leaves, where a word's probability is a product of left-or-right choices along its path, so the cost is about log V per pair) and noise-contrastive estimation (train a classifier to tell real data from noise samples; negative sampling is a simplified relative of it that drops the terms needed to make the outputs a true probability model).

Common mistakes. Saying negative sampling approximates the softmax probabilities (it optimises a different binary objective); sampling negatives uniformly; forgetting that the cost saving applies to the output layer, and that every word still has two vectors.

Direct answer

A semantic parsing pipeline turns a free-text request into something a program can run. Run the four stages in order and give each its own model type, data and metric: intent detection (classify what the user wants), slot filling (find the spans that carry the arguments), canonicalization (map each span to a system identifier or normalized value), and plan construction (assemble a validated, executable call). The design problem is error propagation: a stage can only be as right as its input, so pass scored alternatives instead of single hard labels, validate every hand-off against a schema, measure each stage twice (on gold input and on real upstream output), and let low-confidence cases become a clarifying question instead of an action.

The four stages

Terms used below. A slot is a named argument of an intent (for a flight booking: origin, destination, date). BIO tagging labels every token as B (begins a slot span), I (inside one) or O (outside), so a sequence tagger can find spans: for "fly from san francisco to nyc on tuesday" a tagger outputs fly/O from/O san/B-origin francisco/I-origin to/O nyc/B-dest on/O tuesday/B-date (the program prints this). Span F1 counts a predicted slot as correct only if both its boundaries and its slot type match the gold span exactly. Canonicalization (also called normalization or entity linking) maps "nyc" to the airport code JFK and "friday" to a concrete date. Calibration means a model's stated confidence matches how often it is right. A pretrained encoder is a neural network already trained on large amounts of text that you then fine-tune on your labels. A reranker is a second model that reorders a short list of candidates produced by a cheaper first step. A constrained sequence-to-sequence decoder is a model that writes its output token by token but is only allowed tokens that keep the output valid under a grammar or schema. Multi-task learning trains one network on several tasks at once. An n-best list is the top few alternatives with their scores instead of only the winner. Hard negatives are examples that look like a class but are not in it (a refund request versus a cancellation). The out-of-scope false-accept rate is the share of requests the system should reject that it accepts as some intent. Cross-validated predictions are predictions for each example made by a model that was trained on the other folds and never saw that example. For a first version the core is: a linear intent model, a rule or tagger for slots, alias dictionaries, and templates for plans; the other options in the table are upgrades for when that baseline's measured errors justify them.

Two design choices cut across the table. First, every hand-off is a typed object, not free text: intent from a closed list, slots validated against that intent's schema, canonical values from the catalogue. That makes illegal states unrepresentable and lets a schema check act as an automatic error detector. Second, the stages run in the order shown only if the information flows that way; slot filling usually benefits from knowing the intent (the schema tells it which slots to look for), and the intent classifier can in turn be re-scored using whether its required slots were found.

Handling error propagation

Errors cascade because a wrong early decision is treated as fact later. The techniques, in the order of how much they usually buy for the engineering cost:

1. Measure the propagation first. Report each stage on gold input (an upper bound, with the previous stages' answers injected from the labels) and on real upstream output. The gap between the two numbers is the damage done by the upstream stage. Stage accuracies do not simply multiply, because errors are correlated and some downstream stages are robust to upstream errors, so only the end-to-end run is the real number.
2. Pass scores or an n-best list, not a hard label. Keep the top 2 or 3 intents with probabilities and let the later stages rescore them: an intent whose required slots are missing is less likely than one whose slots were all found. The program traces this on the utterance that cascades: the intent scores are a 0.50 / 0.50 tie between book_flight and cancel_booking; the slot stage then finds origin, destination and date, which are exactly the slots book_flight requires, so that candidate gets a +0.25 bonus (an arbitrary demonstration weight, to be tuned on labelled data) and wins 0.75 to 0.50. Plan accuracy on the 13 utterances becomes 10 of 13, the same as injecting the gold intent.
3. Validate at each hand-off and fail closed. A slot value that is not in the catalogue, a date in the past for a booking, a plan that violates the API schema: all return NONE or a low score instead of a guess.
4. Train downstream stages on realistic upstream output. If the slot filler is trained only on gold intents, it never sees the distribution of wrong intents it receives in production. Use cross-validated predictions of the upstream model as the downstream training input for a share of the examples. The reason to avoid predictions on the upstream model's own training data: the model has memorised those examples, so it looks almost perfect there, and the downstream stage would learn to trust an input that is far cleaner than what it gets in production. The program prints three such rows: each has the utterance, the gold intent and what the upstream stage predicted with its confidence, and the third row is the case where they differ, which is the example the downstream stage must learn to cope with.
5. Share representations or train jointly where the data allows. A single encoder with an intent head and a slot head (multi-task learning) lets the two tasks regularize each other. It couples the stages in training and release, which is a cost.
6. Route uncertainty to the user. Ask a clarifying question when the intent confidence is low or a required canonical value is NONE. The threshold is a product decision: it trades wrong actions against annoying questions.
7. Log stage outputs and run error analysis by stage, so each production failure is attributed to the stage that introduced it.

Worked example

How to read the code: intent() counts keyword hits per intent and returns the winner with its confidence; the key (scores[k], k) breaks a tie by the larger intent name alphabetically, which is why a 2-2 tie between cancel_booking and book_flight goes to cancel_booking. In slots(), the regular expression reads: the word "from", then lazily capture letters and spaces (the origin), then "to", then capture the destination, which ends at " on", the end of the text, " next" or a full stop; the lazy +? takes as few characters as possible so the origin stops at the first " to". canon() looks each span up in the dictionary and gives None if it is missing. run() is the pipeline; passing gold_intent skips the intent stage and uses the label with confidence 1.0, and passing gold_slots replaces the canonical slots with the labels, which is the gold-injection experiment.

The program below is a rule-based stand-in for the learned models (keyword-count intent scoring, a regular expression for slots, a dictionary for canonicalization); a real system would put the models from the table in those places. The 13 hand-written utterances are a toy set chosen to contain realistic failure types, so the printed rates illustrate the measurement method and say nothing about how a real system performs. Intent confidence is the share of keyword hits that went to the winning intent.

import re, json
from collections import Counter

INTENTS = {
    "book_flight": ["fly", "flight", "book"],
    "cancel_booking": ["cancel", "refund"],
    "check_status": ["status", "delayed", "on time"],
}
CITIES = {"nyc": "JFK", "new york": "JFK", "sf": "SFO", "san francisco": "SFO", "boston": "BOS", "bos": "BOS"}
WEEKDAYS = ["monday","tuesday","wednesday","thursday","friday","saturday","sunday"]

def intent(text):
    scores = {k: sum(w in text for w in ws) for k, ws in INTENTS.items()}
    best = max(scores, key=lambda k: (scores[k], k))
    total = sum(scores.values())
    conf = scores[best] / total if total else 0.0
    return best, conf

def slots(text):
    out = {}
    m = re.search(r"from ([a-z ]+?) to ([a-z ]+?)(?: on|$| next|\.)", text)
    if m:
        out["origin"], out["dest"] = m.group(1).strip(), m.group(2).strip()
    for d in WEEKDAYS:
        if d in text:
            out["date"] = d
    return out

def canon(sl):
    out = {}
    for k, v in sl.items():
        if k in ("origin", "dest"):
            out[k] = CITIES.get(v)
        else:
            out[k] = WEEKDAYS.index(v) if v in WEEKDAYS else None
    return out

def plan(i, c):
    if i == "book_flight" and all(c.get(k) is not None for k in ("origin","dest","date")):
        return {"call": "search_flights", "args": c}
    if i == "cancel_booking":
        return {"call": "cancel", "args": {}}
    if i == "check_status":
        return {"call": "status", "args": {}}
    return {"call": "ask_clarification", "args": {}}

DATA = [
 ("book a flight from nyc to sf on friday", "book_flight", {"origin":"JFK","dest":"SFO","date":4}),
 ("i want to fly from boston to new york on monday", "book_flight", {"origin":"BOS","dest":"JFK","date":0}),
 ("book me from sf to bos on sunday", "book_flight", {"origin":"SFO","dest":"BOS","date":6}),
 ("fly from san francisco to nyc on tuesday", "book_flight", {"origin":"SFO","dest":"JFK","date":1}),
 ("book a flight from chicago to sf on friday", "book_flight", {"origin":"ORD","dest":"SFO","date":4}),
 ("cancel my booking and refund me", "cancel_booking", {}),
 ("please cancel the trip", "cancel_booking", {}),
 ("is my flight delayed", "check_status", {}),
 ("what is the status of my booking", "check_status", {}),
 ("cancel my flight to sf on friday", "cancel_booking", {}),
 ("cancel the refund request and book a flight from nyc to sf on friday", "book_flight", {"origin":"JFK","dest":"SFO","date":4}),
 ("book from nyc to sf on monday not friday", "book_flight", {"origin":"JFK","dest":"SFO","date":0}),
 ("book boston to sf on monday", "book_flight", {"origin":"BOS","dest":"SFO","date":0}),
]

def run(text, gold_intent=None, gold_slots=None):
    i, conf = intent(text) if gold_intent is None else (gold_intent, 1.0)
    sl = slots(text)
    c = canon(sl)
    if gold_slots is not None:
        c = gold_slots
    return i, conf, sl, c, plan(i, c)

n = len(DATA)
gold_plans = [plan(gi, gs) for _, gi, gs in DATA]
intent_ok = sum(intent(t)[0] == gi for t, gi, _ in DATA)
slot_ok = sum(canon(slots(t)) == gs for t, gi, gs in DATA if gi == "book_flight")
nb = sum(gi == "book_flight" for _, gi, _ in DATA)
e2e = sum(run(t)[4] == gp for (t, _, _), gp in zip(DATA, gold_plans))
print(f"intent acc (raw text): {intent_ok}/{n}")
print(f"slot+canon exact match on booking utterances (raw text): {slot_ok}/{nb}")
print(f"plan correct end to end: {e2e}/{n}")
print("--- per-utterance trace of failures ---")
for (t, gi, gs), gp in zip(DATA, gold_plans):
    i, conf, sl, c, p = run(t)
    if p != gp:
        print(f"{t!r}\n  intent={i} (conf {conf:.2f}, gold {gi}) slots={sl} canon={c}\n  plan={p} gold={gp}")
# error propagation: plan with gold intent but predicted slots, and gold slots but predicted intent
a = sum(run(t, gold_intent=gi)[4] == gp for (t, gi, _), gp in zip(DATA, gold_plans))
b = sum(run(t, gold_slots=gs)[4] == gp for (t, _, gs), gp in zip(DATA, gold_plans))
print(f"plan correct with gold intent injected: {a}/{n}")
print(f"plan correct with gold canonical slots injected: {b}/{n}")
# abstain policy: clarify when intent confidence < 1 or any canonical value is None
def guarded(text):
    i, conf, sl, c, p = run(text)
    if conf < 1.0 or any(v is None for v in c.values()):
        return {"call": "ask_clarification", "args": {}}
    return p
wrong_act = sum(1 for (t,_,_), gp in zip(DATA, gold_plans) if run(t)[4] != gp and run(t)[4]["call"] != "ask_clarification")
clar_plain = sum(run(t)[4]["call"] == "ask_clarification" for t,_,_ in DATA)
wrong_act_g = sum(1 for (t,_,_), gp in zip(DATA, gold_plans) if guarded(t) != gp and guarded(t)["call"] != "ask_clarification")
clar_g = sum(guarded(t)["call"] == "ask_clarification" for t,_,_ in DATA)
print(f"wrong actions executed, no guard: {wrong_act}; with guard: {wrong_act_g}; clarifications asked without guard: {clar_plain}, with guard: {clar_g}")

# BIO tags: what a learned slot tagger would output for an utterance (B = begins a slot, I = inside, O = outside).
def bio(text, sl):
    toks = text.split()
    tags = ["O"] * len(toks)
    for name in ("origin", "dest", "date"):
        if name in sl:
            words = sl[name].split()
            for s in range(len(toks) - len(words) + 1):
                if toks[s:s + len(words)] == words:
                    tags[s] = "B-" + name
                    for j in range(1, len(words)):
                        tags[s + j] = "I-" + name
                    break
    return list(zip(toks, tags))
print("--- BIO tagging ---")
for t in ("book a flight from nyc to sf on friday", "fly from san francisco to nyc on tuesday"):
    print(" ".join(f"{w}/{g}" for w, g in bio(t, slots(t))))

# Training rows for a downstream stage: gold intent versus the upstream model's prediction for the same utterance.
# (A learned upstream model would make these predictions in cross-validation, from a fold that never saw the utterance.)
print("--- downstream training rows (utterance | gold intent | upstream prediction, confidence) ---")
for t, gi, _ in DATA[8:11]:
    pi, pc = intent(t)
    print(f"{t:68s} | {gi:14s} | {pi}, {pc:.2f}")

# Passing scored alternatives (n-best) to the next stage instead of one hard label.
SCHEMA = {"book_flight": {"origin", "dest", "date"}, "cancel_booking": set(), "check_status": set()}
def intent_nbest(text):
    scores = {k: sum(w in text for w in ws) for k, ws in INTENTS.items()}
    total = sum(scores.values())
    return {k: (v / total if total else 0.0) for k, v in scores.items()}
def rescored(text):
    found = set(slots(text))
    out = {}
    for k, conf in intent_nbest(text).items():
        out[k] = conf + (0.25 if SCHEMA[k] and SCHEMA[k] <= found else 0.0)   # bonus: its required slots were all found
    return out
cascade = DATA[10][0]
print("--- n-best for:", cascade)
print("intent scores   :", {k: round(v, 2) for k, v in intent_nbest(cascade).items()})
print("after slot check:", {k: round(v, 2) for k, v in rescored(cascade).items()})
def run_nbest(text):
    r = rescored(text)
    i = max(r, key=lambda k: (r[k], k))
    c = canon(slots(text))
    return plan(i, c)
nb_ok = sum(run_nbest(t) == gp for (t, _, _), gp in zip(DATA, gold_plans))
print(f"plan correct with n-best rescoring: {nb_ok}/{n}")

Output:

intent acc (raw text): 12/13
slot+canon exact match on booking utterances (raw text): 5/8
plan correct end to end: 9/13
--- per-utterance trace of failures ---
'book a flight from chicago to sf on friday'
  intent=book_flight (conf 1.00, gold book_flight) slots={'origin': 'chicago', 'dest': 'sf', 'date': 'friday'} canon={'origin': None, 'dest': 'SFO', 'date': 4}
  plan={'call': 'ask_clarification', 'args': {}} gold={'call': 'search_flights', 'args': {'origin': 'ORD', 'dest': 'SFO', 'date': 4}}
'cancel the refund request and book a flight from nyc to sf on friday'
  intent=cancel_booking (conf 0.50, gold book_flight) slots={'origin': 'nyc', 'dest': 'sf', 'date': 'friday'} canon={'origin': 'JFK', 'dest': 'SFO', 'date': 4}
  plan={'call': 'cancel', 'args': {}} gold={'call': 'search_flights', 'args': {'origin': 'JFK', 'dest': 'SFO', 'date': 4}}
'book from nyc to sf on monday not friday'
  intent=book_flight (conf 1.00, gold book_flight) slots={'origin': 'nyc', 'dest': 'sf', 'date': 'friday'} canon={'origin': 'JFK', 'dest': 'SFO', 'date': 4}
  plan={'call': 'search_flights', 'args': {'origin': 'JFK', 'dest': 'SFO', 'date': 4}} gold={'call': 'search_flights', 'args': {'origin': 'JFK', 'dest': 'SFO', 'date': 0}}
'book boston to sf on monday'
  intent=book_flight (conf 1.00, gold book_flight) slots={'date': 'monday'} canon={'date': 0}
  plan={'call': 'ask_clarification', 'args': {}} gold={'call': 'search_flights', 'args': {'origin': 'BOS', 'dest': 'SFO', 'date': 0}}
plan correct with gold intent injected: 10/13
plan correct with gold canonical slots injected: 12/13
wrong actions executed, no guard: 2; with guard: 1; clarifications asked without guard: 2, with guard: 7
--- BIO tagging ---
book/O a/O flight/O from/O nyc/B-origin to/O sf/B-dest on/O friday/B-date
fly/O from/O san/B-origin francisco/I-origin to/O nyc/B-dest on/O tuesday/B-date
--- downstream training rows (utterance | gold intent | upstream prediction, confidence) ---
what is the status of my booking                                     | check_status   | check_status, 0.50
cancel my flight to sf on friday                                     | cancel_booking | cancel_booking, 0.50
cancel the refund request and book a flight from nyc to sf on friday | book_flight    | cancel_booking, 0.50
--- n-best for: cancel the refund request and book a flight from nyc to sf on friday
intent scores   : {'book_flight': 0.5, 'cancel_booking': 0.5, 'check_status': 0.0}
after slot check: {'book_flight': 0.75, 'cancel_booking': 0.5, 'check_status': 0.0}
plan correct with n-best rescoring: 10/13

Reading the output: intent is right on 12 of 13 utterances, yet the final plan is right on only 9 of 13, because slot extraction and canonicalization fail on three booking utterances (an airport that is missing from the dictionary, a regular expression that needs the word "from", and a correction "monday not friday" that the date logic reads wrongly). Injecting the correct intent lifts plan accuracy from 9 to 10 of 13; injecting the correct canonical slots lifts it from 9 to 12 of 13. So in this toy the slot and canonicalization stages are the bottleneck, and the gold-injection comparison is what shows it. The failure on "cancel the refund request and book a flight..." is the classic cascade: the intent tie goes to the wrong intent with confidence 0.50 and the plan stage executes a cancellation.

The guard in the wrong actions executed line (ask for clarification when intent confidence is below 1.0 or any canonical value is NONE) removes one of the two wrong actions, the cancellation, but asks 7 clarifications instead of 2. It misses "monday not friday", because that parse is confident and well-formed but wrong; only a stage that models the correction, or a read-back to the user ("Friday, correct?"), catches that kind of error. The BIO, training-row and n-best lines at the end of the output cover the points the table describes. This is a minimal illustration of confidence gating, not a production safety mechanism: it uses one crude confidence value, ignores the cost of each action type, and a production system must also handle confirmation for irreversible actions, authorization, and retries.

Trade-offs and pitfalls

• Modular pipeline versus one end-to-end model. The pipeline gives per-stage metrics, can reuse dictionaries and rules, and lets you fix one stage without retraining the rest. An end-to-end sequence-to-sequence parser removes hand-offs (no cascade) but needs more paired (utterance, plan) data, and its failures are harder to localize. With little plan-level data and a fixed set of intents, prefer the pipeline; with plentiful (utterance, plan) pairs and compositional requests, move to a constrained end-to-end decoder, keeping the schema validator at the end.
• Dictionaries and rules are not a weakness for closed-world slots such as airport codes or dates; they are deterministic, testable and auditable. Use learned models where the surface forms are open-ended.
• Evaluating stages only on gold input hides cascade damage; evaluating only end to end hides where to fix. Report both.
• Out-of-scope input is the most common production failure: without an explicit out-of-scope class and a rejection threshold, every utterance gets forced into some intent.
• Distribution shift in the catalogue (new airports, new product names) breaks canonicalization silently; monitor the NONE rate over time.

Running the code

# Python 3.12, standard library only (no install step)
python parse.py

Direct answer

If 94% of tokens are O (outside any entity), three things go wrong. Token accuracy becomes meaningless, because a model that tags everything O scores 94% while finding no entity. The training loss is dominated by easy O tokens, so the optimizer is rewarded for being confident about O and learns little about the rare entity tags. And at the decision boundary (the point where the model's probability for a tag is just enough to make it pick that tag instead of O) the model is biased toward O, because O is the most likely tag before looking at the token at all (its prior: the base rate, 94% here), which costs recall (finding the entities that exist). The remedies are on three levels: evaluate with entity-level precision, recall and F1 and never token accuracy; reweight or reshape the loss (class weights, focal loss) or the sampling (drop sentences with no entities); and tune the decision threshold or decoder on a validation set. None of these is free: each trades precision for recall, so you choose the operating point (the precision and recall pair the system actually runs at) from the product's needs.

Evaluation: the accuracy trap, computed

NER is scored on entities (spans), not tokens. An entity is correct only if its type and both boundaries match (exact-match, entity-level scoring). Precision is the share of predicted entities that are right, recall is the share of gold entities found, and F1 is their harmonic mean (twice their product divided by their sum), a single score that stays low unless both numbers are reasonably high.

The program builds one 100-token document with 3 entities (6 non-O tokens) and scores two predictors: one that outputs O everywhere, and one that finds all three entities but truncates the last one by two tokens.

import numpy as np
from sklearn.linear_model import LogisticRegression

def spans(tags):
    out, start, typ = set(), None, None
    for i, t in enumerate(tags + ["O"]):
        if start is not None and not (t.startswith("I-") and t[2:] == typ):
            out.add((start, i, typ)); start = None
        if t.startswith("B-"):
            start, typ = i, t[2:]
    return out

# Part 1: a 100-token document with 3 entities (6 non-O tokens) and 94 O tokens.
gold = ["O"] * 100
gold[10:12] = ["B-PER", "I-PER"]
gold[40] = "B-ORG"
gold[70:73] = ["B-LOC", "I-LOC", "I-LOC"]
all_o = ["O"] * 100
near = list(gold); near[71] = "O"; near[72] = "O"       # right start, truncated LOC
for name, pred in [("always O", all_o), ("truncated LOC", near)]:
    acc = np.mean([g == p for g, p in zip(gold, pred)])
    g, p = spans(gold), spans(pred)
    tp = len(g & p)
    prec = tp / len(p) if p else 0.0
    rec = tp / len(g)
    f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
    print(f"{name:14s} token accuracy={acc:.2f}  entity precision={prec:.2f} recall={rec:.2f} F1={f1:.2f}")

# Part 2: share of the untrained loss, class weights, the prior, and focal reweighting.
n_o, n_e = 940, 60
print("O tokens' share of mean cross-entropy at uniform predictions:", round(n_o / (n_o + n_e), 2))
w_o, w_e = (n_o + n_e) / (2 * n_o), (n_o + n_e) / (2 * n_e)      # balanced weights: n / (classes * count)
print(f"class weights: O={w_o:.2f}  entity={w_e:.2f}")
for name, p_t, w in (("easy O token", 0.95, w_o), ("uncertain entity token", 0.30, w_e)):
    print(f"{name:22s} p_t={p_t:.2f}  loss={-np.log(p_t):.3f}  weighted loss={w * -np.log(p_t):.3f}")
prior_loss = -(0.94 * np.log(0.94) + 0.06 * np.log(0.06))      # always output the prior 0.94 / 0.06
print(f"mean loss, O-vs-entity: coin flip={np.log(2):.3f}  prior only={prior_loss:.3f}  prior odds O:entity={0.94 / 0.06:.1f}:1")
gamma = 2.0
for p_t in (0.95, 0.5, 0.2):
    print(f"p_t={p_t:.2f}  focal factor (1-p_t)^{gamma:.0f} = {(1 - p_t) ** gamma:.4f}")

# Part 3: synthetic tokens, 94% O. Two overlapping Gaussian clusters, 20 seeds.
def run(seed, weight, thr):
    rng = np.random.default_rng(seed)
    def make(n):
        y = (rng.random(n) < 0.06).astype(int)
        X = rng.normal(0, 1, (n, 2)) + y[:, None] * np.array([1.5, 1.5])
        return X, y
    Xtr, ytr = make(4000); Xte, yte = make(20000)
    clf = LogisticRegression(class_weight=weight).fit(Xtr, ytr)
    p = (clf.predict_proba(Xte)[:, 1] > thr).astype(int)      # predict "entity" when its probability exceeds thr
    tp = ((p == 1) & (yte == 1)).sum()
    return tp / max(p.sum(), 1), tp / yte.sum()
for weight, thr in ((None, 0.5), ("balanced", 0.5), (None, 0.3), (None, 0.2), (None, 0.15)):
    res = np.array([run(s, weight, thr) for s in range(20)])
    pr, rc = res.mean(axis=0)
    print(f"class_weight={weight!s:9s} threshold={thr:.2f} precision={pr:.2f} recall={rc:.2f} F1={2*pr*rc/(pr+rc):.2f}")

Output:

always O       token accuracy=0.94  entity precision=0.00 recall=0.00 F1=0.00
truncated LOC  token accuracy=0.98  entity precision=0.67 recall=0.67 F1=0.67
O tokens' share of mean cross-entropy at uniform predictions: 0.94
class weights: O=0.53  entity=8.33
easy O token           p_t=0.95  loss=0.051  weighted loss=0.027
uncertain entity token p_t=0.30  loss=1.204  weighted loss=10.033
mean loss, O-vs-entity: coin flip=0.693  prior only=0.227  prior odds O:entity=15.7:1
p_t=0.95  focal factor (1-p_t)^2 = 0.0025
p_t=0.50  focal factor (1-p_t)^2 = 0.2500
p_t=0.20  focal factor (1-p_t)^2 = 0.6400
class_weight=None      threshold=0.50 precision=0.75 recall=0.40 F1=0.52
class_weight=balanced  threshold=0.50 precision=0.27 recall=0.86 F1=0.41
class_weight=None      threshold=0.30 precision=0.59 recall=0.56 F1=0.58
class_weight=None      threshold=0.20 precision=0.49 recall=0.66 F1=0.56
class_weight=None      threshold=0.15 precision=0.42 recall=0.72 F1=0.53

The first two lines are the evaluation lesson: the all-O model has 0.94 token accuracy and 0.00 F1; the model that gets two of three entities exactly right has 0.98 accuracy but F1 of 0.67, and the missing 0.33 comes from a single boundary error that token accuracy barely sees. Token-level accuracy that excludes O still hides boundary errors, so use span-level scores, and report them per entity type.

Training: where the loss goes

Cross-entropy (the usual classification loss: minus the log of the probability assigned to the correct tag) is averaged over tokens. At the start every tag is about equally likely, so every token contributes the same loss and the O tokens supply their share of the total, the printed 0.94 here.

Early in training a model typically learns the prior first. Treat the problem as O against entity: guessing 50/50 costs a mean loss of 0.693 per token, while simply outputting the base rates 0.94 and 0.06 already costs 0.227, a large drop that requires no understanding of the text. Having learned that, the model sits at a plateau where O is the safe answer everywhere. To label a token as an entity it must now see evidence strong enough to beat the prior odds of 15.7 to 1 for O; in terms of probability, the entity's probability has to pass 0.5 before argmax (taking the highest-probability tag) picks it. An entity token whose features only give 0.3 loses to the prior, even though 0.3 is far above the 0.06 base rate. That is the sense in which uncertain entity tokens lose to the prior, and it is why recall suffers first.

Meanwhile the easy O tokens still dominate by count. An O token the model gives 0.95 costs only 0.051, but there are 940 of them, and their steady pressure keeps pushing probability mass toward O. An uncertain entity token at 0.30 costs 1.204 each, about 24 times as much, but there are only 60. Two further effects:

• Missing annotations look like O. Anything an annotator skipped is labelled O, so a fraction of the O mass is really unlabelled entities. The model is then trained to suppress true entities. Audit a sample of high-confidence "false positives" before blaming the model.
• Rare types suffer first. With five entity types the rarest type has the fewest tokens and is the first to collapse to O.

What to do

The focal factors printed above, with $\gamma = 2$: an easy O token the model already gives 0.95 receives a factor of 0.0025, a coin-flip token 0.25, and a hard token with probability 0.20 gets 0.64. The loss then focuses on the tokens the model finds hard, which are mostly entity tokens.

A small experiment: weighting moves the operating point

The last five output lines come from a synthetic two-feature problem (not NER text): 6% of examples belong to the positive class, the classes overlap, and a logistic regression (a simple classifier that turns a weighted sum of the features into a probability; scikit-learn 1.5.2) is trained with and without class_weight="balanced", averaged over 20 seeds. With balanced weights recall rises from 0.40 to 0.86 while precision falls from 0.75 to 0.27, and F1 falls from 0.52 to 0.41. Keeping the plain model and only lowering the decision threshold gives a similar movement with a better result: at 0.30 precision is 0.59, recall 0.56 and F1 0.58, and at 0.15 recall is 0.72 with precision 0.42. So weighting and thresholds do not make a model better; they move it along the precision-recall trade-off, and the weighted run moved past the best F1 point while a moderate threshold landed near it. That is why the sentence "reweight the loss" is incomplete without "then choose the threshold on a validation set": pick the weight, or the threshold, by the F1 or by the cost of a miss versus a false alarm that the product actually has.

Recommendation

Evaluate with entity-level F1 per type. Train with the plain loss first, add a mild weight or focal loss only if recall on rare types is the measured problem, and finish with threshold tuning on validation data. Audit labels for missing entities before adding any trick. What flips the choice: if a missed entity is far costlier than a false alarm (for example screening for personal data to redact), push toward recall with heavier weights and a lower threshold; if every alert is reviewed by hand, favour precision.

Running the code

pip install numpy scikit-learn    # numpy 2.1.3, scikit-learn 1.5.2
# save the program above to a file and run it with python

Direct answer

Chinese and Japanese are written without spaces between words, so a tokenizer has to decide where tokens begin. Character-level tokenization has a closed vocabulary (every character is known in advance, a few thousand symbols for Chinese, far fewer than a word list) and always lines up with entity boundaries, but sequences are long and each token carries little meaning. Lexicon-based word segmentation (tools such as Jieba for Chinese, a dictionary-and-statistics segmenter, or MeCab for Japanese, a morphological analyser that splits text into morphemes, the smallest meaningful units such as a word stem and a grammatical ending) gives short, meaningful tokens, but its errors cannot be undone downstream, its boundaries can cut across entity boundaries, and anything outside the dictionary falls apart. Subword tokenization (SentencePiece, a library that learns a vocabulary directly from raw text with no pre-splitting, using BPE, or the unigram model, which starts from a large pool of candidate pieces and repeatedly drops the ones whose removal hurts the fit to the corpus least) sits in between: a vocabulary size you choose before training (for example 32,000 pieces), no dictionary, and boundaries that follow corpus frequency rather than linguistics, so they also can cross entities. For NER (named entity recognition: marking spans such as people, organisations and places) my recommendation is subword or character tokens, with offsets kept so labels map back to character spans, and a lexicon used as an extra feature only when a domain gazetteer (a list of known names) matters.

The three schemes on the same text

The program uses six invented Chinese training sentences, each printed with an English gloss and its gold entities (the output starts with them, so the entity positions can be checked), a hand-made 33-word lexicon as a stand-in for a real segmenter (a real one also uses statistics for unknown words), a plain forward-maximum-matching segmenter (take the longest lexicon entry at each position, else one character), and a BPE (byte pair encoding: repeatedly merge the most frequent adjacent pair) learned with 12 merges on the six sentences. Everything is tiny on purpose; the numbers show mechanisms, not real-corpus statistics.

from collections import Counter

# Tiny training corpus written for this demo. Gold entity spans are (start, end, type) in characters.
train = [
    ("张伟在北京大学研究数学。", [(0, 2, "PER"), (3, 7, "ORG")], "Zhang Wei studies mathematics at Peking University."),
    ("华为总部位于深圳。", [(0, 2, "ORG"), (6, 8, "LOC")], "Huawei's headquarters is in Shenzhen."),
    ("李娜在北京工作。", [(0, 2, "PER"), (3, 5, "LOC")], "Li Na works in Beijing."),
    ("他的儿子喜欢计划旅行。", [], "His son likes planning trips."),
    ("数量很多，北京大学很大。", [(5, 9, "ORG")], "The quantity is large; Peking University is big."),
    ("我们开会研究这个计算问题。", [], "We hold a meeting to study this computation problem."),
]
test_text = "他在美国会见了专家。"           # "He met the experts in the United States."
test_ents = [(2, 4, "LOC")]                 # 美国 (the United States), 美国会 is 'Congress'
new_term = "量子计算很有趣。"               # "Quantum computing is interesting."; 量子 = quantum, 计算 = computing

# Lexicon stand-in for a dictionary segmenter such as Jieba or MeCab (those also use statistical models).
LEX = {"张伟", "北京大学", "研究", "数学", "华为", "总部", "位于", "深圳", "李娜", "北京", "工作",
       "他的", "儿子", "喜欢", "计划", "旅行", "数量", "很多", "很大", "我们", "开会", "这个",
       "计算", "问题", "美国", "美国会", "会见", "专家", "他", "在", "了", "很", "有趣"}

def chars(text):
    return list(text)

def fmm(text, lex=LEX, maxlen=4):
    """Forward maximum matching: take the longest lexicon entry at each position, else one character."""
    out, i = [], 0
    while i < len(text):
        for n in range(min(maxlen, len(text) - i), 0, -1):
            if n == 1 or text[i:i + n] in lex:
                out.append(text[i:i + n]); i += n; break
    return out

def learn_bpe(sentences, n_merges):
    seqs = [list(s) for s in sentences]
    merges = []
    for _ in range(n_merges):
        pairs = Counter((a, b) for q in seqs for a, b in zip(q, q[1:])
                        if "。" not in a + b and "，" not in a + b)   # never merge across punctuation
        if not pairs:
            break
        best = max(pairs, key=lambda p: (pairs[p], p))       # ties broken by pair order
        merges.append(best)
        for q in seqs:
            i = 0
            while i < len(q) - 1:
                if (q[i], q[i + 1]) == best:
                    q[i:i + 2] = [q[i] + q[i + 1]]
                else:
                    i += 1
    return merges

def apply_bpe(text, merges):
    q = list(text)
    for a, b in merges:
        i = 0
        while i < len(q) - 1:
            if (q[i], q[i + 1]) == (a, b):
                q[i:i + 2] = [a + b]
            else:
                i += 1
    return q

merges = learn_bpe([t for t, _, _ in train], 12)
schemes = {"character": chars, "lexicon (FMM)": fmm, "BPE (12 merges)": lambda t: apply_bpe(t, merges)}

def aligned(tokens, ents):
    bounds, pos = {0}, 0
    for t in tokens:
        pos += len(t); bounds.add(pos)
    return sum(s in bounds and e in bounds for s, e, _ in ents), len(ents)

print("training sentences, English gloss, gold entities:")
for s, ents, gloss in train:
    print(f"  {s} = {gloss}  entities: {[(s[a:b], t) for a, b, t in ents]}")
print("learned merges:", ["".join(m) for m in merges])
for name, f in schemes.items():
    vocab = {t for s, _, _ in train for t in f(s)}
    lens = [len(f(s)) for s, _, _ in train]
    ok = tot = 0
    for s, ents, _ in train:
        a, n = aligned(f(s), ents); ok += a; tot += n
    print(f"{name:16s} vocab={len(vocab):2d} mean tokens/sentence={sum(lens)/len(lens):.1f} "
          f"entity boundaries on token edges={ok}/{tot}")
print()
for name, f in schemes.items():
    toks = f(test_text); a, n = aligned(toks, test_ents)
    print(f"{name:16s} {' | '.join(toks)}   aligned {a}/{n}")
print()
train_chars = {c for s, _, _ in train for c in s}
for name, f in schemes.items():
    print(f"{name:16s} {' | '.join(f(new_term))}")
print("all characters of the new term seen in training:", all(c in train_chars for c in new_term[:4]))

# Japanese: the same three schemes on a few invented sentences (hiragana endings, katakana loanword name).
ja_train = [
    ("私は寿司を食べました。", "I ate sushi."),
    ("彼は東京タワーを見ました。", "He saw Tokyo Tower."),
    ("彼女は大阪へ行きました。", "She went to Osaka."),
    ("私は東京へ行きました。", "I went to Tokyo."),
    ("東京タワーは高いです。", "Tokyo Tower is tall."),
    ("彼は寿司を作りました。", "He made sushi."),
]
ja_test, ja_gloss, ja_ents = "彼女は東京タワーへ行きました。", "She went to Tokyo Tower.", [(3, 8, "LOC")]
# Stand-in lexicon with morpheme-sized entries (stem, polite marker, past marker), as a morphological analyser splits them.
JLEX = {"私", "彼", "彼女", "は", "を", "へ", "寿司", "食べ", "見", "行き", "作り", "まし", "た", "です",
        "東京", "大阪", "タワー", "高い"}
ja_merges = learn_bpe([s for s, _ in ja_train], 10)
print()
print("Japanese test:", ja_test, "=", ja_gloss)
print("Japanese merges:", ["".join(m) for m in ja_merges])
for name, f in (("character", chars), ("lexicon (FMM)", lambda t: fmm(t, JLEX)),
                ("BPE (10 merges)", lambda t: apply_bpe(t, ja_merges))):
    toks = f(ja_test); a, n = aligned(toks, ja_ents)
    print(f"{name:16s} {' | '.join(toks)}   tokens={len(toks)} aligned {a}/{n}")

Output:

training sentences, English gloss, gold entities:
  张伟在北京大学研究数学。 = Zhang Wei studies mathematics at Peking University.  entities: [('张伟', 'PER'), ('北京大学', 'ORG')]
  华为总部位于深圳。 = Huawei's headquarters is in Shenzhen.  entities: [('华为', 'ORG'), ('深圳', 'LOC')]
  李娜在北京工作。 = Li Na works in Beijing.  entities: [('李娜', 'PER'), ('北京', 'LOC')]
  他的儿子喜欢计划旅行。 = His son likes planning trips.  entities: []
  数量很多，北京大学很大。 = The quantity is large; Peking University is big.  entities: [('北京大学', 'ORG')]
  我们开会研究这个计算问题。 = We hold a meeting to study this computation problem.  entities: []
learned merges: ['北京', '研究', '大学', '在北京', '问题', '量很', '量很多', '部位', '部位于', '部位于深', '部位于深圳', '这个']
character        vocab=46 mean tokens/sentence=10.8 entity boundaries on token edges=7/7
lexicon (FMM)    vocab=27 mean tokens/sentence=5.8 entity boundaries on token edges=7/7
BPE (12 merges)  vocab=38 mean tokens/sentence=8.0 entity boundaries on token edges=4/7

character        他 | 在 | 美 | 国 | 会 | 见 | 了 | 专 | 家 | 。   aligned 1/1
lexicon (FMM)    他 | 在 | 美国会 | 见 | 了 | 专家 | 。   aligned 0/1
BPE (12 merges)  他 | 在 | 美 | 国 | 会 | 见 | 了 | 专 | 家 | 。   aligned 1/1

character        量 | 子 | 计 | 算 | 很 | 有 | 趣 | 。
lexicon (FMM)    量 | 子 | 计算 | 很 | 有趣 | 。
BPE (12 merges)  量 | 子 | 计 | 算 | 很 | 有 | 趣 | 。
all characters of the new term seen in training: True

Japanese test: 彼女は東京タワーへ行きました。 = She went to Tokyo Tower.
Japanese merges: ['まし', 'ました', '東京', '行き', '行きました', '私は', '東京タ', '東京タワ', '東京タワー', '彼は']
character        彼 | 女 | は | 東 | 京 | タ | ワ | ー | へ | 行 | き | ま | し | た | 。   tokens=15 aligned 1/1
lexicon (FMM)    彼女 | は | 東京 | タワー | へ | 行き | まし | た | 。   tokens=9 aligned 1/1
BPE (10 merges)  彼 | 女 | は | 東京タワー | へ | 行きました | 。   tokens=7 aligned 1/1

Reading the results against the question's three axes

Three points follow from the table.

1. Alignment to entity spans. Character tokens align by construction. A lexicon token can span an entity boundary when the dictionary prefers a longer word (美国会 versus the entity 美国). Subword merges are driven by frequency, so they can cross boundaries too: in this run the merges 在北京 and 部位于深圳 join a particle or verb with an entity, and 3 of the 7 entities lose their edges. A real SentencePiece model trained on a large corpus can also cross entity boundaries, and nothing in training prevents it; measure the boundary-crossing rate of gold entities on your own dev set.
2. Vocabulary size and compute. Fewer, longer tokens shorten sequences (attention cost grows with the square of sequence length) but enlarge the embedding table (one learned vector per vocabulary entry) and the output layer (the final layer that scores every vocabulary entry at each position), which scale with vocabulary size times hidden size. Character tokens are cheap in parameters and expensive in length; a lexicon is the reverse; subword lets you pick the point on that curve.
3. Out-of-vocabulary (OOV) terminology. New names, product terms and transliterated foreign words are the content that NER must find. A lexicon segmenter breaks them according to whatever shorter entries match; character and subword models can always represent them, and a byte-level fallback (available in some SentencePiece configurations: a character never seen in training is spelled out as its raw bytes, so nothing is ever unknown) removes the unknown-character case entirely.

Japanese specifics

Japanese mixes three scripts. Kanji are characters borrowed from Chinese, thousands of them, and carry the meaning of nouns and verb stems. Hiragana is a syllable alphabet of about 46 symbols that writes grammatical endings and particles. Katakana is a second syllable alphabet that mostly writes foreign words and names, for example タワー (tower), usually as one unbroken run of symbols. A dictionary segmenter handles a katakana name well only if it is in the dictionary.

The last block of the program runs the same three schemes on the Japanese sentence 彼女は東京タワーへ行きました ("She went to Tokyo Tower"), where the entity is 東京タワー. The verb 行きました ("went") is a stem 行き plus the polite past ending ました, written in hiragana; a dictionary-based morphological analyser such as MeCab splits past-tense endings into separate morphemes (its documentation shows 渡した segmented as 渡し and た), and the hand-made lexicon here is written in that style (行き, まし, た); the exact split of a polite ending depends on the dictionary. Character tokens spell the verb as five tokens, so the model has to relearn the ending every time. Subword training sees ました after 食べ, 見, 行き and 作り, so the pair まし and then ました are merged early and the ending becomes one reusable piece; with enough merges the frequent name 東京タワー also becomes one piece. That is the sense in which subword tokenization helps with verb endings: frequency teaches it the endings without a dictionary. It gives no guarantee, because the same frequency also glued the pronoun and particle in 私は into one piece. These are reasons to prefer subword tokenization with a shared multilingual vocabulary when you need Chinese and Japanese in one model: one learned vocabulary covers kanji, kana and Chinese characters without a dictionary for each language, and you verify the boundary cuts on your own dev set.

Aligning predictions back to text

Whichever tokenizer you use, keep each token's character offsets and label the first piece of a word (or every character for character models). With character models the BIO tags (Beginning, Inside, Outside) are per character and the spans are direct. With segmenters or subwords, merge pieces back into spans through the offsets, and score entity-level exact match, so a boundary cut shows up as an error.

Recommendation and what flips it

For a new NER model: start from a pretrained multilingual encoder and keep its own subword tokenizer (changing tokenizers breaks the pretrained embeddings), keep offsets, and measure boundary crossing on gold entities. If you train from scratch on a closed domain with a fixed gazetteer, a lexicon segmenter plus character features is a reasonable choice; if boundary crossing is above what your quality bar allows, fall back to character tokens and accept longer sequences. Tuning the segmenter's dictionary with your entity list reduces cross-boundary cuts but must be redone whenever the list changes.

Running the code

Python 3.12, standard library only
# save the program above to a file and run it with python

Direct answer

A translation model here is a sequence-to-sequence model: an encoder reads the source sentence and a decoder writes the target sentence. With little parallel data (sentence pairs in two languages), quality comes from using other data: a pretrained multilingual model that has already seen related languages, monolingual target-language text turned into synthetic pairs by back-translation, parallel data mined from the web and cleaned hard, and transfer from a related higher-resource language. Decide the order by expected gain per unit of cost and by dependency, and confirm each step with a controlled comparison on a trustworthy test set big enough to see the gain. The order I would use: (1) build the test set and a baseline, (2) fine-tune a pretrained multilingual translation model, (3) clean and filter all available parallel data, (4) back-translation with the best model so far, (5) related-language transfer or pivoting (translating through a third, related language), (6) targeted human translation for what the error analysis shows is missing. The numbers in the list of levers below match these numbers; step 1 is Step 0 below, and small-data regularization is a side lever that applies only when you train your own model.

Step 0: what you need before any technique

• A test set that is real, human-translated, and from the domain you will serve: a few hundred to a few thousand sentences, separate from every training source. Remove from all training data any sentence that appears in it (mined web data often contains it).
• Metrics: BLEU measures word n-gram overlap (runs of n consecutive words) between the output and a human reference, with a penalty for outputs that are too short; chrF measures overlap of character n-grams (runs of a few consecutive characters), which is less brittle for morphologically rich languages (words change form a lot) and short text. Report chrF alongside BLEU, plus a small human check of adequacy (does the translation carry the source's meaning?) and fluency (does it read as natural target-language text?) on a sample.
• A baseline: either a small transformer trained from scratch on the available pairs, or a pretrained model used as-is. The baseline sets the bar every other idea has to beat.

The levers, in the order I would try them

2. Start from a pretrained multilingual translation or sequence-to-sequence model that includes the target language or a close relative, and fine-tune on the available pairs. Cheap to try, and in the low-data regime it avoids learning everything from scratch. Risks: the target language may be missing from the pretraining vocabulary (check the tokenizer: unknown or heavily fragmented words), and fine-tuning on a tiny set can forget other abilities.
3. Clean the parallel data. Neural translation models are sensitive to some kinds of noise (misaligned pairs, untranslated copies of the source), and when parallel data is scarce the noisy pairs are a larger share of what the model learns from. The program below shows the shape of a filter with a bilingual dictionary standing in for a learned similarity model: dictionary coverage of the source content words, and a bound on the length ratio. Production filters use language ID (detecting which language a text is in), deduplication, and a trained cross-lingual similarity score instead.
4. Back-translation. Train a target-to-source model, translate abundant monolingual target text into the source language, and add the synthetic pairs, with the real target side as the output, to training. The real, human-written text is on the output side, so the decoder always learns to write good target-language text; any noise in the synthetic source only affects what the encoder reads, which is why the method tolerates some noise. Tagging synthetic sources with a marker token lets the model tell them from real data. Iterate: a better forward model makes better synthetic data. It depends on the reverse model being decent; with a poor reverse model you add noisy sources, so filter and cap the synthetic share (try a few ratios of synthetic to real).
5. Related-language transfer (or pivoting, which translates source to a related language and then to the target). Pool the data of closely related languages in a shared vocabulary (a shared script, or transliteration, which rewrites text from one writing system into another's letters, helps), train multilingually, or pretrain on the related pair and fine-tune on yours. Benefit depends on linguistic similarity; test it.

• Side lever, small-data regularization when training your own model (regularization means limiting how closely the model can fit a small training set): a smaller model, higher dropout (randomly switching off a share of units during training), label smoothing (training targets that give the correct word slightly less than all the probability), a smaller subword vocabulary and shared source and target vocabularies for related languages.

6. Human translation of targeted sentences (the domain gaps and frequent error classes found in error analysis) is slow and costly but gives the cleanest signal; spend it after cheap levers are exhausted. Using a general-purpose language model to generate synthetic pairs can add data, but its quality for the target language must be measured by the same test set; it is not assumed to be good.

Filtering, computed on a toy example

The next program applies the filter to eight candidate pairs. The dictionary and the thresholds are made up for illustration (coverage of at least 0.5, length ratio between 0.6 and 1.7). It is a minimal stand-in for a learned filter and shows how both rules fire.

import numpy as np

# ---- Part 1: a toy filter for noisy mined sentence pairs (stand-in for a learned similarity model) ----
lexicon = {"cat": "gato", "sleeps": "duerme", "sofa": "sofá", "want": "quiero", "coffee": "café",
           "weather": "tiempo", "nice": "buen", "today": "hoy", "station": "estación",
           "reads": "lee", "book": "libro", "thank": "gracias", "children": "niños",
           "play": "juegan", "park": "parque"}
stop = {"the", "a", "is", "on", "in", "i", "she", "where", "you"}
pairs = [
    ("the cat sleeps on the sofa", "el gato duerme en el sofá"),
    ("i want a coffee", "quiero un café"),
    ("the weather is nice today", "hoy hace buen tiempo"),
    ("the cat sleeps on the sofa", "compre ahora con descuento exclusivo"),
    ("where is the station", "dónde está la estación"),
    ("she reads a book", "ella lee un libro y también escribe una larga carta a su hermana en la ciudad"),
    ("thank you", "gracias"),
    ("the children play in the park", "los niños juegan"),
]
print(f"{'pair':>4} {'coverage':>9} {'len ratio':>10} keep")
kept = 0
for i, (en, es) in enumerate(pairs):
    content = [w for w in en.split() if w not in stop]
    hits = sum(lexicon.get(w, "\0") in es.split() for w in content)
    cov = hits / len(content)
    ratio = len(es.split()) / len(en.split())
    keep = cov >= 0.5 and 0.6 <= ratio <= 1.7
    kept += keep
    print(f"{i:>4} {cov:>9.2f} {ratio:>10.2f} {'yes' if keep else 'no'}")
print(f"kept {kept} of {len(pairs)}")

# ---- Part 2: how large a test set is needed to see a small gain from one idea? ----------------------
# Assumed setup: per-sentence quality score (0-100) of system B minus system A has standard deviation 15;
# the true mean gain is 1.0 point; a two-sided 95% normal interval on the mean difference is used.
gain, sd, n_trials = 1.0, 15.0, 4000
rng = np.random.default_rng(0)
print(f"\nassumed true gain {gain} point, per-sentence difference sd {sd}")
print(f"{'test sentences':>15} {'interval half-width':>20} {'share of trials where the interval excludes 0':>46}")
for n in (100, 500, 2000, 8000):
    d = rng.normal(gain, sd, size=(n_trials, n))
    m = d.mean(axis=1)
    se = d.std(axis=1, ddof=1) / np.sqrt(n)
    detected = np.mean(m - 1.96 * se > 0)
    print(f"{n:>15} {1.96 * sd / np.sqrt(n):>20.2f} {detected:>46.3f}")

# ---- Part 3: ranking ideas by (expected gain x chance it works) / cost. All inputs below are assumed, for illustration. ----
ideas = [("fine-tune a pretrained multilingual model", 6.0, 0.8, 2),   # (name, gain in chrF points, chance it works, cost in person-days)
         ("clean the parallel data", 2.0, 0.9, 1),
         ("back-translation", 3.0, 0.6, 4),
         ("related-language transfer", 2.5, 0.5, 3),
         ("human translation of 5,000 sentences", 4.0, 0.9, 20)]
print("\nidea (assumed gain, chance, cost)                       score = gain x chance / cost")
for name, g, p, c in sorted(ideas, key=lambda t: -t[1] * t[2] / t[3]):
    print(f"{name + f' ({g}, {p}, {c})':<55} {g * p / c:>8.2f}")

Output:

pair  coverage  len ratio keep
   0      1.00       1.00 yes
   1      1.00       0.75 yes
   2      1.00       0.80 yes
   3      0.00       0.83 no
   4      1.00       1.00 yes
   5      1.00       4.00 no
   6      1.00       0.50 no
   7      0.67       0.50 no
kept 4 of 8

assumed true gain 1.0 point, per-sentence difference sd 15.0
 test sentences  interval half-width  share of trials where the interval excludes 0
            100                 2.94                                          0.104
            500                 1.31                                          0.318
           2000                 0.66                                          0.846
           8000                 0.33                                          1.000

idea (assumed gain, chance, cost)                       score = gain x chance / cost
fine-tune a pretrained multilingual model (6.0, 0.8, 2)     2.40
clean the parallel data (2.0, 0.9, 1)                       1.80
back-translation (3.0, 0.6, 4)                              0.45
related-language transfer (2.5, 0.5, 3)                     0.42
human translation of 5,000 sentences (4.0, 0.9, 20)         0.18

Pair 3 is boilerplate with the wrong content (coverage 0; its Spanish side says "buy now with exclusive discount"), pair 5 pairs a short English sentence with a very long target sentence (ratio 4.00; the Spanish side says "she reads a book and also writes a long letter to her sister in the city"), pair 7 is a truncated target (ratio 0.50). Pair 6 (thank you and gracias) is a correct translation that the length rule rejects, a reminder that thresholds must be tuned on a labeled sample, with attention to short sentences. The filter keeps 4 of 8.

Deciding what to try first

Rank the ideas by (expected gain x chance it works) / cost, and let the dependencies set the order. The third table in the output applies this rule to five ideas with assumed inputs (expected chrF gain in points, chance it works, cost in person-days): fine-tuning scores 6.0 x 0.8 / 2 = 2.40, cleaning 2.0 x 0.9 / 1 = 1.80, back-translation 3.0 x 0.6 / 4 = 0.45, related-language transfer 2.5 x 0.5 / 3 = 0.42 and human translation 4.0 x 0.9 / 20 = 0.18. The inputs are guesses you replace with your own estimates and a first quick experiment; the table shows the mechanics, not a measurement. Dependencies then adjust the order: filtering and the pretrained starting point come before back-translation because back-translation needs the best possible reverse model. Then verify each with a controlled experiment: change one thing, keep the same test set, report the difference with an interval. The second table in the output shows why test set size matters. Under its stated assumption (per-sentence score differences with a standard deviation of 15 points and a true gain of 1 point) a 95% interval on the mean difference (the measured gain plus or minus the half-width shown in the table, which is 1.96 standard errors) excludes zero, meaning the gain is distinguishable from noise, in about 10% of trials with 100 sentences, 32% with 500, 85% with 2,000 and essentially every trial (the printed share is 1.000) with 8,000. A set of a few hundred sentences cannot tell a one-point gain from noise, so decisions about small gains need a larger set, a paired bootstrap on the real per-sentence scores (resample the test sentences with replacement many times, compute system B minus system A on each resample, and count how often B comes out ahead), or human evaluation. The numbers come from the simulation's assumptions, not from translation data.

Failure modes

• Test leakage from mined or synthetic data; deduplicate against the test set.
• Back-translation adding noisy sources: the synthetic text sits on the input side and the real, human-written text on the output side, which makes the method tolerant of some noise, but a weak reverse model still teaches wrong input patterns. Cap the synthetic share and measure.
• Domain mismatch: monolingual text from news does not help a medical product.
• Metric gains that are not quality gains: BLEU and chrF can disagree, so check both and a human sample.
• Script, normalization and tokenization differences for the target language silently capping quality.