Privacy by Design and Default Questions

Direct answer

My default is that raw keystrokes and typed text never leave the device automatically; the one exception is a snippet the user has seen and chosen to send. The suggestion model is trained and evaluated against data that stays local, and the only thing the company receives is a small set of aggregate quality counters, from users who opted in. Typing is some of the most sensitive data a phone produces (passwords, messages, health searches), so "collect it and filter later" is not acceptable.

What to collect, if anything

Collect counters about the feature, not the content:

• Suggestions shown, accepted, rejected and manually corrected, per model version.
• Latency bucket of suggestion generation and crash or out-of-memory events (the keyboard being shut down by the phone for using too much memory).
• Coarse context: language, keyboard layout, app version, device class.

Nothing about which app, field or text was typed. The keyboard also excludes password fields, payment fields, and any field marked secure or incognito by the OS and host app, so those events are never even generated.

What stays on the device

• Typed text, n-gram history (counts of short runs of consecutive words, such as "see you soon", used to predict the next word), personal dictionary, and the learned personalization state.
• Per-user evaluation: the device takes its own recent typing, hides each next word, asks the old and the new model to predict it, and records how often each was right, for example old 2 of 10 and new 3 of 10. That result stays on the device and only feeds the daily counters.
• Raw per-user history is purged on a schedule and when the user clears keyboard data.

How to measure whether the model is improving

A worked example with illustrative numbers: on model v1, devices report 10,000 suggestions shown and 2,600 accepted, so acceptance rate is 26.0%. On v2, 10,000 shown and 2,900 accepted, so 29.0%. Compare the two as a randomized split of opted-in devices (devices get v1 or v2 at random, not by user type). Without randomization, v2 might go mostly to newer phones or heavy typists, whose acceptance rate would differ for reasons unrelated to the model; random assignment makes the difference come from the model and not from who is typing. Also track a guardrail (a second metric that must not get worse while you chase the first), the correction-after-accept rate (users accept then immediately delete), because a model can raise acceptance while being worse. Only counters leave the device, each as a daily total per device, and the server reports a metric only when a cohort has enough devices that no single device dominates (for example 1,000 or more, illustrative).

There are also optional, deeper options: training across devices without centralizing text (federated learning: each phone trains on its own typing and sends back only a model update, never the text) and adding mathematical noise to reports (random noise added to each report so that no single person's value can be recovered). They add real protection but carry their own cost and design effort (federated learning needs enough opted-in devices and update infrastructure, and added noise lowers the accuracy of small counters), so I would bring in the privacy engineering team to design them rather than improvise them inside the feature.

What I would refuse

Uploading keystroke logs or typed sentences "for debugging" without the user seeing and approving the exact text, collecting by default without a clear opt-in, joining keyboard telemetry to an account or advertising ID, and keeping full text samples from crash reports.

Trade-offs

On-device-only evaluation means slower iteration and no ability to read failure examples, so the team relies on opt-in volunteer submissions (the user sees and approves the exact snippet) and on public or licensed corpora for offline model work. What would change my call: a regulator requirement, or a limited, consented research programme with strict retention and review.

Direct answer. Treat the feature store as a governed catalog: every feature carries metadata (sensitivity class, allowed purposes, source data, retention), access policy reads that metadata at request time, and user-keyed lineage lets a deletion or consent change reach the online store, offline store and training datasets. A feature store is a shared service that computes, stores and serves model input features, with an online store for low-latency serving and an offline store for training.

1. Sensitivity tracked per feature
(An entity ID is the key a feature is stored under, here the user's ID. Lineage is the recorded chain of which source tables fed which feature and which features fed which training set. A manifest is a saved list that records exactly what went into one training set.)

• Registering a feature requires: owner, source datasets, classification (for example public, personal, sensitive), allowed purposes, retention.
• Derived features inherit the highest sensitivity of their inputs unless a reviewer records why it is lower. Example: avg_night_heart_rate_30d, derived from health data, is classed sensitive-health, allowed for sleep-coach, denied for ads-ranking.
• Lineage (which tables fed which feature) is stored, so reclassifying a source cascades to its features.

2. Access follows sensitivity

• Read policy checks the caller's declared purpose against the feature's allowed purposes, for training jobs and for online serving, and logs every read.
• A model that requests a denied feature fails at training-set build time, not in production.

3. Deletion and consent propagation

• Keys are the user (entity) ID. A consent service publishes events on withdrawal or deletion.
• Online store: consume the event and delete or invalidate the user's keys quickly, or check a consent flag at read time. Real-time enforcement suits serving. Worked example: user 8841 withdraws consent for ads-ranking. Within minutes the flag ads-ranking: withdrawn takes effect on the online path: keys for features whose only allowed purpose is ads-ranking are deleted for 8841, while keys for a feature shared with a purpose the user still allows (such as sleep-coach) stay and are simply refused to any ads-ranking read, so the next ads request gets no personalised features; offline rows are purged in the nightly job only for features with no remaining allowed purpose, and the training set built that weekend for ads-ranking carries the filter consent[ads-ranking] != withdrawn, so its manifest lists 8841 as excluded.
• Offline store: batch purge or rewrite partitioned tables on a schedule, and apply the consent flag as a join filter (when the training set is assembled by joining tables, rows for users whose flag says withdrawn are dropped) when building each training set. Batch enforcement suits training, because a training set is built at a point in time.
• Dynamic consent flags: a per-user, per-purpose flag in the store (for example user 8841, ads-ranking: withdrawn on 12 March) drives both paths, so changing a purpose does not require recomputing every feature.

4. Dataset versioning and audit trail. Every training set is saved as a versioned manifest: feature list and versions, snapshot time, consent filter applied, and the model trained on it. Auditors can see which data a model used and prove withdrawn users were excluded after a date. Deleting a user from a trained model needs a retraining policy (for example retrain within a set window); state this limit honestly.

5. Pseudonymization and noise. Keyed pseudonymous IDs (a secret key turns each real ID into a code, so analysts see codes only) reduce exposure but the data stays personal data while the key exists. Adding noise (small random changes to feature values, the idea behind differential privacy) hides any one person's contribution but costs model accuracy, and the more noise the stronger the protection; it suits aggregate statistics and sensitive features, and many feature stores never need it.

Trade-off. Strict purpose checks slow feature reuse; offset with a self-service request flow for new purposes.

Direct answer. The checklist fails because it is a late, separate, binary step owned by someone else, so teams optimize for passing it. I would move privacy into the existing workflow: design docs, tickets, code review, pipelines and retrospectives, and let the review gate become a small escalation path for genuinely new risk rather than the main control.

What is wrong with the current setup (the critique)

• Review at the end finds problems after the architecture and data model are fixed, so fixes are expensive and get waived.
• A checklist asks "did you do privacy?" instead of "which personal data does this feature touch and why?".
• Nobody on the team owns the answer, so privacy bugs are the review team's problem.

What I would change, in order

1. Start from the bugs. Classify the last few quarters of shipped privacy bugs (for example PII in logs, features trained on data without a purpose check, missing deletion path). Build the first changes around the top classes, so the effort answers a real pain.
2. Put the questions in the design doc template. A short section: what personal data, what purpose, retention, who can access, what happens on user deletion or consent withdrawal. A design is not review-ready with it blank. For ML, add: training data sources, whether labels or outputs are personal data.
3. Add to the definition of done. Stories touching personal data carry acceptance criteria (for example "event schema tagged with data classification", "deletion path tested"), reviewed like any other criterion.
4. Automate the repeatable. Pipeline checks for PII in logs and untagged schema changes catch the common mistakes at pull request time, so humans spend time on judgement.
5. Name a privacy champion per team (an engineer with a few hours a week, not a new role), trained by the privacy team, who is the first reviewer and escalates only new patterns.
6. Close the loop. Every privacy bug gets a blameless retrospective and a new automated check or template question.

Reasoning on ownership. The engineering manager owns the outcome, the privacy team owns the standard and the tooling, and the gate owns only escalations such as a new sensitive data category.

Measures. Privacy bugs found before merge versus after release, time from first design doc to privacy sign-off, and share of designs that arrive with the section filled in.

What would change my call. If most bugs come from one shared platform, fix it there instead of training every team.

Direct answer. Treat training data as a governed asset with a lifecycle: collect only what the model needs, label it with provenance, retain it for a defined period, honor deletion by removing it from datasets and scheduling retraining, and keep manifests so you can show which model was trained on which data. Be honest that removing a person's data from a trained model is hard: a model stores what it learned as numbers (weights) spread across the whole model, not as rows you can look up, so deleting the training row does not remove its influence. There is no cheap, reliable way yet to subtract one person from the weights, so the strategy relies on retraining cadence and on avoiding personal data in training where possible.

Lifecycle stages.

1. Collection and consent: record the purpose and lawful basis (the legal ground for using the data; a question for counsel) at ingestion. Exclude users who opted out of training use. Collect fewer fields: drop or tokenize direct identifiers before they reach the training store.
2. Labeling: labelers see the minimum, through access-controlled tools; vendors are bound by contract; labels link to records by pseudonymous ID, not names.
3. Storage and retention: the raw store, the processed features and the training snapshots each have a retention class. Raw data expires sooner than the derived snapshots only if the derived ones no longer identify anyone.
4. Dataset manifests: every training run records a manifest: the list of record IDs (pseudonymous), schema and filter version, and code commit, plus a hash of the list. This gives auditable reproducibility and lets you answer "was this user's data in model v7?" by looking the ID up in the stored list, without keeping the underlying data forever. The hash alone cannot answer membership; it only proves the stored list has not been altered since training. An illustrative entry: {model: "v7", trained: "2026-04-12", snapshot: "snap-0412", id_list_hash: "5b1e...", filter_version: "f19", code_commit: "a3c91e7"}, where the hash is a short fingerprint that changes if any listed ID changes (the full ID list sits in the manifest store under snap-0412).
5. PII audit: scan datasets before training for emails, phone numbers, ID numbers and secrets using detectors plus sampled human review; block runs that fail; scan model outputs for memorized personal data in evaluation (a model can sometimes reproduce rare training text, such as an email address, word for word).
6. Deletion on request: delete the user's rows from raw data, features and snapshots using the manifest index. Persisted artifacts such as embeddings (lists of numbers that represent a text or user so models can compare them), cached features and fine-tuned weights (the model's adjustable numbers after extra training on your data) are tracked as derived data too: delete derived features directly; for weights, record the request and remove the user in the next scheduled retrain.
7. Model versioning: each model version links to its manifest; versions older than a defined age are retired so no stale model keeps data in it indefinitely.

Deletion-aware retraining cadence. Choose a cadence (for example, retrain quarterly and always before a model is relaunched) where each retrain excludes everyone who has deleted or opted out. In between, apply an output-side suppression filter (a check that stops the model from returning data about people on the pending-deletion list) and keep that list. Whether an interim lag is acceptable is a legal and risk decision, and you state the maximum lag to users.

In practice the usual default is periodic retrain plus suppression; the other rows apply when the deletion lag or the cost is unacceptable. Sharded training example: split users into 10 slices, each with its own sub-model, combined when serving; a deletion from slice 4 retrains only slice 4, about one tenth of the compute of a full retrain.

Worked example. A model is trained on 10 million user records. Over a quarter, 4,000 users request deletion (0.04%). The manifest finds their IDs, the next scheduled retrain excludes them, and the old model version is retired 30 days after rollout, so no model trained with them stays in production beyond about one retrain cycle plus 30 days. Numbers are illustrative.

Trade-offs. Faster deletion costs compute and may hurt model quality; stronger minimization may lower accuracy. As product lead, I would set the cadence with legal, publish the lag, and measure it.

Direct answer

I would not allow raw transcripts into analytics or training. I would require a documented purpose and legal basis (under GDPR you need one of a short list of lawful reasons, such as consent or legitimate interests, before processing personal data), a risk assessment, a redaction pipeline that runs before any data scientist sees text, separated access, and a retention limit. Health details are special-category data (types of personal data that GDPR protects more strictly, such as health, beliefs and biometrics) under GDPR Art. 9, so they get the strictest treatment, and if reliable redaction is not possible for a use, I would refuse that use.

What I would require first

• Purpose and lawful basis. "Improve support" is too vague. Name each use (for example topic analytics, and a model that suggests replies). Customers were told the chat was for support, so using it for model training may be a new purpose that needs its own basis, notice or opt-out. Counsel decides the basis.
• An extra condition for the health content. Special-category data needs an Art. 9(2) condition (such as explicit consent) on top of the ordinary lawful basis, and legitimate interests alone does not cover it. This is another reason to strip health text before any use.
• A DPIA (data protection impact assessment), a structured risk review, because this is large-scale, sensitive, and new technology.
• Owner and sign-off from the data owner and privacy lead before any copy is made.

Processing design

1. Detect. Combine pattern rules for account numbers and emails, a named-entity model (software trained to spot names of people, places and organisations in free text) for names, and a keyword or classifier layer for health terms. Measure recall (the share of the sensitive items that really exist which the detector catches) on a hand-labelled sample, since a missed entity is a leak, and accept that no detector is perfect.
2. Redact or replace. Swap with typed placeholders (for example [NAME], [ACCOUNT]) or consistent fake tokens (the same fake name each time the same real name appears, so a reader can follow who said what) so conversation structure survives. Before: "Hi, I'm Priya Nair, account 48213, my diabetes medication was refused." After: "Hi, I'm [NAME], account [ACCOUNT], my [HEALTH] was refused." Drop whole transcripts that contain health content if the use does not need them.
3. Separate. Raw transcripts stay in a restricted store with a few named readers. The redacted copy goes to the analytics zone. Any re-link key (the table that maps a fake token back to the real value) lives elsewhere with its own access.
4. Retain briefly. Keep raw text only as long as the redaction pipeline needs, and delete the redacted copy when the model and analysis no longer need it. Add deletion-request handling so a customer's transcripts can be located by ID.
5. Guard the model. A model trained on residual text can memorize it, so test generations for leaked names and numbers before release, and keep the training run's data version recorded.

Worked example

On a sample of hand-labelled transcripts containing 500 sensitive spans in total, say the detector finds 460 of the 500 (92% recall). The 8% left (40 spans) would flow into training. That is too many for health content, so I would add a second pass or exclude those transcripts, and only proceed when a re-measured sample meets a bar the privacy lead signs. An illustrative bar for health content is 99% or better, and the sample must be big enough to show it: if a fresh sample of 500 spans has zero misses, the 95% upper limit on the miss rate is still about 3/500 = 0.6%, so recall of roughly 99.4% or better is supported. A 92% result is nowhere near it.

When to refuse

Refuse if consent or basis cannot cover the use, if health data is essential to the text and cannot be removed, or if the team cannot show redaction quality.