Privacy by Design and Default Questions

Direct answer. Keep personal data out of the immutable store in the first place, and where it must be there, protect it so it can be made unreadable later. The best design is to log pseudonymous IDs and encrypt any personal fields with a per-user key held outside the log; erasure then means destroying the key (crypto-shredding), which leaves the log's hash chain and integrity intact. Redaction by rewriting works only when the store is not truly write-once.

Why the conflict exists. The GDPR right to erasure (Art. 17) has exceptions, including processing needed for a legal obligation and for establishing, exercising or defending legal claims (Art. 17(3)(b) and (e)). Some audit logs may qualify; whether a given log or backup does is a decision for counsel and the DPO (data protection officer, the person an organization appoints to advise on and monitor data protection compliance), not engineering. Where no exception applies, you need a technical way to honor erasure.

What "tamper-evident" means here. Each log entry stores a hash of its own content plus the previous entry's hash, which forms a hash chain (a hash is a short fingerprint of data, and any change to the data produces a different fingerprint). Edit one old entry and its fingerprint changes, which no longer matches what the next entry recorded, so the tampering shows. That is why rewriting a log to delete personal data counts as tampering unless you re-seal it, meaning recompute and re-sign every later fingerprint, which is exactly the privilege an immutable log is meant to remove. Crypto-shredding is the technique that makes the conflict go away, because it erases without touching a single stored byte.

A worked example (runnable, standard library only; the cipher is a toy for the demo, use a vetted library such as AES-GCM in production).

import hashlib, hmac, json

def keystream(key, nonce, n):
    out, i = b"", 0
    while len(out) < n:
        out += hmac.new(key, nonce + i.to_bytes(4, "big"), hashlib.sha256).digest()
        i += 1
    return out[:n]

def crypt(key, nonce, data):  # XOR with keystream: same call encrypts and decrypts
    return bytes(a ^ b for a, b in zip(data, keystream(key, nonce, len(data))))

keys = {"u_7f3a": hashlib.sha256(b"demo-key-u_7f3a").digest()}  # held by the key service
nonce = (0).to_bytes(8, "big")
blob = crypt(keys["u_7f3a"], nonce, b"jane.doe@example.com")
prev = "0" * 8
h = hashlib.sha256((prev + "u_7f3a" + "login" + blob.hex()).encode()).hexdigest()[:8]
entry = {"seq": 0, "user": "u_7f3a", "event": "login", "pii": blob.hex(), "prev": prev, "hash": h}
print(json.dumps(entry))
print("before:", crypt(keys["u_7f3a"], nonce, bytes.fromhex(entry["pii"])).decode())

del keys["u_7f3a"]  # erasure: destroy the key
check = hashlib.sha256((prev + "u_7f3a" + "login" + entry["pii"]).encode()).hexdigest()[:8]
print("after: key gone, entry unchanged, chain valid:", check == entry["hash"])
redacted = hashlib.sha256((prev + "u_7f3a" + "login" + b"REDACTED".hex()).encode()).hexdigest()[:8]
print("rewrite instead: new hash", redacted, "!= stored", entry["hash"])

Output:

{"seq": 0, "user": "u_7f3a", "event": "login", "pii": "c3fa910e37d893fcf804e5dbe612983d19f63572", "prev": "00000000", "hash": "1a6b5f66"}
before: jane.doe@example.com
after: key gone, entry unchanged, chain valid: True
rewrite instead: new hash 163c0af8 != stored 1a6b5f66

The stored line never changes. Before erasure the key turns the pii field back into the email. After the key is destroyed the same bytes stay in the log, the fingerprint still matches, and the email is unrecoverable. Redacting by rewriting the field produces a different fingerprint, which breaks the chain.

Three approaches compared.

Recommended design.

1. Minimize at the source: logs carry a pseudonymous user ID and event type; names, emails and IP addresses go through a tokenization step (swap the value for a random placeholder, with the real value kept in a separate protected lookup table) or are dropped.
2. Envelope fields: where personal data must appear, encrypt just that field (the envelope) with a per-user data key, one secret key per person as in the demo. A key service (a separate, access-controlled system that stores and hands out keys) holds it with its own audit trail.
3. Erasure procedure: verify the request, check exceptions with legal, delete the user's key from the key service and its backups, and record a signed erasure event (user hash, time, scope). Verified by trying to decrypt a sample entry.
4. Backups and WORM: apply the same keys, so shredding covers backups; for older WORM data with plaintext, let it expire on its retention date and document it, since you cannot change it.
5. Integrity: log entries are hashed over the ciphertext, so shredding does not break the chain.

End-to-end encrypted backups where only the user holds keys. Here the provider cannot read the data at all, because it stores only ciphertext (data scrambled by encryption) and never has the key. Erasure at design level means deleting the encrypted blobs the provider holds. Where blobs sit in immutable storage and cannot be deleted, the user's key is the control: if the user discards it and the provider can never obtain it, the ciphertext is unreadable, which is the same crypto-shredding as above with the key held by the user instead of a key service. The provider documents this. Whether unreadable ciphertext counts as erased is a legal judgment, since encrypted data can still be personal data if the key exists somewhere.

Trade-offs. Per-user keys cost key management; not all logs need to be immutable; shorter retention reduces the problem more than any technique.

Direct answer. Treat the feature store as a governed catalog: every feature carries metadata (sensitivity class, allowed purposes, source data, retention), access policy reads that metadata at request time, and user-keyed lineage lets a deletion or consent change reach the online store, offline store and training datasets. A feature store is a shared service that computes, stores and serves model input features, with an online store for low-latency serving and an offline store for training.

1. Sensitivity tracked per feature
(An entity ID is the key a feature is stored under, here the user's ID. Lineage is the recorded chain of which source tables fed which feature and which features fed which training set. A manifest is a saved list that records exactly what went into one training set.)

• Registering a feature requires: owner, source datasets, classification (for example public, personal, sensitive), allowed purposes, retention.
• Derived features inherit the highest sensitivity of their inputs unless a reviewer records why it is lower. Example: avg_night_heart_rate_30d, derived from health data, is classed sensitive-health, allowed for sleep-coach, denied for ads-ranking.
• Lineage (which tables fed which feature) is stored, so reclassifying a source cascades to its features.

2. Access follows sensitivity

• Read policy checks the caller's declared purpose against the feature's allowed purposes, for training jobs and for online serving, and logs every read.
• A model that requests a denied feature fails at training-set build time, not in production.

3. Deletion and consent propagation

• Keys are the user (entity) ID. A consent service publishes events on withdrawal or deletion.
• Online store: consume the event and delete or invalidate the user's keys quickly, or check a consent flag at read time. Real-time enforcement suits serving. Worked example: user 8841 withdraws consent for ads-ranking. Within minutes the flag ads-ranking: withdrawn takes effect on the online path: keys for features whose only allowed purpose is ads-ranking are deleted for 8841, while keys for a feature shared with a purpose the user still allows (such as sleep-coach) stay and are simply refused to any ads-ranking read, so the next ads request gets no personalised features; offline rows are purged in the nightly job only for features with no remaining allowed purpose, and the training set built that weekend for ads-ranking carries the filter consent[ads-ranking] != withdrawn, so its manifest lists 8841 as excluded.
• Offline store: batch purge or rewrite partitioned tables on a schedule, and apply the consent flag as a join filter (when the training set is assembled by joining tables, rows for users whose flag says withdrawn are dropped) when building each training set. Batch enforcement suits training, because a training set is built at a point in time.
• Dynamic consent flags: a per-user, per-purpose flag in the store (for example user 8841, ads-ranking: withdrawn on 12 March) drives both paths, so changing a purpose does not require recomputing every feature.

4. Dataset versioning and audit trail. Every training set is saved as a versioned manifest: feature list and versions, snapshot time, consent filter applied, and the model trained on it. Auditors can see which data a model used and prove withdrawn users were excluded after a date. Deleting a user from a trained model needs a retraining policy (for example retrain within a set window); state this limit honestly.

5. Pseudonymization and noise. Keyed pseudonymous IDs (a secret key turns each real ID into a code, so analysts see codes only) reduce exposure but the data stays personal data while the key exists. Adding noise (small random changes to feature values, the idea behind differential privacy) hides any one person's contribution but costs model accuracy, and the more noise the stronger the protection; it suits aggregate statistics and sensitive features, and many feature stores never need it.

Trade-off. Strict purpose checks slow feature reuse; offset with a self-service request flow for new purposes.

Direct answer. I would run it as a 12-month program in four phases (a reduced version can stop after phase 2 at about month 5): baseline and quick wins, a pilot on two or three teams, platform-wide rollout in advisory mode, then enforcement with metrics. Checks start as warnings, only become blocking once measured as accurate, and the program is sold to engineers as less rework and to executives as fewer incidents and faster reviews. CI/CD means continuous integration and continuous delivery, the automated build, test and deploy pipeline. PII means personally identifiable information. Advisory mode means a check shows a warning and never blocks a merge; a policy gate is an automated check that can stop a deployment; a suppression is a recorded decision to ignore one finding.

Roadmap

The phase lengths sum to 2 + 3 + 4 + 3 = 12 months.

Resources (illustrative, scale to org size). One program lead (technical program manager), two platform or DevOps engineers, one privacy engineer to write and own rules, a part-time counsel or privacy lead for policy questions, and a named champion per pilot team. Compliance owns the written standard and the exception record; engineering owns the checks.

Pilot plan. Pick teams with different stacks, one that ships often. Run advisory-only for several weeks, label every finding as a true positive (the finding is real) or a false positive (it is wrong), and set a go-forward bar before the pilot, for example precision of at least 90 percent (at least 9 in 10 findings are real) over at least 50 labeled findings (illustrative). A rule with 60 findings of which 51 are confirmed has precision 51/60 = 85 percent, so it stays advisory and is tuned; after tuning, 48 confirmed out of 50 is 96 percent and it clears the bar. Pilot exit means the bar is met or the rule is cut.

KPIs

• Coverage: share of repos and pipelines running the checks.
• Precision: share of findings engineers confirm as real, per rule.
• Mean time to fix a finding, and number of suppressions and their expiry.
• Privacy bugs found in production or review versus found in CI (leading indicator).
• Pipeline time added, to prove it does not slow delivery.

Buy-in. Executives: frame as reduced incident and rework risk, with the pilot data as proof. Engineers: findings in the pull request, clear fix guidance, a fast exception path, and a rule that is wrong gets fixed within days. Guard against a tool that engineers learn to bypass.

Risks. Blocking too early causes workarounds; a program with no platform owner stalls after the pilot.

Direct answer. Treat it as a habit and tooling problem, not a discipline problem. Engineers log IDs because the logs are their fastest debugger, so the change must give them a better way to debug before it takes the old way away. I would sequence it: stop the leak this week, replace the practice over the next weeks, then enforce with guardrails so it stays fixed.

Why it matters. A user ID that can be linked to an account is personal data, usually pseudonymous (it stands in for a name but can still be linked back to a person) rather than anonymous (not linkable at all), and logs are typically retained widely and readable by many people. The risk is broad access, long retention and copies in third-party log tools. A log index is the searchable collection a log tool keeps, and each index has its own retention and access list.

Immediate fixes (first week)

• Measure first: scan a day of logs per service for IDs, emails and tokens and rank services by volume, so the first fixes go where the most data is. A week-1 result might look like this (illustrative): checkout 1,840 hits per day, mostly emails; search 310; notifications 95; every other service under 20. The first fixes then go to checkout.
• Shorten retention on the worst log indexes and restrict who can search them.
• Stop the new logging: a lightweight change in the shared logging library that masks fields named email, user_id, phone, plus tokens.
• Do not shame anyone. Announce the why, the date and the replacement.

Habits (weeks 2 to 6)

• Give people a better way to debug. Log a request ID or trace ID (a random value per request) instead of the user. Support passes along the user's request ID; engineers follow it across services.
• Where the same user must be followed across requests, log a keyed hash of the ID (HMAC, a hash made with a secret key). A plain hash of an email can be reversed by guessing, so the key matters and sits with a small group. A privileged lookup tool resolves it for a ticket: support enters the user named on the ticket, the tool recomputes that user's keyed hash and shows whether it matches the user_ref in the logs, and it records who asked and the ticket number.
• Add a pull request checklist item, "does this log line contain user data?", and run a short session using real log examples.
• Before and after example:

BEFORE: payment failed for user 48213 (jane.doe@example.com) card ending 4242
AFTER : payment failed request_id=7f3a9c2e user_ref=hmac:9d1e4b amount_bucket=50-100 reason=card_declined

Long-term guardrails

1. Typed logging: the logger accepts declared fields only, for example log.info("payment_failed", request_id=rid, amount_bucket="50-100"). A call that adds user_email=... is rejected at build time (by the type checker in a typed language, or by a schema check in CI where the language cannot enforce it) because that field was never declared, so a free-form user field cannot be added casually.
2. CI (continuous integration) lint that flags log calls with banned field names, and a pipeline scrubber (a filter between the application and log storage) that drops or masks PII (personally identifiable information) patterns such as email addresses before storage.
3. Retention tiers: debug logs for days, audit logs separate and locked down.
4. A weekly count of scanner hits by team, reviewed with managers. Success is that count trending to near zero, with exceptions tracked.
5. An owner for the logging standard, and an exception process that ends in an expiry date.

What would change my call. If a regulator inquiry or an incident is already open, the first week becomes purging and evidence preservation, with counsel involved. If debugging time visibly worsens, fix the tooling before tightening the rules, or engineers will route around it.

Direct answer

My default is that raw keystrokes and typed text never leave the device automatically; the one exception is a snippet the user has seen and chosen to send. The suggestion model is trained and evaluated against data that stays local, and the only thing the company receives is a small set of aggregate quality counters, from users who opted in. Typing is some of the most sensitive data a phone produces (passwords, messages, health searches), so "collect it and filter later" is not acceptable.

What to collect, if anything

Collect counters about the feature, not the content:

• Suggestions shown, accepted, rejected and manually corrected, per model version.
• Latency bucket of suggestion generation and crash or out-of-memory events (the keyboard being shut down by the phone for using too much memory).
• Coarse context: language, keyboard layout, app version, device class.

Nothing about which app, field or text was typed. The keyboard also excludes password fields, payment fields, and any field marked secure or incognito by the OS and host app, so those events are never even generated.

What stays on the device

• Typed text, n-gram history (counts of short runs of consecutive words, such as "see you soon", used to predict the next word), personal dictionary, and the learned personalization state.
• Per-user evaluation: the device takes its own recent typing, hides each next word, asks the old and the new model to predict it, and records how often each was right, for example old 2 of 10 and new 3 of 10. That result stays on the device and only feeds the daily counters.
• Raw per-user history is purged on a schedule and when the user clears keyboard data.

How to measure whether the model is improving

A worked example with illustrative numbers: on model v1, devices report 10,000 suggestions shown and 2,600 accepted, so acceptance rate is 26.0%. On v2, 10,000 shown and 2,900 accepted, so 29.0%. Compare the two as a randomized split of opted-in devices (devices get v1 or v2 at random, not by user type). Without randomization, v2 might go mostly to newer phones or heavy typists, whose acceptance rate would differ for reasons unrelated to the model; random assignment makes the difference come from the model and not from who is typing. Also track a guardrail (a second metric that must not get worse while you chase the first), the correction-after-accept rate (users accept then immediately delete), because a model can raise acceptance while being worse. Only counters leave the device, each as a daily total per device, and the server reports a metric only when a cohort has enough devices that no single device dominates (for example 1,000 or more, illustrative).

There are also optional, deeper options: training across devices without centralizing text (federated learning: each phone trains on its own typing and sends back only a model update, never the text) and adding mathematical noise to reports (random noise added to each report so that no single person's value can be recovered). They add real protection but carry their own cost and design effort (federated learning needs enough opted-in devices and update infrastructure, and added noise lowers the accuracy of small counters), so I would bring in the privacy engineering team to design them rather than improvise them inside the feature.

What I would refuse

Uploading keystroke logs or typed sentences "for debugging" without the user seeing and approving the exact text, collecting by default without a clear opt-in, joining keyboard telemetry to an account or advertising ID, and keeping full text samples from crash reports.

Trade-offs

On-device-only evaluation means slower iteration and no ability to read failure examples, so the team relies on opt-in volunteer submissions (the user sees and approves the exact snippet) and on public or licensed corpora for offline model work. What would change my call: a regulator requirement, or a limited, consented research programme with strict retention and review.