Privacy by Design and Default Questions

Direct answer. Use a cascade: cheap, high-precision rules run on everything, a machine-learning (ML) named entity recognition (NER) model (software that reads text and marks the spans that are names of people, places or organizations) runs only where rules are uncertain or risk is highest, and a confidence score decides between pass, mask and quarantine. Start in shadow mode (detect and report only, change nothing), measure precision (of the lines flagged, the share that really hold PII) and recall (of the lines that really hold PII, the share flagged) on labelled data (lines a human has marked as PII or not), then turn on enforcement one service at a time.

Why not run the model on everything. At an illustrative 500,000 log events per second at 1 KB each, that is 0.5 GB per second, or about 43.2 TB per day. Running a model on every line at that scale is expensive, so the cascade does the cheap work first.

Pipeline

1. Stage 1, regex with validators (stream processor, high precision): patterns for email, phone, national ID formats and card numbers. Add validators so the pattern is not enough alone: a Luhn checksum for card numbers, and for formats with no checksum a structural check (an email needs a plausible domain; a US Social Security number shape such as 123-45-6789 is rejected if its area part is 000, 666 or 900 to 999, which are never issued). The rule: from the right, double every second digit (subtracting 9 if the result exceeds 9), add all digits, and the total must be divisible by 10. 4111 1111 1111 1111 sums to 30 and passes; 4111 1111 1111 1112 sums to 31 and fails. This rejects most random 16-digit numbers such as order IDs. Keywords like ssn= or password add context.
2. Stage 2, NER on a subset: names, addresses and free-text identities have no clean pattern, so the NER model scores them: given "called Jane Doe from Leeds", it marks "Jane Doe" as a person and "Leeds" as a place, with a probability for each. Send it lines that have candidate tokens, lines from high-risk services, and a random sample from everything for measurement.
3. Stage 3, decision by combined confidence. One simple combination rule: each detector gives the line a score (a regex hit that passes its validator scores 0.95, a regex hit that fails scores 0.30, NER gives its own probability) and the line takes the highest score. Illustrative walk: ssn=123-45-6789 retry=3 scores 0.95 from regex, so it is quarantined. user Jane Doe timed out has no regex hit and NER scores the name 0.70, so it is masked and queued for review. cache miss for key 4929 scores 0.10 and passes. Thresholds (values illustrative, tuned on labelled data):
   • 0.90 or above: quarantine. Store the line redacted in the main index, move the original to a restricted store with short expiry.
   • 0.50 to under 0.90: mask in place and add to a human review queue.
   • Under 0.50: pass.

Labelled data

• Weak labels (automatic, imperfect labels, here the regex hits with validators) to bootstrap.
• Synthetic injection: generate fake PII with a faker library and inject it into a test copy of real log formats. This gives known positives.
• Human labelling of a stratified sample (sampling each group, here each service and each detector-score band, separately so rare groups are not missed) because PII is rare in random samples.
• Active learning: the lines the model is least sure about go to analysts, and their verdicts feed the next training set.

Why sample size matters (computed). If 1% of lines hold PII, a random 5,000-line sample has about 50 positives. If the detector finds 45 of those 50, recall is 90%, but the 95% Wilson interval is roughly 79% to 96%. A Wilson interval is a standard way to turn "found 45 of 50" into the range of true recall values consistent with that result. With p = 45/50 = 0.90, n = 50 and z = 1.96, the centre is (p + z^2/2n) / (1 + z^2/n) = 0.871 and the half-width is z * sqrt(p(1-p)/n + z^2/4n^2) / (1 + z^2/n) = 0.085, giving 0.786 to 0.957 (computed in python). That range is 17 points wide, too wide to promise a recall target. The width depends on how many positives you labelled, not how many lines: with 500 positives and 450 found the same recall of 90% gives 87.1% to 92.3%, only 5 points wide. So the fix is to oversample positives (label more of the lines likely to hold PII and weight the result back) or label more.

Indexing for search. Never index the raw PII. Index detection findings: service, field path, type, count, template hash (a fingerprint of the log line's fixed text with variable values removed, so identical message shapes group together) and timestamp, so analysts can search "where are emails leaking" without the values. Tokenise the redacted log text for normal search.

Rollout

1. Shadow mode on two services: detect and tag only, measure precision and recall, review false positives.
2. Mask mode for one service, with a performance budget (added latency and CPU per host) and an off switch.
3. Quarantine for high-confidence types, expanding service by service.
4. Trend false-positive rate (people complaining useful logs vanish) and false-negatives found by canary injection weekly.

Trade-offs. A false positive hides debugging detail. A false negative leaves PII exposed. Favour recall on high-severity types (cards, government IDs) and precision on noisy ones (names). Detection is a backstop: the better fix is stopping the logging at the source.

Direct answer

I would build a user-data service as the single, narrow door to personal data: personal data is separated from the rest of the system, referenced by pseudonymous IDs (random tokens that mean nothing without the lookup) everywhere else, tagged with purpose and retention, accessed through audited APIs, and deletable through one pipeline. Regional data stays in regional stores by default, and consent, retention and deletion are first-class features, not afterthoughts. I do not decide residency law myself: I design for configurable region placement (residency: the legal question of which country's servers may hold which people's data) and counsel decides what each region needs.

Architecture

flowchart LR
  A[Clients] --> G[API gateway]
  G --> U[User-data service]
  U --> P[(PII store per region)]
  U --> K[Key service]
  U --> C[Consent and preference service]
  U --> L[(Audit log)]
  G --> S[Other services]
  S -->|pseudonymous user ID only| U
  D[Deletion orchestrator] --> U
  D --> S

One request, end to end: the billing service needs a customer's email. It holds only the token u_93f1, so it calls the user-data service with that token and purpose=billing. The service checks consent, reads the PII store in the customer's home region, returns only the billing fields, and writes an audit entry. The arrow from other services into the user-data service is that call, and it carries the token, never a name or email.

Storage

• Separation. Direct identifiers (name, email, address) live in a PII store (a database holding personally identifiable information, and nothing else). All other services hold a random user token only, and join through the user-data service, so a breach elsewhere does not expose identities.
• Pseudonymization and tenant separation. A tenant is one business customer of the product. Each tenant gets its own logical partition and its own encryption key. I require per-tenant keys (the security team specifies the key-management details) so a tenant can be deleted by destroying its key: its data is stored only in encrypted form, so once the key is gone every copy, backups included, is unreadable noise.
• Regions. Each user has a home region, data written there, and cross-region reads go through the service with logging. Where a product needs global features, replicate only non-personal or pseudonymous data.
• Storage choice for a profile service with an availability target. A replicated relational or key-value store inside the region, with deletion implemented as a soft-delete marker (the record is flagged as deleted and hidden, but still stored for a short period) followed by a hard purge (physical removal), so the service stays available while erasure completes.

Access

Only the user-data service reads the PII store. Other services call it with a purpose claim (for example purpose=billing) and receive only the fields that purpose allows (the purpose claim is the stated reason attached to the call). Humans reach data through just-in-time access (granted only when needed, then removed) that is logged. Cross-service PII handling has a cost: each call adds latency and an audit write, so I cache non-sensitive derived fields briefly and make audit writes asynchronous, but never skip them.

Consent

Consent and preferences are stored as versioned records (what was agreed, when, which text version, per purpose). Services check the consent service before using data for an optional purpose, and a withdrawal event propagates to dependents.

Retention and deletion

Each field has a retention class set at schema time. A deletion orchestrator receives a request, fans out (sends the request in parallel) to every registered store (including search indexes, caches, analytics, backups by expiry), and tracks completion per store with a deadline well within the one-month response window GDPR sets (Article 12(3), extendable by two months for complex requests; counsel confirms the rule per region). Backups are not edited, they expire on schedule and the restore procedure re-applies deletions: the orchestrator keeps a list of deletion requests (token and date), and after any restore it replays every request made since that backup was taken, so deleted people do not reappear.

Audit and redaction

An append-only audit log (entries can be added but never edited or removed) records who accessed which user's data and why. Logs and traces pass through a redaction filter that removes personal fields. The performance cost is a small per-call overhead and log volume that I would measure in load tests, not assume.

Scale example (illustrative)

At 10 million users and 10,000 requests per second, I would shard the PII store by user token (split it across machines by a hash of the token), for example 16 shards of about 625,000 users and about 625 requests per second each (10,000 / 16), keep the audit write path asynchronous through a queue, and run deletion in batches, for example 500 users per shard per run overnight, so a request does not hit every shard at peak. If each of the 10,000 reads writes one 200-byte audit entry, that is about 2 MB per second, roughly 173 GB per day (10,000 x 200 x 86,400), which is why the audit path goes through a queue and a retention period of its own.

Where I would compromise

Ownership in practice

The architecture lead owns the design, a named team owns the user-data service and deletion orchestrator, and product teams own registering their stores. Requests from users are handled by support through a tool that calls the orchestrator, so the process has an owner and a ticket trail.

Approach. Copy the event recursively and replace sensitive values in the copy. Compare keys after normalizing them (lowercase, punctuation removed) so Api-Key and api_key match one name. Walk arrays, parse JSON hidden in strings, scrub obvious patterns (emails, and bearer tokens, the long secret strings sent in an Authorization: Bearer ... header) inside free text, and stop on cycles (an object that contains itself, directly or through a chain, which would make a naive walk loop forever and crash the logger). The original is never modified. Matching is by denylist, a list of names to hide; the alternative is an allowlist, a list of the only names permitted through, discussed below.

import json, re

REDACTED = "[REDACTED]"
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
BEARER = re.compile(r"(?i)\b(bearer|basic)\s+[A-Za-z0-9._~+/=-]{8,}")


def _norm(name):
    return re.sub(r"[^a-z0-9]", "", name.lower())


def make_safe(event, sensitive, scrub_strings=True, _seen=None):
    """Return a deep copy of event that is safe to log. Never mutates input."""
    banned = {_norm(s) for s in sensitive}
    seen = _seen if _seen is not None else set()

    def walk(value):
        if isinstance(value, dict):
            if id(value) in seen:
                return "[CYCLE]"
            seen.add(id(value))
            out = {}
            for k, v in value.items():
                out[k] = REDACTED if _norm(str(k)) in banned else walk(v)
            seen.discard(id(value))
            return out
        if isinstance(value, (list, tuple)):
            if id(value) in seen:
                return "[CYCLE]"
            seen.add(id(value))
            out = [walk(v) for v in value]
            seen.discard(id(value))
            return out
        if isinstance(value, str):
            s = value
            if s[:1] in "{[":
                try:
                    return json.dumps(walk(json.loads(s)))
                except ValueError:
                    pass
            if scrub_strings:
                s = EMAIL.sub("[EMAIL]", s)
                s = BEARER.sub(r"\1 " + REDACTED, s)
            return s
        if isinstance(value, (int, float, bool)) or value is None:
            return value
        return "[UNSERIALIZABLE:" + type(value).__name__ + "]"

    return walk(event)


event = {
    "msg": "login failed for ana@example.com",
    "user": {"id": 7, "Email": "ana@example.com", "Api-Key": "k-123"},
    "headers": {"Authorization": "Bearer abcdef123456", "X_Trace": "t1"},
    "items": [{"password": "p1"}, {"note": "ok"}],
    "payload": "{\"token\": \"t-9\", \"n\": 1}",
    "blob": b"raw",
}
event["self"] = event  # a cycle: the event contains itself
snapshot = repr({k: v for k, v in event.items() if k != "self"})
safe = make_safe(event, {"email", "api_key", "password", "token", "authorization"})
print(json.dumps(safe, indent=1))
print("original untouched:", repr({k: v for k, v in event.items() if k != "self"}) == snapshot)

Running it prints: msg becomes login failed for [EMAIL]; Email and Api-Key are [REDACTED]; Authorization is [REDACTED]; the password inside the list is [REDACTED]; the embedded JSON string comes out as {"token": "[REDACTED]", "n": 1}; blob becomes [UNSERIALIZABLE:bytes]; self becomes [CYCLE]; the last line is original untouched: True.

Cases where naive redaction fails

• Key variants. Exact-match on password misses Password and Api-Key versus api_key; the normalized match above fixes those, but a name like user_password still needs a suffix or substring rule that you add and test for false positives. For example, treat a key as sensitive if its normalized form ends with a banned name: _norm("user_password") is userpassword, which ends with password, so it is hidden; but passwordhint would not match a suffix rule, and a substring rule would also hide an innocent key such as token_count, which is the false positive to test for.
• Nesting and arrays. Top-level-only loops miss items[0].password.
• Mutation. Redacting in place destroys the caller's data or leaks through shared references.
• Hidden JSON. A body or payload logged as a string.
• PII in values (personal data, here meaning anything identifying a person, such as an email or phone number). The sensitive part is in the message text, not a field name. Regexes help but have false negatives (phone numbers, names) and false positives (a git remote such as git@github.com:org/repo.git, where git@github.com matches the email pattern; and the basic branch of the bearer pattern above, which turns ordinary text such as "basic information" into "basic [REDACTED]", so keep that branch only if the header form needs it or require the Authorization: prefix).
• Headers and bodies. Middleware must handle header maps and request bodies, not just the top-level event.
• Cycles, huge or unserializable values. Crash the logger or the request.
• Denylist drift. A new field added next month is not on the list. For high-risk services, log only allowlisted fields.
• URLs. Tokens in query strings are not covered by the code above: parse and scrub URL query parameters too.

Complexity. Time and memory are O(n) in the size of the event. Recursion depth equals nesting depth, so cap depth for hostile input (Python's default recursion limit is about 1,000 frames).

Node.js version. The same walk applies: build new objects, normalize keys, and accept a mask function per field instead of one constant. For a card number 4111111111111111, full redaction gives [REDACTED], last-four gives ************1111, and a plain hash gives a fixed fingerprint (SHA-256 of that number starts 9bbef194...), but card numbers are a small guessable space, so anyone can hash every candidate and match it: a plain hash does not hide the value. Use a keyed hash (HMAC with a secret key held outside the logs); it still lets you check that two events refer to the same value, and without the key it cannot be enumerated.

Direct answer. Run three scanners on every pull request, one per artifact type (code, infrastructure definitions, data schemas), each looking for a new personal-data flow, and make them useful by starting advisory, scanning only the diff, and promoting a rule to blocking only after its measured precision is acceptable. CI/CD is the automated build and deploy pipeline; IaC means infrastructure as code, such as Terraform or Kubernetes manifests. A classification catalog is the list that says which fields count as personal, sensitive or public.

What each layer checks

A new personal-data flow is then recorded: the check updates a machine-readable data inventory file in the repo, for example:

- field: users.date_of_birth
  classification: personal
  purpose: age verification
  retention_days: 365
  stores: [postgres.users]

A changed inventory routes the pull request to the privacy reviewer through code ownership rules (a file in the repository, such as CODEOWNERS, that maps paths to the people who must approve changes there).

Keeping false positives from training people to ignore it

• Advisory first. Post findings as pull request comments, record whether each is accepted or dismissed, and compute precision per rule (the share of a rule's findings that engineers accept as real). For example, a rule posts 40 comments in a month, 34 accepted and 6 dismissed as wrong: precision is 34/40 = 85 percent. Block only rules above a bar the team agreed in advance, such as 90 percent over at least 30 findings (illustrative), so this rule stays advisory until tuned. After excluding test fixtures it posts 28 comments with 27 accepted, which is 96 percent. That clears the precision figure but not the sample minimum (28 is below 30), so the rule stays advisory for a few more weeks, and it is promoted to blocking once it reaches 30 labeled findings with precision still at or above 90 percent.
• Diff-only scanning (checking only the lines the pull request changes), so engineers never see findings from code they did not touch.
• Fail fast and cheap first: schema tag and secret-pattern checks run in seconds; deeper data-flow analysis (tracing a value from where it enters the code to where it is written or sent) runs in parallel or on merge to main.
• Specific messages with a fix: "field email has no classification; add classification: personal and a purpose" instead of "privacy violation".
• Suppressions need a reason and an expiry, and are reviewed monthly; a rule with many suppressions is rewritten or retired.
• Rule owner and a response time for reports of wrong findings.

Pitfall. Pattern matching on names misses renamed fields and flags harmless ones, so treat the schema tag as the source of truth and the name pattern as a nudge to add the tag.

Direct answer

Privacy by design means making privacy a property the system has from the first design meeting, not a review step added before launch. "Personal data" is any information that relates to an identifiable person (an email, a device ID, a location trail). The practical test: if the team deleted the privacy policy tomorrow, would the product still collect, keep and expose as little as it needs to? The most widely cited framework is Ann Cavoukian's seven foundational principles. Article 25 of the GDPR (the EU General Data Protection Regulation) turned the idea into a legal duty of "data protection by design and by default"; its second paragraph requires that, by default, only the personal data necessary for each specific purpose is processed, which is principle 2 below written as law. What matters day to day is what each principle changes in a ticket or a design doc.

The seven principles as real decisions

Worked example: a "find friends" feature

A team wants to upload address books to suggest friends. Principle 1 triggers a design question before code, for example this line in the design doc template: "What personal data does this add, and what is the least we can collect?" Principle 2 means the upload is off until the user taps an explicit prompt. Principle 3 means contacts are matched and then discarded rather than stored. Principle 4 keeps the feature: suggestions still appear. Principle 5 sets a deletion step for the matching data. Principles 6 and 7 add a plain explanation of what happens to contacts and a way to withdraw.

Pitfalls

• Treating the seven principles as a poster. If a principle cannot be tied to a ticket, a default or a test, it is a slogan.
• Equating privacy with security. Security protects data from outsiders; privacy also asks whether you should hold the data at all.
• Claiming "privacy by design" because of a compliance checklist run at the end, which is the opposite of principle 1.