Privacy by Design and Default Questions

Direct answer. Use a cascade: cheap, high-precision rules run on everything, a machine-learning (ML) named entity recognition (NER) model (software that reads text and marks the spans that are names of people, places or organizations) runs only where rules are uncertain or risk is highest, and a confidence score decides between pass, mask and quarantine. Start in shadow mode (detect and report only, change nothing), measure precision (of the lines flagged, the share that really hold PII) and recall (of the lines that really hold PII, the share flagged) on labelled data (lines a human has marked as PII or not), then turn on enforcement one service at a time.

Why not run the model on everything. At an illustrative 500,000 log events per second at 1 KB each, that is 0.5 GB per second, or about 43.2 TB per day. Running a model on every line at that scale is expensive, so the cascade does the cheap work first.

Pipeline

1. Stage 1, regex with validators (stream processor, high precision): patterns for email, phone, national ID formats and card numbers. Add validators so the pattern is not enough alone: a Luhn checksum for card numbers, and for formats with no checksum a structural check (an email needs a plausible domain; a US Social Security number shape such as 123-45-6789 is rejected if its area part is 000, 666 or 900 to 999, which are never issued). The rule: from the right, double every second digit (subtracting 9 if the result exceeds 9), add all digits, and the total must be divisible by 10. 4111 1111 1111 1111 sums to 30 and passes; 4111 1111 1111 1112 sums to 31 and fails. This rejects most random 16-digit numbers such as order IDs. Keywords like ssn= or password add context.
2. Stage 2, NER on a subset: names, addresses and free-text identities have no clean pattern, so the NER model scores them: given "called Jane Doe from Leeds", it marks "Jane Doe" as a person and "Leeds" as a place, with a probability for each. Send it lines that have candidate tokens, lines from high-risk services, and a random sample from everything for measurement.
3. Stage 3, decision by combined confidence. One simple combination rule: each detector gives the line a score (a regex hit that passes its validator scores 0.95, a regex hit that fails scores 0.30, NER gives its own probability) and the line takes the highest score. Illustrative walk: ssn=123-45-6789 retry=3 scores 0.95 from regex, so it is quarantined. user Jane Doe timed out has no regex hit and NER scores the name 0.70, so it is masked and queued for review. cache miss for key 4929 scores 0.10 and passes. Thresholds (values illustrative, tuned on labelled data):
   • 0.90 or above: quarantine. Store the line redacted in the main index, move the original to a restricted store with short expiry.
   • 0.50 to under 0.90: mask in place and add to a human review queue.
   • Under 0.50: pass.

Labelled data

• Weak labels (automatic, imperfect labels, here the regex hits with validators) to bootstrap.
• Synthetic injection: generate fake PII with a faker library and inject it into a test copy of real log formats. This gives known positives.
• Human labelling of a stratified sample (sampling each group, here each service and each detector-score band, separately so rare groups are not missed) because PII is rare in random samples.
• Active learning: the lines the model is least sure about go to analysts, and their verdicts feed the next training set.

Why sample size matters (computed). If 1% of lines hold PII, a random 5,000-line sample has about 50 positives. If the detector finds 45 of those 50, recall is 90%, but the 95% Wilson interval is roughly 79% to 96%. A Wilson interval is a standard way to turn "found 45 of 50" into the range of true recall values consistent with that result. With p = 45/50 = 0.90, n = 50 and z = 1.96, the centre is (p + z^2/2n) / (1 + z^2/n) = 0.871 and the half-width is z * sqrt(p(1-p)/n + z^2/4n^2) / (1 + z^2/n) = 0.085, giving 0.786 to 0.957 (computed in python). That range is 17 points wide, too wide to promise a recall target. The width depends on how many positives you labelled, not how many lines: with 500 positives and 450 found the same recall of 90% gives 87.1% to 92.3%, only 5 points wide. So the fix is to oversample positives (label more of the lines likely to hold PII and weight the result back) or label more.

Indexing for search. Never index the raw PII. Index detection findings: service, field path, type, count, template hash (a fingerprint of the log line's fixed text with variable values removed, so identical message shapes group together) and timestamp, so analysts can search "where are emails leaking" without the values. Tokenise the redacted log text for normal search.

Rollout

1. Shadow mode on two services: detect and tag only, measure precision and recall, review false positives.
2. Mask mode for one service, with a performance budget (added latency and CPU per host) and an off switch.
3. Quarantine for high-confidence types, expanding service by service.
4. Trend false-positive rate (people complaining useful logs vanish) and false-negatives found by canary injection weekly.

Trade-offs. A false positive hides debugging detail. A false negative leaves PII exposed. Favour recall on high-severity types (cards, government IDs) and precision on noisy ones (names). Detection is a backstop: the better fix is stopping the logging at the source.

Direct answer. Redact before upload, not after. A step on the test runner scans each log, trace and screenshot, masks what it finds, and only the redacted artifact reaches the shared reporting system. Stop PII entering artifacts at all by using synthetic test data and masking sensitive elements before a screenshot is taken. Detection and redaction are the backstop for what slips through.

Prevention first

• Test accounts use obviously fake data (for example @example.test addresses). A detector can treat that domain as an allowlist so it does not flag noise.
• The test framework masks sensitive selectors (password fields, card inputs, profile pages) before capturing a screenshot.
• Never log request headers like Authorization or cookies in test output.

Detection heuristics for logs

A log line before and after the redaction step (the card number 4111111111111111 passes the Luhn check, so it is flagged):

BEFORE: POST /signup email=jane.doe@corp.com card=4111111111111111 token=abc123
AFTER : POST /signup email=[EMAIL_1] card=[CARD_1] token=[SECRET_1]

Screenshots (machine-assisted redaction)

1. Run OCR (optical character recognition, which turns pixels into text, for example with an engine such as Tesseract) to get words with bounding boxes (the pixel rectangle around each recognised word).
2. Run the same regex rules over the recognised text.
3. For each match, burn a solid black box into the image pixels (for example, OCR finds jane.doe@corp.com on a profile page and the rectangle around it becomes opaque black in the saved file). Do not use an overlay layer or blur: overlays can be removed and light blur can sometimes be reversed.
4. If a region contains text-like pixels that OCR read with low confidence, mask the region. Failing closed (hiding when unsure) costs some readability but OCR misses low-contrast and stylised text.
5. Prefer DOM-based capture where possible. The DOM is the browser's tree of page elements, so the test framework can find an element by its selector (an address such as input[type=password]) and mask it before the screenshot, which is more reliable than guessing from pixels.

Developer override workflow

• A false positive (a harmless value redacted) can be unredacted by the artifact owner with a justification, but anything of type card, government ID or credential needs a second person's approval.
• A missed item (PII seen in a report) can be reported with one click, which re-runs the redaction with an extra rule and purges the old artifact.
• The original unredacted artifact is deleted at job end, or held in a restricted quarantine for 24 hours (illustrative) for override requests only.

Audit log. Record who or what acted, artifact ID, rule ID, PII type, action (redacted, override requested, override approved), justification and time. Never record the matched value, or the audit log becomes the leak.

Preserving debugging usefulness

• Typed, consistent placeholders: [EMAIL_1] stays the same across a test run, so repeated references still correlate.
• Keep structure: stack traces, timestamps, status codes, layout and element positions in screenshots.
• Report redaction counts in the test summary so developers know why a value is missing.

Pitfalls. Over-redaction trains people to bypass the system. OCR can miss text in images and video frames. A noisy detector with no override path gets disabled by the first frustrated team.

Direct answer

Treat accidental PII (personally identifiable information) in logs as a pipeline problem with three layers: prevent it at the source, filter it on the way out, and detect what still gets through. Keep debuggability by logging stable pseudonymous identifiers and structured fields, so engineers can still correlate events without seeing emails or tokens. Secrets such as tokens and passwords are never logged in any form, because pseudonymizing a credential does not make it safe.

Layer 1: stop it at the source

• Use structured logging with an allow-list of fields (a fixed list of field names that may be logged; anything else is dropped), so only named fields reach the log line, instead of dumping whole request objects.
• Wrap sensitive types (a Secret or Email class) whose string form is masked, so an accidental log(user) prints a placeholder.
• Provide one shared logging library that engineers use, with safe defaults, and a lint or code-review rule against raw logging of request bodies and headers.
• Turn off verbose HTTP client and framework logging that prints headers by default in production.

Layer 2: filter before storage

Put a redaction step in the logging library or the log collector (a log collector is a background program that gathers log lines from services and ships them to storage). Redact secrets entirely; replace identifiers with a keyed pseudonym so the same user maps to the same value. A keyed hash (HMAC, a hash combined with a secret key) is better than a plain hash, because plain hashes of emails can be reversed by guessing: an attacker hashes a list of likely emails and compares. Without the secret key they cannot compute the same values. (A salt is a random value stored beside each hash so identical inputs differ; here we want the same email to give the same pseudonym, so one secret key is used for every value, which some call a pepper.) The key sits in a secrets manager and is rotated by policy.

import logging, re, hmac, hashlib, sys

SECRET_KEY = b"demo-key-held-in-a-secrets-manager"
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
BEARER = re.compile(r"(?i)bearer\s+[A-Za-z0-9._~+/=-]+")

def pseudonym(value: str) -> str:
    return "u_" + hmac.new(SECRET_KEY, value.lower().encode(), hashlib.sha256).hexdigest()[:10]

class Email(str):
    def __repr__(self): return "Email([masked])"
    def __str__(self): return "[masked-email]"
    def __format__(self, spec): return "[masked-email]"

class RedactFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        msg = BEARER.sub("Bearer [REDACTED]", msg)
        msg = EMAIL.sub(lambda m: pseudonym(m.group(0)), msg)
        record.msg, record.args = msg, ()
        return True

handler = logging.StreamHandler(sys.stdout)
handler.addFilter(RedactFilter())
log = logging.getLogger("app")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.info("login failed for ana@example.com, header=Bearer abc123.def456")
log.info("login failed for ANA@example.com again")
log.info("password reset sent to bob@example.org")
u = Email("carol@example.com")
print("raw object logged: %s" % u)
print(f"f-string: {u}")

Output:

login failed for u_25f09a190b, header=Bearer [REDACTED]
login failed for u_25f09a190b again
password reset sent to u_da00333422
raw object logged: [masked-email]
f-string: [masked-email]

The Email class at the end of the code is the layer 1 wrapper: its string form is masked, so an accidental log of the object prints a placeholder in both %s and f-string formatting. The two lines about ana@example.com produce the same pseudonym (case-insensitive), so an engineer can still follow one user's failures. Pattern-based redaction is a safety net only: regexes miss formats they were not written for, which is why layer 3 exists.

Layer 3: catch what slips through

• Run a scanner over stored logs (and in CI over test logs) for email, token and ID patterns and alert on hits.
• Seed canary values (fake data planted on purpose, such as a made-up email in a test account, that should never appear in production logs) and alert if one ever appears there, because that means redaction failed.
• Treat a hit as a defect with an owner: fix the call site, then purge the affected log range.

Keeping logs useful for incidents and audits

Trade-offs

Redaction costs some CPU and can hide a detail you needed; the keyed lookup answers most of those cases. Deterministic pseudonyms make cross-event tracing possible but also allow linking a person's activity over time, so access to the key and to the lookup stays tightly limited.

Direct answer

A retention period is the answer to "how long do we have a valid purpose for this data?", not a round number. For each data class I identify the purpose, the legal or contractual minimum that forces us to keep it, and the operational need, then keep it for the longest justified of those and no longer. Deletion only counts when it is automated and verified. The periods below are illustrative starting points; the legal minimums are set by counsel or the relevant regulation for your jurisdiction.

Decision approach

1. Purpose: why do we hold it? (Purpose limitation: data collected for one purpose should not linger because it might be useful for another. For example, login IP addresses collected to detect account takeover should not be kept to build ad audiences.)
2. Legal and contractual floor (the minimum time a rule forces us to keep it): is there a rule that requires us to keep it (tax, accounting, sector law)? For instance, many countries' tax rules require invoices to be kept for a set number of years, and the number differs by country. Counsel confirms.
3. Operational need: how long is it useful for debugging, fraud, security investigation or support?
4. Set the period at the longest of the justified inputs, then decide the end state: delete, or aggregate into data that is no longer about individuals.

The three data classes

Making deletion actually happen

• Mark it at write time: store a created_at and a data-class tag, so an expiry can be computed instead of guessed.
• Automate: time-to-live (TTL, automatic expiry) settings on log stores and object storage, and a scheduled job for database rows.
• Cover copies: replicas, search indexes, the data warehouse and backups. Backups usually cannot be edited, so document that they age out on their own schedule and that restored data is re-purged.
• Verify: the job emits a count of rows deleted, and a periodic check queries for records older than the limit plus a grace margin sized to how often the purge job runs (for a nightly job, one day) and expects zero, so that rows merely waiting for the next purge do not raise a false alarm. Plant a test record with an old date and confirm it disappears.
• Own it: each data class has a named owner and the retention schedule lives in version control with a review date.

Worked example

Transaction records: purpose is billing and disputes; the floor is whatever finance and counsel confirm, say 7 years for invoices (illustrative; 7 x 365 = 2,555 days); the operational need is dispute handling, say 1 year; the longest justified input wins, so 7 years, then delete. The nightly check is "count of transaction rows older than 7 years plus 30 days", expected 0. Marketing analytics: purpose is product and campaign insight, floor none, operational need is comparing this month with the same month last year, so 13 months (12 plus one month of slack), then aggregate; the check is "count of event rows with a user identifier older than 13 months plus 1 day", expected 0.

Auth logs: purpose is security, floor is none, operational need is investigations that surface within about a month, so a 90-day raw window gives a safety margin of 60 days over 30 days. The verification query "count of auth log rows older than 91 days" (90 days plus one day of grace for the nightly purge) should return 0 every night; a non-zero result pages the owner.

Pitfalls

A policy on paper with no deletion job; one blanket period for all data; forgetting derived copies; and keeping data "in case" with no named purpose.

Direct answer. Do not try to clean the monolith in one cut. First find and classify what PII (personally identifiable information) it holds, then add lifecycle controls in the least risky order: stop new over-collection, tag data with retention rules, archive or delete the cold data in reversible stages, and only then drop columns. Each stage ships behind a switch with a checkpoint that decides go or stop.

Phases and milestone checkpoints.

1. Discover (weeks, not months). Build a data map: tables, columns, logs, caches, exports, backups, downstream copies. Tag each field with purpose, owner, sensitivity and a retention proposal. Checkpoint: owners sign the map; unknown fields are flagged, not guessed.
2. Stop the bleeding. Block new collection that has no purpose, mask PII in logs, and add a retention-class column to new records. Checkpoint: new-write audit shows no unapproved fields.
3. Soft-delete first. Records past retention move to a "pending deletion" state (hidden from the app, still restorable) for a window such as 30 days. Checkpoint: zero user-facing errors; support tickets about missing data stay at baseline.
4. Hard delete in batches. A scheduled job deletes in small keyset batches (each batch continues from the last ID it deleted instead of re-scanning the table), rate-limited so production load is protected, writing a count log. Checkpoint: row counts match the pending list.
5. Minimize the schema. Replace stored values with derived ones where possible (store age band, not birth date), then drop unused columns using expand-then-contract (add the replacement first, move every reader over to it, and only then remove the old column): stop writes, then stop reads, then drop.
6. Enforce continuously. Retention job becomes permanent, with evidence logs and an alert if data older than its class exists.

Two checkpoints are non-negotiable whatever the team size: owner sign-off on the data map (phase 1) and a proven restore drill before any hard delete (phase 4). The others can be tightened or relaxed to fit the system.

Worked example: users.birth_date. Illustrative table of 2,400,000 rows. Phase 1 tags the column: purpose "age check at signup", owner the identity team, proposed retention "not needed after signup". Before any birth date is hidden, the replacement is built first (the expand step of Phase 5): add an age_band column beside birth_date and backfill it from the existing birth_date values in batches of 2,000 rows (2,400,000 / 2,000 = 1,200 batches), because once birth dates are soft-deleted or hard-deleted there is nothing left to derive the band from. Phase 2 then switches the signup path: with the column in place, new signups write age_band (for example "25-34") and no birth date. Then run the Phase 3 job in shadow mode: it reports what it would hide and, by tracing who reads the column, finds three readers, a marketing age report, a fraud model and a finance export; the owners agree all three can use age_band, and each is switched over and verified. Only then does Phase 3 move stored birth dates of users past the retention rule to pending deletion, Phase 4 hard-delete them after the restore drill, and Phase 5 stop any remaining writes to birth_date (the signup path already stopped in Phase 2, so this means leftovers such as an admin edit form or an import job), stop reads, and drop the column. Doing the soft-delete before the readers move would hide data that three downstream consumers still need.

Minimizing disruption. Feature flags (runtime switches) per data class; shadow mode first (the job runs but only reports what it would delete, and engineers compare that to expectations); announce changes to downstream teams that read these tables before touching them; check reports, analytics and finance dependencies.

Rollback paths. Phases 1 to 3 roll back by flipping a flag or restoring the soft-deleted state. After hard delete, rollback is only a restore from backup, so the gate before phase 4 is strict: a snapshot, a restore drill proven on a copy, and sign-off from the data owner and legal for the retention period. Schema drops are the last, most irreversible step.

Testing strategy. Replay tests (re-running real production data, with personal values masked, through the new job); unit tests for the retention rule; a restore drill (actually restoring a backup onto a copy to prove it works); consumer contract tests (tests each downstream reader owns that fail if a column they depend on changes); and a canary (a first run on 1% of tenants or rows, watched before the full run). For the retention boundary, take a 365-day class with the rule "delete when age is greater than 365 days": a record created on 2025-01-01 is kept through 2026-01-01 (age 365) and deleted by the run on 2026-01-02 (age 366). Test ages 364, 365 and 366, and write the choice into the policy so the job cannot drift.

Compliance. Which retention period applies is a legal and policy decision (counsel and the data owner), including legal-hold exceptions (records a lawyer has told you to keep for a dispute or investigation). Engineering's job is to make the period enforceable and evidenced.

Trade-offs. Soft-delete windows prolong storage but buy safety; batch deletes take longer but protect uptime.