High-level goal (TPM perspective)
Enable third‑party devs to debug production safely and cost‑effectively by linking traces → request dumps → replay while protecting PII and platform security.
Architecture & components
- Ingest: API gateway tags incoming requests with stable request_id, inject trace context (W3C).
- Tracing store: distributed trace collector (OpenTelemetry) with indices by request_id, service, span attributes.
- Debug dump service: on-demand per-request snapshot of headers, payloads, span logs, metrics; stored encrypted.
- Replay sandbox: deterministic mock environment with configurable fixtures and rate limits; replays consume recorded inputs only.
- Access & auth: RBAC token service, tenant-scoped scopes, audit logging.
- Portal/API: developer portal to search by request_id, view traces, request dumps, trigger replay, request temporary log access.
PII redaction & safety
- Dual-layer redact: static schema redaction at ingest (JSON path rules + regex), plus dynamic redact at access (contextual NLP tags).
- Tokenization: store pointers to encrypted raw payloads; full raw access requires elevated justification & short-lived keys.
- Policy engine: per-tenant redact policies, GDPR/CCPA flags, automated data retention enforcement.
Cost-control
- Sampling: adaptive sampling (head-based + tail-based) with configurable per-tenant rates; auto-elevate for errors/SLAs.
- Retention tiers: hot (7–30d) for full dumps, warm (30–90d) for traces, cold (archive) for metadata; configurable by plan.
- Quotas/budgets: developer-level budgets for replays & dump retrievals; rate limits and quota alerts.
- Compression & deduplication: store diffs for repeated payloads.
Developer portal UX
- Secure sign-in with org SSO; consent and audit visible.
- Unified search: request_id → trace timeline, annotated spans, links to dump and replay.
- Inline redaction highlights with a “show masked” flow requiring just-in-time justification & MFA.
- One-click replay with sandbox config panel (env, fixture overrides), estimated cost, and required approvals.
- Usage dashboard: budgets, retention settings, sampled rate, and request audit trail.
Trade-offs & KPIs
- Trade security vs. developer speed: tokenization + justification adds friction but reduces risk.
- KPIs: mean time to resolution, percent of issues solved via replay, cost per debug session, PII access incidents.
This design balances developer productivity, compliance, and cost; next steps: define schemas for redact rules, sampling defaults by plan, and a phased rollout with pilot customers.
