**Clarify requirements & constraints**- Capture authN/authZ events across web, API, and services; support fast forensic queries (minutes), regulatory retention (e.g., 1–7 years), low cost, tamper-evidence, realtime alerting.**Structured log format**- JSON schema (compact, typed) with required fields and schema version.json
{
"ts":"2026-03-02T12:34:56Z",
"event_type":"auth.login",
"service":"api-gateway",
"user_id":"u-1234",
"principal_type":"user|service|key",
"outcome":"success|failure",
"ip":"1.2.3.4",
"client":"web:1.2.3",
"reason":"invalid_password",
"request_id":"r-uuid",
"trace_id":"t-uuid",
"raw_request":{ /* optional redacted */ }
}
- Validate at source (schema registry/JSON Schema), redact PII before transmission.**Secure transport**- Use TLS 1.3 + mTLS from services to collectors (Fluentd/Vector) over private VPC endpoints.- Backpressure + local buffering (disk) for outages; sign payloads with service key to detect spoofing.**Collectors & ingestion**- Edge collectors (Kubernetes DaemonSet or sidecars) aggregate, validate, add metadata, compute per-batch SHA256 and optional signature, forward to central pipeline (Kafka or managed Kinesis).**Storage & indexing**- Hot index: Elasticsearch/OpenSearch or ClickHouse for 90 days for fast queries, with time-based indices and mappings on event_type, user_id, ip, outcome.- Cold archive: Compressed Parquet/JSONL to S3 with lifecycle (hot -> infrequent -> Glacier) for long-term retention and compliance.- Use partitioning by date + hashed service to accelerate scans; maintain inverted indices for common forensic fields.**Tamper-evidence / immutability**- Write-once S3 with Object Lock in Governance/Compliance mode for retention windows.- Maintain an append-only ledger: store per-batch Merkle root in immutable store (e.g., blockchain anchor or signed daily checkpoint in KMS-backed DB). Validate hashes during audits.- Sign and timestamp batches with HSM/KMS keys.**Alerting & detection**- Stream to SIEM (Splunk/Elastic SIEM/Chronicle) for realtime rules: multiple failed logins, impossible travel, privilege escalations.- Use threshold and anomaly detection (rate, geo, device fingerprint). Integrate with PagerDuty/Slack and include runbook links.**Cost-effective retention**- Hot for 30–90 days (fast search); warm for 90–365 (less replicas); archive beyond to S3 Glacier. Keep indexed metadata (small footprint) for longer if needed.- Retention policy tied to compliance with automated lifecycle and audit logs of retention changes.**Operational notes**- Monitor ingestion lag, schema drift, and data volume. Automate schema migrations and use sampling for full event retention when volume spikes.- As a full-stack dev I’d implement client SDKs (JS/Node) to enforce schema, redaction, and sign logs before sending to collectors.