Log Analysis and Diagnostic Data Gathering Questions

Direct answer

When wall-clock timestamps across regions can't be trusted and some traces are missing, don't build one linear timeline from raw timestamps. Instead, anchor on the most reliable time sources you have, order events causally wherever a correlation ID or a request/response pair links them directly, and explicitly attach a confidence level to every event in the reconstructed timeline rather than presenting a single false-precision sequence.

Structured elaboration

Data sources to pull, roughly in order of trust

• A single, centralized ingress point (a global load balancer or CDN edge) if one exists: its clock is one clock, not many, so events it recorded are internally consistent with each other even if every other host disagrees.
• Any surviving distributed traces, since a trace's internal span ordering (parent started before child) is causal, not just timestamp-based, and survives clock skew even when the absolute times are off.
• Metrics and alert history, which are often still complete even when detailed traces were dropped, and which give you a coarse but reliable window for when things degraded.
• NTP (Network Time Protocol, the standard system-clock synchronization protocol) or chrony logs, if collected, which tell you how far each host's clock had actually drifted at the relevant time.
• Deploy and configuration-change records, usually from a system with its own independent clock, useful as an anchor point that doesn't depend on the skewed hosts at all.

Correlating and reconstructing

• Join everything you can on a request or correlation ID first; causal links from an ID match are far more trustworthy than "these two lines have similar timestamps."
• Where you know or can estimate a host's clock offset (from NTP logs, or by comparing a request's arrival time at an ingress point against that same request's logged time on the skewed host), apply the correction and keep the original alongside it, don't overwrite the raw evidence.
• For traces that were dropped entirely, use surrounding evidence (the request's presence in an access log, a metric spike, a queue-depth anomaly) to infer that something happened in that window even without a full span, being explicit that this is an inference, not a direct observation.

Confidence levels

Assign each event or causal link an explicit confidence tier rather than a single flat timeline: high confidence for events tied together by a shared trace or request ID with no clock dependency at all, medium confidence for events correlated by corrected/estimated time plus a plausible causal story, and low confidence for anything relying on raw uncorrected timestamps alone. State the tier next to each claim in the final writeup.

Worked example

Suppose a request enters through a single global load balancer at 14:02:00.100 (the load balancer's own clock, one source, trusted), and two downstream services in different regions log timestamps of 14:01:58 and 14:02:03 for what should be the same request. Rather than concluding the second service ran before the first, or that there's a 5-second gap, you'd check whether both services log the same request/correlation ID; if they do, you know the causal order (load balancer received it, then downstream service A, then downstream service B) regardless of what the raw clocks say, and you'd note the load-balancer-anchored event as high confidence while flagging the two regional timestamps as unreliable without a computed clock offset.

Trade-offs & pitfalls

The pitfall that produces the most confidently wrong postmortems is presenting a single merged timeline built from raw, uncorrected timestamps as if it were ground truth. It looks authoritative and it usually is not; readers act on it as if every ordering claim were equally certain. The discipline of tagging confidence per event is more work up front, but it's what keeps a plausible-looking but wrong causal story from becoming the official root cause. A second pitfall is treating "dropped trace" as "nothing happened": absence of evidence in a system known to lose data under load is not evidence of absence, and inferring from surrounding signals (however lower-confidence) is usually better than leaving a silent gap in the narrative.

Direct answer

Don't treat 1TB/day as one blob: split it by log class, keep security logs at full fidelity for a long time because you can't predict when you'll need them, and be aggressive about compressing, shrinking the searchable ("hot") window, and sampling everything else. The cost reduction mostly comes from moving bytes out of expensive, instantly-searchable storage into cheap archival storage sooner, not from deleting data outright.

Structured elaboration

Three levers do almost all the work, and they compose:

1. Compression. Switching from raw text to a compressed columnar format (a column-oriented file format, meaning it stores each field together across rows rather than row-by-row, e.g. Parquet with Snappy or gzip) typically shrinks log data several-fold, because timestamps, service names, and log levels repeat constantly and compress well. This is free fidelity: every original field is still there, just packed tighter.
2. Tiered storage. Keep a short "hot" window (fully indexed, fast to search) and move everything older into a cold/archive storage class that costs an order of magnitude less per gigabyte but takes longer to retrieve (minutes to hours instead of seconds). The hot window should match how far back people actually search during a live incident, not how far back you need to retain data.
3. Sampling and selective indexing for low-value classes. Debug logs are the biggest volume and the least likely to matter weeks later; keep errors/exceptions at full fidelity but sample routine debug chatter (e.g. 5-10%) once it's past the first few days. Index only the fields you actually query on, not every field in every log line.

Security logs get none of the shrinking: the whole point of the policy is that you don't know today which security event will matter in six weeks, so fidelity there is non-negotiable, and only the storage tier (hot vs. cold) changes over time, not the completeness of the data.

Worked example (concrete policy by class)

I modeled the cost impact with an illustrative unit-cost model (hot storage = 1 unit per GB-month, cold/archive = 0.1 units per GB-month, a directionally realistic 10x ratio between instantly-queryable and archival object storage, not a specific vendor's price list) against a 1,000 GB/day baseline split 10% security / 40% access / 50% debug, with today's baseline being everything flat, uncompressed, and hot for 90 days:

HOT_UNIT, COLD_UNIT, COMPRESSION = 1.0, 0.1, 3.0
classes = {
    "security": dict(daily_gb=100, hot_days=30, cold_days=335, sample=1.00),
    "access":   dict(daily_gb=400, hot_days=30, cold_days=60,  sample=0.10),
    "debug":    dict(daily_gb=500, hot_days=7,  cold_days=23,  sample=0.05),
}
baseline_days = 90

def new_cost(c):
    compressed = c["daily_gb"] / COMPRESSION
    return compressed * c["hot_days"] * HOT_UNIT + compressed * c["sample"] * c["cold_days"] * COLD_UNIT

def baseline_cost(c):
    return c["daily_gb"] * baseline_days * HOT_UNIT

for name, c in classes.items():
    b, n = baseline_cost(c), new_cost(c)
    print(f"{name:<10} baseline={b:>7.0f}  new={n:>7.0f}  reduction={((b-n)/b*100):5.1f}%")

Output:

security   baseline=   9000  new=   2117  reduction= 76.5%
access     baseline=  36000  new=   4080  reduction= 88.7%
debug      baseline=  45000  new=   1186  reduction= 97.4%

Combined, that's a 91.8% reduction in the cost model (90,000 to 7,383 units), comfortably past the 60% target even for the untouched-fidelity security class, because moving bytes to a 10x-cheaper tier does most of the work by itself. If 90%+ feels too aggressive for a real environment (e.g. you actually need more than 30 days of hot access-log search), you have headroom to extend the hot windows and still land above 60%; the model makes that trade-off a single parameter change rather than a re-architecture.

Multi-tenant extension

If this is a multi-tenant product rather than a single environment, the same three-tier structure still applies, but retention windows and isolation now vary per customer SLA tier instead of per log class alone: a Gold-tier customer's logs might warrant a dedicated index (not just a shared index with a tenant_id filter) and a dedicated encryption key so their data can be cryptographically deleted or exported independently, while a Bronze-tier customer's logs live in a shared, field-isolated index with a shared key. That per-tenant key strategy is also what makes an emergency eDiscovery (electronic discovery, producing data for litigation or a regulator) export tractable: you can export and later destroy one tenant's key without touching anyone else's data.

Trade-offs & pitfalls

• Cold storage is cheap per gigabyte but not free to use: rehydrating archived data for an investigation takes time and sometimes direct retrieval cost, so "we kept everything" is only true in a useful sense if you also budget for and rehearse rehydration.
• Sampling debug logs means some incidents literally have less evidence than others by design; make sure whoever owns incident response knows sampling exists and isn't surprised mid-investigation that 95% of a trace is missing.
• A retention policy that isn't automated (relying on someone to manually delete or tier data) tends to silently regress back toward "keep everything hot forever" within a few months; the lifecycle transitions need to be enforced by the storage system itself, not by a runbook.

Direct answer

jq is a command-line JSON processor: it reads JSON (here, one JSON object per line, sometimes called NDJSON) and lets you filter, project, and reshape it with a small query language, similar in spirit to how you'd use SQL SELECT/WHERE but for JSON text streams. The three asks below split naturally: two are jq one-liners, and the p95 latency computation is clearer in a few lines of Python since jq has no built-in percentile function.

Structured elaboration

#!/bin/bash
set -e
cat > /tmp/loganalysis_b1_s4_log.jsonl <<'EOF'
{"ts":"2024-03-10T14:12:05Z","level":"info","user_id":123,"action":"login","latency_ms":42}
{"ts":"2024-03-10T14:12:20Z","level":"info","user_id":124,"action":"login","latency_ms":55}
{"ts":"2024-03-10T14:13:01Z","level":"info","user_id":125,"action":"login","latency_ms":39}
{"ts":"2024-03-10T14:12:10Z","level":"info","action":"login","latency_ms":48}
{"ts":"2024-03-10T14:12:12Z","level":"info","user_id":126,"action":"search","latency_ms":100}
{"ts":"2024-03-10T14:12:40Z","level":"info","user_id":127,"action":"search","latency_ms":220}
{"ts":"2024-03-10T14:13:05Z","level":"info","user_id":128,"action":"search","latency_ms":180}
{"ts":"2024-03-10T14:13:30Z","level":"info","user_id":129,"action":"search","latency_ms":300}
{"ts":"2024-03-10T14:13:45Z","level":"info","user_id":130,"action":"search","latency_ms":150}
EOF

echo "1) login counts per minute:"
jq -r 'select(.action=="login") | .ts[0:16]' /tmp/loganalysis_b1_s4_log.jsonl | sort | uniq -c

echo ""
echo "3) events missing user_id:"
jq -c 'select(has("user_id") | not)' /tmp/loganalysis_b1_s4_log.jsonl

The p95 computation below embeds the same nine records directly (rather than reading the file the bash snippet above writes), so it can be copied and run on its own, independent of the bash block:

import json
import math

# Same nine records as the bash heredoc above, embedded directly here so this
# snippet runs on its own without needing that bash block to have run first.
SAMPLE_LINES = [
    '{"ts":"2024-03-10T14:12:05Z","level":"info","user_id":123,"action":"login","latency_ms":42}',
    '{"ts":"2024-03-10T14:12:20Z","level":"info","user_id":124,"action":"login","latency_ms":55}',
    '{"ts":"2024-03-10T14:13:01Z","level":"info","user_id":125,"action":"login","latency_ms":39}',
    '{"ts":"2024-03-10T14:12:10Z","level":"info","action":"login","latency_ms":48}',
    '{"ts":"2024-03-10T14:12:12Z","level":"info","user_id":126,"action":"search","latency_ms":100}',
    '{"ts":"2024-03-10T14:12:40Z","level":"info","user_id":127,"action":"search","latency_ms":220}',
    '{"ts":"2024-03-10T14:13:05Z","level":"info","user_id":128,"action":"search","latency_ms":180}',
    '{"ts":"2024-03-10T14:13:30Z","level":"info","user_id":129,"action":"search","latency_ms":300}',
    '{"ts":"2024-03-10T14:13:45Z","level":"info","user_id":130,"action":"search","latency_ms":150}',
]

def p95_latency(lines, action):
    latencies = []
    for line in lines:
        line = line.strip()
        if not line:
            continue
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            continue
        if rec.get("action") == action and "latency_ms" in rec:
            latencies.append(rec["latency_ms"])
    if not latencies:
        return None
    latencies.sort()
    idx = math.ceil(0.95 * len(latencies)) - 1
    idx = max(0, min(idx, len(latencies) - 1))
    return latencies[idx]


p95 = p95_latency(SAMPLE_LINES, "search")
print(f"p95 latency for search: {p95}ms")

Worked example (executed output)

1) login counts per minute:
   3 2024-03-10T14:12
   1 2024-03-10T14:13

3) events missing user_id:
{"ts":"2024-03-10T14:12:10Z","level":"info","action":"login","latency_ms":48}

p95 latency for search: 300ms

Key points

• Per-minute counts: truncating the ISO 8601 timestamp string to its first 16 characters (ts[0:16]) collapses "2024-03-10T14:12:05Z" to "2024-03-10T14:12", a minute bucket, then sort | uniq -c counts occurrences per bucket. This only works because the timestamps are already zero-padded and lexicographically sortable, which ISO 8601 guarantees by design.
• p95 latency: p95 means "95% of requests were faster than this value." With n samples sorted ascending, the nearest-rank method used here takes the value at index ceil(0.95 * n) - 1. It is a common, simple choice for ad hoc analysis; a statistics library's percentile function may use linear interpolation between ranks instead, which gives a slightly different number on small samples, so don't be surprised if two tools disagree by a few milliseconds on the same data.
• Missing user_id: select(has("user_id") | not) filters to objects where the key is absent entirely. That's different from select(.user_id == null), which would also match a key that exists but was explicitly set to null. Decide which case you actually mean; production logs sometimes have both.

Trade-offs & pitfalls

The assumption doing the most work here is that every ts is UTC and already zero-padded, so a plain string prefix and a plain string sort behave correctly. If timestamps come from multiple sources with mixed timezones (a local offset instead of Z), string-prefix bucketing silently groups events into the wrong minute; normalize to UTC with a real datetime parser first in that case, don't trust the string. Malformed JSON lines are another common trap: a bare jq filter without a try/? guard aborts the whole run on the first bad line, whereas the Python snippet here explicitly skips lines that fail json.loads so one corrupt line doesn't take down the whole aggregation.

Direct answer

Treat this like any forensic investigation: preserve what still exists before it decays further, rebuild the timeline from every surviving source (not just the logs that were lost), and only then fix the rotation problem so it cannot happen again. The rotation loss itself becomes part of the incident, not just an inconvenience while investigating the original outage.

Structured elaboration

Evidence preservation (first)

• Immediately copy every surviving log file, rotated archive, and journal export to a separate location before anything else runs rotation, cleanup cron jobs, or disk pressure that could delete more evidence. Treat the current state as read-only.
• Snapshot configuration alongside the data: the logrotate config (logrotate is the standard Linux utility that rotates, compresses, and deletes log files on a schedule) or the journald config (journald is systemd's log-collecting daemon) / container logging driver config, any log-shipping agent config, and the retention policy in effect at the time. You will need these to explain the loss, not just to fix it.
• Preserve non-log evidence that was not affected by the rotation bug: metrics/dashboards, alert history, deploy and change-management records, and any centralized copy a shipping agent may have already forwarded before local deletion happened.

Reconstructing the timeline from what remains

• Anchor on sources the rotation bug did not touch: monitoring alert timestamps, deploy pipeline timestamps, load balancer or gateway logs (often on a separate host/retention policy), and database slow-query or error logs.
• If a log-shipping agent was running, check whether a central log store already has a copy of what was lost locally; local rotation deleting a file does not delete an already-shipped copy.
• Build the timeline in UTC from the start, and note explicitly which windows have zero surviving log coverage, so the gap itself is documented rather than silently absent from the narrative.

Correlating remaining traces and metrics

• Use metric anomalies (error-rate spikes, latency jumps, restart counts) to bound the incident window even where no log line survives inside it.
• Cross-reference any request or correlation IDs that do survive (in a load balancer log, for instance) against the services you have log coverage for, to partially reconstruct request flow through the gap.
• Where two independent sources agree on a timestamp and a plausible causal link, treat that as higher confidence than a single source alone.

Long-term fixes to logging and retention

• Separate rotation from retention: local rotation should only ever archive or compress, never delete, until a shipping agent has confirmed successful delivery to a durable off-host store.
• Ship logs off the host in near real time so a single host's local rotation bug is not a single point of failure for the evidence.
• Add an explicit alert on log-volume dropping to near zero for a service that is still running; a rotation misconfiguration usually shows up as a sudden, suspicious silence before anyone notices logs are actually gone.
• Periodically test the rotation and retention configuration itself (a scheduled check that logs from N days ago are still retrievable where policy says they should be), rather than only discovering it is broken during an incident.

Worked example

Say the rotation config was set to keep only 2 rotated files at 10 MB each (20 MB total), but the service was logging at a rate that filled 20 MB in under an hour during the incident. By the time anyone started investigating hours later, the relevant window had already rotated out. The forensic reconstruction in this case would lean on the load balancer's access log (a separate host, unaffected), the deploy record showing a config push 12 minutes before the first alert, and the database's own slow-query log, which together bound the incident to a 20-minute window and identify the deploy as the most likely trigger, even though the application's own logs for that window are gone.

Trade-offs & pitfalls

The biggest mistake here is spending the first hour trying to recover the deleted logs (which is usually not possible once rotation has removed and overwritten disk blocks) instead of immediately preserving what remains and pivoting to alternate sources. A second common mistake is fixing only the rotation size/count without addressing the systemic issue: any purely host-local retention policy is one misconfiguration away from doing this again, which is why the durable long-term fix is decoupling deletion from local rotation entirely.

Direct answer

Pull per-service error and request counts for the last 24 hours and the 24 hours before that, compute each service's error rate (errors divided by requests) in both windows, and rank services by how much that rate increased, with a minimum absolute-error floor so a service going from 1 error to 2 doesn't dominate the list.

Structured elaboration

• Inputs: two time-bounded queries against your metrics or log backend (Prometheus, an ELK-style log store, or a data warehouse table): current_window covering [now - 24h, now] and previous_window covering [now - 48h, now - 24h], each returning (service, error_count, request_count) rows.
• Aggregation method: sum errors and requests per service within each window (this is a single GROUP BY service in whichever backend you're querying), then compute error_rate = errors / requests for each service in each window.
• Threshold logic: flag a service as "rising" only if both hold: (1) the percentage increase in error rate versus the previous window exceeds a chosen threshold (e.g. 50%), and (2) the current window has at least some minimum absolute error count (e.g. 20). The second condition matters because a rate can double from a statistically meaningless base (1 error to 2) without being operationally significant; the floor filters that noise out.
• Output: a ranked list of (service, percent_increase, current_rate, current_error_count), truncated to the top N (e.g. top 3), suitable for a Slack message or a dashboard panel.

Worked example

def find_rising_services(current, previous, top_n=3, pct_threshold=50.0, min_errors=20):
    results = []
    for service in set(current) | set(previous):
        cur = current.get(service, {"errors": 0, "requests": 1})
        prev = previous.get(service, {"errors": 0, "requests": 1})
        rate_cur = cur["errors"] / (cur["requests"] or 1)
        rate_prev = prev["errors"] / (prev["requests"] or 1)
        pct_increase = ((rate_cur - rate_prev) / rate_prev * 100 if rate_prev > 0
                         else 100.0 if rate_cur > 0 else 0.0)
        if cur["errors"] >= min_errors and pct_increase >= pct_threshold:
            results.append((service, pct_increase, rate_cur, cur["errors"]))
    results.sort(key=lambda r: r[1], reverse=True)
    return results[:top_n]

Run against pinned example data for four services (current vs. previous 24h window):

current_window = {
    "checkout":  {"errors": 480, "requests": 40000},
    "search":    {"errors": 55,  "requests": 50000},
    "auth":      {"errors": 12,  "requests": 30000},
    "recommend": {"errors": 300, "requests": 20000},
}
previous_window = {
    "checkout":  {"errors": 200, "requests": 39000},
    "search":    {"errors": 50,  "requests": 49000},
    "auth":      {"errors": 10,  "requests": 29500},
    "recommend": {"errors": 40,  "requests": 19800},
}
for service, pct, rate, errs in find_rising_services(current_window, previous_window):
    print(f"{service:<10} +{pct:6.1f}%   current rate={rate:.4%}   errors={errs}")

Output:

recommend  + 642.5%   current rate=1.5000%   errors=300
checkout   + 134.0%   current rate=1.2000%   errors=480

search (7.8% increase) and auth (18% increase, and only 12 errors, below the floor) are correctly excluded: their rate barely moved and/or their volume is too low to act on.

Trade-offs & pitfalls

• Comparing only two adjacent 24-hour windows is sensitive to day-of-week effects (a Monday will look "worse" than a quiet Sunday even with no real regression); a production version would compare against the same weekday a week prior, or a rolling baseline, instead.
• A hard percentage threshold treats a jump from 0.001% to 0.002% the same as 10% to 20% unless the minimum-error floor is tuned well; the floor is doing real work here, not just cosmetic filtering.
• This is a scan-and-report utility, not a real-time alert; running it every few minutes as a scheduled job is usually enough, since "rising over 24h" is inherently a slow-moving signal, not something that needs sub-second reaction.