Log Analysis and Diagnostic Data Gathering Questions

Direct answer

Start from the config: read the logrotate (the standard Linux utility that rotates, compresses, and deletes log files on a schedule) stanza for this application and work out how many days of history it's actually supposed to retain, since "only 2 days survive" might be exactly what the config says, just not what anyone expected. If the config genuinely promises more than 2 days, the next question is whether rotation is happening more often than assumed, most commonly because it's triggered by file size, not purely by a daily schedule, and a burst of logging quietly burned through the retention budget faster than usual.

Structured elaboration

1. Read the actual config, don't assume it says what people remember. Find the stanza for this app (typically under /etc/logrotate.d/) and check rotate N (how many old copies to keep before deleting), and whether rotation is daily/weekly or size-triggered (e.g. size 100M). A config with rotate 14 genuinely only promises 14 rotation cycles, not 14 calendar days, those are the same thing only if rotation is purely time-based and never fires early.
2. Check whether rotation is size-triggered and could have fired more than once a day. This is the most common surprise: if the stanza has a size directive (or maxsize alongside daily), a burst of unusually verbose logging (a bug spamming warnings, a traffic spike) can trigger multiple rotations in a single day. If that happened, rotate 14 might only cover 3-4 calendar days instead of 14, and the config was never wrong, the assumption that "rotation count equals days" was.
3. Check when rotation actually last ran and how often. Logrotate records the last rotation timestamp per config stanza in a status file (commonly /var/lib/logrotate/status on newer Debian/Ubuntu packaging, or /var/lib/logrotate.status on some older systems, check whichever exists on this host). Comparing that against the timestamps on the surviving rotated files (ls -la --time-style=full-iso /var/log/myapp/*.gz) tells you the actual rotation cadence over the retention window, not the cadence the config implies.
4. Rule out something deleting files outside logrotate entirely. A disk-pressure cleanup cron job, a container image rebuild that wiped a non-persistent volume, or a manual cleanup someone ran and forgot about can all produce the exact same symptom as a rotation misconfiguration. Check for other scheduled jobs touching that directory (grep -r logrotate /etc/cron*, and anything else with find/rm against /var/log/myapp) before concluding it's purely a logrotate issue.
5. Dry-run the config to see what it would actually do right now. logrotate -d /etc/logrotate.d/myapp runs a dry run (debug mode: shows what actions would be taken without taking them), which is the fastest way to confirm your reading of the stanza matches what logrotate itself thinks it should do, catching a typo or an unexpected included/overriding config you missed on a manual read.

Worked example

# 1. What does the config actually promise?
cat /etc/logrotate.d/myapp
# e.g.:
#   /var/log/myapp/application.log {
#       daily
#       rotate 14
#       size 100M
#       compress
#   }
# This promises "up to 14 rotations, whichever of daily/100M triggers first"
# -- NOT unconditionally 14 calendar days.

# 2. When did each surviving rotation actually happen, and how big were they?
ls -la --time-style=full-iso /var/log/myapp/application.log*.gz

# 3. What does logrotate's own status file say about last-rotation timing?
cat /var/lib/logrotate/status 2>/dev/null || cat /var/lib/logrotate.status 2>/dev/null

# 4. Confirm the config parses the way you think it does, without applying it.
logrotate -d /etc/logrotate.d/myapp

If step 2's timestamps show several rotations within a single day around the time of a known traffic spike or noisy deploy, that's a strong, checkable signal that size 100M fired early and repeatedly, burning through the 14-rotation budget in far fewer than 14 days, exactly the "expected behavior, unexpected outcome" case rather than a misconfiguration.

What to change so it doesn't bite the next investigation

• If size-triggered rotation is the cause, either raise rotate N enough to cover your actual worst-case rotation cadence during a noisy period, or add maxage semantics via a longer retention window, or (often the real fix) ship logs off-host to a central store where retention isn't coupled to a single host's disk and a burst of local logging can't shrink your effective history.
• If the config is genuinely fine and the real gap is "logrotate's local retention was never meant to serve as your incident-investigation retention," that's a signal the two need to be decoupled: local rotation exists to bound disk usage, not to be the system of record for four-day-old incident evidence.
• Whatever the root cause, document the actual effective retention (in calendar days, under realistic worst-case log volume) somewhere a future investigator will find it before they hit the same surprise.

Trade-offs & pitfalls

• rotate N with a size trigger is a genuinely reasonable config for bounding disk usage; the mistake isn't the config, it's treating "N rotations" and "N days" as interchangeable when planning how far back an investigation can reach.
• Raising rotate N just enough to cover today's worst case doesn't protect against tomorrow's worse case (a bigger traffic spike, a noisier bug); if four-day-plus retention actually matters for incident response, local rotation on a single host is the wrong system to depend on at all.

Direct answer

Treat this like any forensic investigation: preserve what still exists before it decays further, rebuild the timeline from every surviving source (not just the logs that were lost), and only then fix the rotation problem so it cannot happen again. The rotation loss itself becomes part of the incident, not just an inconvenience while investigating the original outage.

Structured elaboration

Evidence preservation (first)

• Immediately copy every surviving log file, rotated archive, and journal export to a separate location before anything else runs rotation, cleanup cron jobs, or disk pressure that could delete more evidence. Treat the current state as read-only.
• Snapshot configuration alongside the data: the logrotate config (logrotate is the standard Linux utility that rotates, compresses, and deletes log files on a schedule) or the journald config (journald is systemd's log-collecting daemon) / container logging driver config, any log-shipping agent config, and the retention policy in effect at the time. You will need these to explain the loss, not just to fix it.
• Preserve non-log evidence that was not affected by the rotation bug: metrics/dashboards, alert history, deploy and change-management records, and any centralized copy a shipping agent may have already forwarded before local deletion happened.

Reconstructing the timeline from what remains

• Anchor on sources the rotation bug did not touch: monitoring alert timestamps, deploy pipeline timestamps, load balancer or gateway logs (often on a separate host/retention policy), and database slow-query or error logs.
• If a log-shipping agent was running, check whether a central log store already has a copy of what was lost locally; local rotation deleting a file does not delete an already-shipped copy.
• Build the timeline in UTC from the start, and note explicitly which windows have zero surviving log coverage, so the gap itself is documented rather than silently absent from the narrative.

Correlating remaining traces and metrics

• Use metric anomalies (error-rate spikes, latency jumps, restart counts) to bound the incident window even where no log line survives inside it.
• Cross-reference any request or correlation IDs that do survive (in a load balancer log, for instance) against the services you have log coverage for, to partially reconstruct request flow through the gap.
• Where two independent sources agree on a timestamp and a plausible causal link, treat that as higher confidence than a single source alone.

Long-term fixes to logging and retention

• Separate rotation from retention: local rotation should only ever archive or compress, never delete, until a shipping agent has confirmed successful delivery to a durable off-host store.
• Ship logs off the host in near real time so a single host's local rotation bug is not a single point of failure for the evidence.
• Add an explicit alert on log-volume dropping to near zero for a service that is still running; a rotation misconfiguration usually shows up as a sudden, suspicious silence before anyone notices logs are actually gone.
• Periodically test the rotation and retention configuration itself (a scheduled check that logs from N days ago are still retrievable where policy says they should be), rather than only discovering it is broken during an incident.

Worked example

Say the rotation config was set to keep only 2 rotated files at 10 MB each (20 MB total), but the service was logging at a rate that filled 20 MB in under an hour during the incident. By the time anyone started investigating hours later, the relevant window had already rotated out. The forensic reconstruction in this case would lean on the load balancer's access log (a separate host, unaffected), the deploy record showing a config push 12 minutes before the first alert, and the database's own slow-query log, which together bound the incident to a 20-minute window and identify the deploy as the most likely trigger, even though the application's own logs for that window are gone.

Trade-offs & pitfalls

The biggest mistake here is spending the first hour trying to recover the deleted logs (which is usually not possible once rotation has removed and overwritten disk blocks) instead of immediately preserving what remains and pivoting to alternate sources. A second common mistake is fixing only the rotation size/count without addressing the systemic issue: any purely host-local retention policy is one misconfiguration away from doing this again, which is why the durable long-term fix is decoupling deletion from local rotation entirely.

Direct answer

Do not try to write one regex that fully validates IPv4 and IPv6 addresses. A permissive regex should only find candidate tokens; a standard-library address parser (Python's ipaddress, or an equivalent in another language) should validate and normalize them. Splitting the job this way avoids the two classic failure modes: a regex so strict it misses valid addresses, or one so permissive it accepts garbage like 999.999.999.999.

Approach

• Match IPv4 candidates with a loose digit-dot pattern, optionally followed by :port.
• Match bracketed IPv6 ([addr]:port), the only unambiguous way to attach a port to an IPv6 address, since IPv6 addresses themselves contain colons.
• Match bare (unbracketed) IPv6 candidates separately, using a negative lookbehind ((?<!...), which matches only when the text immediately before is NOT one of the listed characters) and a negative lookahead ((?!...), the same idea for the text immediately after) so the pattern doesn't start or stop its match in the middle of a longer hex-and-colon run already covered by the bracketed branch above.
• Feed every candidate through ipaddress.IPv4Address / ipaddress.IPv6Address, which rejects invalid octets and normalizes IPv6 to its RFC 5952 zero-compressed canonical form (str() on the parsed object).

import re
import ipaddress

IPV4_RE = re.compile(r'\b(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?\b')
BRACKETED_V6_RE = re.compile(r'\[([0-9A-Fa-f:]+)\](?::(\d{1,5}))?')
BARE_V6_RE = re.compile(r'(?<![0-9A-Fa-f:.\[])([0-9A-Fa-f]{0,4}(?::[0-9A-Fa-f]{0,4}){2,7})(?![0-9A-Fa-f:.\]])')


def extract_addresses(line):
    found = []

    for m in BRACKETED_V6_RE.finditer(line):
        addr_txt, port_txt = m.group(1), m.group(2)
        try:
            addr = ipaddress.IPv6Address(addr_txt)
        except ValueError:
            continue
        found.append(('IPv6', str(addr), port_txt, m.span()))

    for m in IPV4_RE.finditer(line):
        addr_txt, port_txt = m.group(1), m.group(2)
        try:
            addr = ipaddress.IPv4Address(addr_txt)
        except ValueError:
            continue
        found.append(('IPv4', str(addr), port_txt, m.span()))

    covered = [f[3] for f in found]
    for m in BARE_V6_RE.finditer(line):
        span = m.span()
        if any(span[0] >= c[0] and span[1] <= c[1] for c in covered):
            continue
        candidate = m.group(1)
        try:
            addr = ipaddress.IPv6Address(candidate)
            found.append(('IPv6', str(addr), None, span))
            continue
        except ValueError:
            pass
        if ':' in candidate:
            head, _, tail = candidate.rpartition(':')
            if tail.isdigit():
                try:
                    addr = ipaddress.IPv6Address(head)
                    found.append(('IPv6-ambiguous', str(addr), tail, span))
                except ValueError:
                    pass
    return found


lines = [
    "2024-01-01T00:00:00Z connect from 192.0.2.1:8080 to service",
    "2024-01-01T00:00:01Z peer 2001:db8::1:443 negotiating tls",
    "2024-01-01T00:00:02Z peer [2001:db8::1]:443 negotiating tls",
    "2024-01-01T00:00:03Z bad token 999.999.999.999 in payload",
    "2024-01-01T00:00:04Z full form 2001:0db8:0000:0000:0000:0000:0000:0001 seen",
]

for line in lines:
    print(line)
    for kind, addr, port, span in extract_addresses(line):
        print(f"  -> {kind}: {addr}" + (f" port={port}" if port else ""))

Worked example

Output on five representative log lines:

2024-01-01T00:00:00Z connect from 192.0.2.1:8080 to service
  -> IPv4: 192.0.2.1 port=8080
2024-01-01T00:00:01Z peer 2001:db8::1:443 negotiating tls
  -> IPv6: 2001:db8::1:443
2024-01-01T00:00:02Z peer [2001:db8::1]:443 negotiating tls
  -> IPv6: 2001:db8::1 port=443
2024-01-01T00:00:03Z bad token 999.999.999.999 in payload
2024-01-01T00:00:04Z full form 2001:0db8:0000:0000:0000:0000:0000:0001 seen
  -> IPv6: 2001:db8::1

The interesting case is the third line: 2001:db8::1:443. A naive "split on the last colon for a port" rule would read this as address 2001:db8::1 with port 443, but 443 in hexadecimal is also a perfectly valid last group of an IPv6 address, so ipaddress.IPv6Address("2001:db8::1:443") parses successfully as a complete, valid 128-bit address. There is no way to tell which the log author meant from the text alone. That is exactly why RFC 3986 requires brackets, [2001:db8::1]:443, when a port follows an IPv6 host: the bracket is the only unambiguous separator, and the fourth line above shows the parser resolving it correctly once brackets are present.

Key points

• Validate, don't just match: 999.999.999.999 matches a naive \d{1,3}(\.\d{1,3}){3} pattern but fails ipaddress.IPv4Address, which enforces each octet is 0 to 255.
• Normalize through the standard library, not by hand: 2001:0db8:0000:0000:0000:0000:0000:0001 and 2001:db8::1 are the same address; only ipaddress reliably produces the canonical compressed form.
• Bracket notation is the only safe way to disambiguate a trailing port on IPv6; if your logs never bracket IPv6 hosts, you cannot recover the port programmatically and should say so rather than guessing.

Complexity

Each candidate match and validation is O(length of the token); scanning a line of length n is O(n) since none of the character classes here cause catastrophic regex backtracking.

Edge cases

• IPv4-mapped IPv6 addresses (::ffff:192.0.2.1) still validate correctly through ipaddress.IPv6Address.
• A raw colon-separated fragment like 00:00:00 inside a timestamp can match a loose hex-and-colon regex; this is caught downstream because ipaddress.IPv6Address("00:00:00") raises ValueError (an IPv6 address needs 8 groups, or a :: compression, not 3 bare groups), so validation silently discards the false positive.
• Zone IDs on link-local IPv6 (fe80::1%eth0) are, in fact, standard ipaddress input as of Python 3.9 (ipaddress.IPv6Address("fe80::1%eth0") parses successfully and round-trips the zone in str()); don't strip the %zone suffix before validating on a modern interpreter, since ipaddress itself can be handed the whole token. The real gap is upstream of validation: BARE_V6_RE above doesn't include % in its character class, so it only ever captures fe80::1 and silently drops %eth0 before ipaddress gets a chance to see it. If zone IDs matter for your logs, extend the regex's trailing character class to allow a %[\w.-]+ suffix rather than relying on ipaddress to reject or accept the untouched token. (A link-local address is only valid on the local network segment, not routable beyond it; the zone ID after the % says which network interface it applies to, since the same link-local address can exist on more than one interface at once.)

Trade-offs & pitfalls

The common mistake is trying to encode the full IPv6 grammar (8 groups, one :: compression, embedded IPv4 tail forms) directly in a regex. It is technically possible but produces a pattern that is nearly unreadable and still gets edge cases wrong. Letting the regex be permissive and pushing correctness to a real parser is both simpler and more correct; the cost is a second pass over each candidate, which is negligible next to a log pipeline's overall I/O cost. A related mistake worth naming explicitly: a shared character class like [0-9a-fA-F:.%] for "anything IP-like" will also match ordinary hex-looking words (letters a-f) and will swallow a trailing :port into what it thinks is one IPv6 token, silently dropping the whole match when validation then rejects the combined string. Keeping the IPv4 and bracketed-IPv6 branches structurally separate, as above, avoids that trap.

Direct answer

For an occasional, ad-hoc 90-day investigation, reach for Athena (a serverless SQL query engine that reads directly from files in object storage, like Amazon S3) over logs converted to partitioned Parquet (a column-oriented file format, meaning it stores each field together across rows rather than row-by-row, which lets a query skip whole columns and files it doesn't need). It has no idle cost between investigations, and partition pruning plus columnar scanning is usually enough to hit "minutes" on a well-laid-out dataset. Reach for a small, narrowly-scoped Elasticsearch (or OpenSearch) tier only if you need frequent full-text or fuzzy search across raw unstructured message text; reach for Presto/EMR (a distributed SQL engine, Presto or its fork Trino, run on a managed Hadoop/Spark cluster service like Amazon EMR) only if your queries are heavier than Athena's engine handles well, e.g. large multi-way joins across datasets, or you already operate that cluster for other workloads.

Structured elaboration

The three options solve different problems, and the mistake is picking based on familiarity rather than the actual query shape:

• Athena over Parquet. You pay per byte scanned, not for idle cluster time, so it's ideal for a query pattern that's bursty (someone runs a handful of ad-hoc queries during an incident, then nothing for days). Its speed comes almost entirely from two things: converting raw JSON/text logs to a columnar format (so a query touching 3 of 20 fields only reads those 3 columns) and partitioning by date, and ideally also by a high-cardinality field like service, so a query for "the last 3 days, payments-api" physically skips every file outside that slice before scanning a single byte.
• A frozen/cold Elasticsearch (or OpenSearch) tier. Full-text search across raw, unstructured message text (fuzzy matching, "search for anything containing this stack trace fragment") is something a SQL engine over structured Parquet genuinely can't do well. A cold tier keeps that capability available for older data at lower cost than a fully hot cluster, but it's still a cluster you operate continuously, with real fixed cost, so it only earns its keep if full-text search across old data is a frequent, not occasional, need.
• Presto/EMR. Gives you the most raw compute and flexibility (arbitrary joins, UDFs (user-defined functions, custom logic you plug directly into the query), heavier transforms) but you're paying for cluster time even when idle, or paying the startup latency each time you spin one up on demand. For a single active investigation needing ad-hoc queries "back in minutes," standing up a cluster is usually a worse cost/latency trade than Athena unless the specific queries are things Athena's engine struggles with.

Worked example

Partitioning and format conversion aren't just an optimization, they're what makes "minutes not hours" possible at all. Say an investigation targets a 3-day window for one service. Without date/service partitioning, a query has to scan the entire 90-day corpus and filter afterward; with dt (date) and service as partition columns, the query planner discards every file outside that slice before reading anything, since partition values are encoded in the S3 key path itself (e.g. s3://bucket/logs/dt=2026-05-01/service=payments-api/) and never require opening the file to check:

-- Illustrative Athena/Presto SQL over a Glue-cataloged, partitioned Parquet table.
-- I have not executed this against a live Athena endpoint; the syntax (date
-- literals, partition predicates) is standard Presto/Trino SQL I'm confident in.
SELECT request_id, timestamp, level, message
FROM logs_parquet
WHERE dt BETWEEN date '2026-05-01' AND date '2026-05-03'
  AND service = 'payments-api'
  AND level = 'ERROR'
ORDER BY timestamp
LIMIT 500;

For illustration only (not a measured figure from a real system): if the full 90-day corpus is, say, 30 TB, and this query's partition predicate confines it to 3 days of one service's data, that's roughly a 30x reduction in candidate files before columnar pruning even applies, and columnar pruning (only reading the 4 selected columns instead of all 20) shrinks the scanned bytes further still. Both Athena's cost (billed per byte scanned) and its latency scale with bytes scanned, so partitioning and format conversion attack the same number that both cost and speed depend on.

Keeping it affordable without making cold logs unusable

• Run a daily batch job (Glue, Lambda, or a scheduled Spark job) that converts that day's raw logs to compacted, partitioned Parquet shortly after ingestion, so by the time anyone needs to query "yesterday," it's already in the fast format, not something you convert reactively during an incident.
• Keep the Parquet-converted data itself in S3's cheaper storage classes as it ages (moving from frequently-accessed to infrequent-access tiers) since Athena can query Parquet directly from those classes; you're not forced to choose between "cheap" and "queryable in minutes," only between "cheap and queryable in minutes" and "even cheaper but with retrieval latency," which is a separate, later archive tier for data you rarely touch.
• Maintain the Glue Data Catalog (the partition/schema metadata Athena queries against) so partition pruning keeps working as new partitions are added; a catalog that's out of sync with what's actually in S3 forces full scans and quietly destroys the whole point of partitioning.

Trade-offs & pitfalls

• Athena's per-query cost model means a badly-written query (missing partition predicates, SELECT * over unpruned data) can be surprisingly expensive and slow on the same dataset that a well-written query handles cheaply in seconds; enforcing partition predicates (and warning or blocking scans above a size threshold) matters operationally, not just as a style preference.
• A frozen Elasticsearch tier is still infrastructure you run 24/7; don't stand one up "just in case" if full-text search across 90-day-old logs is actually a rare need, the ongoing fixed cost won't be worth it.
• If schema drifts over time (new fields added, types changed), Parquet conversion and the catalog both need to handle schema evolution deliberately, or old partitions become unreadable or return nulls for fields that didn't exist yet when they were written.

Direct answer

Free space by shrinking the journal's retained archive, not by deleting arbitrary files: journalctl has a built-in, safe pruning mechanism for exactly this. In parallel, find which service is actually driving the growth so the same page doesn't repeat in an hour, then fix the retention limits so a single noisy service can't fill the disk again.

Structured elaboration

Immediate mitigation (minutes)

# See how much disk the journal is actually using right now
journalctl --disk-usage

# Shrink retained (archived) journal files down to a size ceiling
sudo journalctl --vacuum-size=500M

# Or shrink by age instead of size
sudo journalctl --vacuum-time=2d

--vacuum-size/--vacuum-time only remove already-archived journal files, oldest first, down to the limit you give; they do not touch the actively-written current file or corrupt anything, which is what makes this safe to run under pressure without a lot of ceremony.

Finding the growth cause

# Per-unit log volume in the last hour, roughly, via line count
journalctl --since "1 hour ago" -o json | jq -r '._SYSTEMD_UNIT' | sort | uniq -c | sort -rn | head

A service that is crash-looping (restarting repeatedly, each restart logging a full startup sequence and a stack trace) is the single most common cause of a sudden journal blowup; systemctl --failed and a quick look at restart counts (systemctl status <unit>) will usually surface it immediately once you know which unit is dominating the volume above.

Long-term prevention

Set explicit limits in /etc/systemd/journald.conf so the journal can never again consume unbounded disk, rather than relying on ad hoc vacuuming after the fact:

SystemMaxUse=500M
SystemKeepFree=1G

SystemMaxUse caps the journal's total footprint; SystemKeepFree reserves headroom on the filesystem regardless of the journal's own cap, so the journal backs off before the disk actually hits zero free space. Beyond the config change, add monitoring on disk usage and on journal growth rate specifically, and fix the crash loop at its source (the actual root cause), since the disk-full page is a symptom, not the underlying problem.

Worked example

Say journalctl --disk-usage reports 9.8 GB against a 10 GB partition, and the per-unit breakdown above shows one unit responsible for 90% of the last hour's volume. Cross-referencing systemctl status <that-unit> shows a restart count climbing every few seconds: the service is crash-looping, and each crash logs a multi-line stack trace. journalctl --vacuum-size=500M immediately reclaims disk so the box stops paging on disk pressure, while the actual fix (why the service keeps crashing) becomes the follow-up work, not something you solve mid-page.

Trade-offs & pitfalls

The pitfall to avoid under pressure is reaching for rm on files under /var/log/journal directly instead of journalctl --vacuum-*. Manually deleting the currently-open journal file can corrupt it or confuse journald's internal bookkeeping; the vacuum commands are specifically designed to be safe to run against a live, actively-writing journal. A second pitfall is stopping at the vacuum step: reclaiming space without setting SystemMaxUse/SystemKeepFree and without fixing the crash loop just delays the next page by however long it takes to refill the disk.