Log Analysis and Diagnostic Data Gathering Questions
Extracting signal from existing logs and diagnostic output to find a root cause: parsing and querying log data, correlating traces and metrics during an investigation, and gathering the right diagnostic information (including asking clarifying questions) before drawing conclusions. Covers text-processing and query techniques for locating evidence in logs (structured log parsing, ElasticSearch/SQL-style log queries, log aggregation and retention trade-offs) and reconstructing a timeline from the data on hand. This is the analysis-of-existing-data skill used during troubleshooting and investigation across infrastructure and operations roles: distinct from monitoring and observability, which is about instrumenting a system so telemetry exists in the first place (see the observability topics for that), and distinct from SIEM-based security detection and formal digital-forensics practice (chain of custody, artifact/disk/memory analysis), which have their own dedicated coverage elsewhere in the catalog.
Write a robust regular expression or short code snippet that matches both IPv4 and IPv6 addresses in arbitrary log lines. Include normalization behavior for IPv6 (zero-compression) and handle edge cases where ports are appended (e.g., '2001:db8::1:443' or '192.0.2.1:8080'). Explain common pitfalls with naive regexes and how to validate extracted addresses.
Sample Answer
Direct answer
Do not try to write one regex that fully validates IPv4 and IPv6 addresses. A permissive regex should only find candidate tokens; a standard-library address parser (Python's ipaddress, or an equivalent in another language) should validate and normalize them. Splitting the job this way avoids the two classic failure modes: a regex so strict it misses valid addresses, or one so permissive it accepts garbage like 999.999.999.999.
Approach
- Match IPv4 candidates with a loose digit-dot pattern, optionally followed by
:port. - Match bracketed IPv6 (
[addr]:port), the only unambiguous way to attach a port to an IPv6 address, since IPv6 addresses themselves contain colons. - Match bare (unbracketed) IPv6 candidates separately, using a negative lookbehind (
(?<!...), which matches only when the text immediately before is NOT one of the listed characters) and a negative lookahead ((?!...), the same idea for the text immediately after) so the pattern doesn't start or stop its match in the middle of a longer hex-and-colon run already covered by the bracketed branch above. - Feed every candidate through
ipaddress.IPv4Address/ipaddress.IPv6Address, which rejects invalid octets and normalizes IPv6 to its RFC 5952 zero-compressed canonical form (str()on the parsed object).
import re
import ipaddress
IPV4_RE = re.compile(r'\b(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?\b')
BRACKETED_V6_RE = re.compile(r'\[([0-9A-Fa-f:]+)\](?::(\d{1,5}))?')
BARE_V6_RE = re.compile(r'(?<![0-9A-Fa-f:.\[])([0-9A-Fa-f]{0,4}(?::[0-9A-Fa-f]{0,4}){2,7})(?![0-9A-Fa-f:.\]])')
def extract_addresses(line):
found = []
for m in BRACKETED_V6_RE.finditer(line):
addr_txt, port_txt = m.group(1), m.group(2)
try:
addr = ipaddress.IPv6Address(addr_txt)
except ValueError:
continue
found.append(('IPv6', str(addr), port_txt, m.span()))
for m in IPV4_RE.finditer(line):
addr_txt, port_txt = m.group(1), m.group(2)
try:
addr = ipaddress.IPv4Address(addr_txt)
except ValueError:
continue
found.append(('IPv4', str(addr), port_txt, m.span()))
covered = [f[3] for f in found]
for m in BARE_V6_RE.finditer(line):
span = m.span()
if any(span[0] >= c[0] and span[1] <= c[1] for c in covered):
continue
candidate = m.group(1)
try:
addr = ipaddress.IPv6Address(candidate)
found.append(('IPv6', str(addr), None, span))
continue
except ValueError:
pass
if ':' in candidate:
head, _, tail = candidate.rpartition(':')
if tail.isdigit():
try:
addr = ipaddress.IPv6Address(head)
found.append(('IPv6-ambiguous', str(addr), tail, span))
except ValueError:
pass
return found
lines = [
"2024-01-01T00:00:00Z connect from 192.0.2.1:8080 to service",
"2024-01-01T00:00:01Z peer 2001:db8::1:443 negotiating tls",
"2024-01-01T00:00:02Z peer [2001:db8::1]:443 negotiating tls",
"2024-01-01T00:00:03Z bad token 999.999.999.999 in payload",
"2024-01-01T00:00:04Z full form 2001:0db8:0000:0000:0000:0000:0000:0001 seen",
]
for line in lines:
print(line)
for kind, addr, port, span in extract_addresses(line):
print(f" -> {kind}: {addr}" + (f" port={port}" if port else ""))
Worked example
Output on five representative log lines:
2024-01-01T00:00:00Z connect from 192.0.2.1:8080 to service
-> IPv4: 192.0.2.1 port=8080
2024-01-01T00:00:01Z peer 2001:db8::1:443 negotiating tls
-> IPv6: 2001:db8::1:443
2024-01-01T00:00:02Z peer [2001:db8::1]:443 negotiating tls
-> IPv6: 2001:db8::1 port=443
2024-01-01T00:00:03Z bad token 999.999.999.999 in payload
2024-01-01T00:00:04Z full form 2001:0db8:0000:0000:0000:0000:0000:0001 seen
-> IPv6: 2001:db8::1
The interesting case is the third line: 2001:db8::1:443. A naive "split on the last colon for a port" rule would read this as address 2001:db8::1 with port 443, but 443 in hexadecimal is also a perfectly valid last group of an IPv6 address, so ipaddress.IPv6Address("2001:db8::1:443") parses successfully as a complete, valid 128-bit address. There is no way to tell which the log author meant from the text alone. That is exactly why RFC 3986 requires brackets, [2001:db8::1]:443, when a port follows an IPv6 host: the bracket is the only unambiguous separator, and the fourth line above shows the parser resolving it correctly once brackets are present.
Key points
- Validate, don't just match:
999.999.999.999matches a naive\d{1,3}(\.\d{1,3}){3}pattern but failsipaddress.IPv4Address, which enforces each octet is 0 to 255. - Normalize through the standard library, not by hand:
2001:0db8:0000:0000:0000:0000:0000:0001and2001:db8::1are the same address; onlyipaddressreliably produces the canonical compressed form. - Bracket notation is the only safe way to disambiguate a trailing port on IPv6; if your logs never bracket IPv6 hosts, you cannot recover the port programmatically and should say so rather than guessing.
Complexity
Each candidate match and validation is O(length of the token); scanning a line of length n is O(n) since none of the character classes here cause catastrophic regex backtracking.
Edge cases
- IPv4-mapped IPv6 addresses (
::ffff:192.0.2.1) still validate correctly throughipaddress.IPv6Address. - A raw colon-separated fragment like
00:00:00inside a timestamp can match a loose hex-and-colon regex; this is caught downstream becauseipaddress.IPv6Address("00:00:00")raisesValueError(an IPv6 address needs 8 groups, or a::compression, not 3 bare groups), so validation silently discards the false positive. - Zone IDs on link-local IPv6 (
fe80::1%eth0) are, in fact, standardipaddressinput as of Python 3.9 (ipaddress.IPv6Address("fe80::1%eth0")parses successfully and round-trips the zone instr()); don't strip the%zonesuffix before validating on a modern interpreter, sinceipaddressitself can be handed the whole token. The real gap is upstream of validation:BARE_V6_REabove doesn't include%in its character class, so it only ever capturesfe80::1and silently drops%eth0beforeipaddressgets a chance to see it. If zone IDs matter for your logs, extend the regex's trailing character class to allow a%[\w.-]+suffix rather than relying onipaddressto reject or accept the untouched token. (A link-local address is only valid on the local network segment, not routable beyond it; the zone ID after the%says which network interface it applies to, since the same link-local address can exist on more than one interface at once.)
Trade-offs & pitfalls
The common mistake is trying to encode the full IPv6 grammar (8 groups, one :: compression, embedded IPv4 tail forms) directly in a regex. It is technically possible but produces a pattern that is nearly unreadable and still gets edge cases wrong. Letting the regex be permissive and pushing correctness to a real parser is both simpler and more correct; the cost is a second pass over each candidate, which is negligible next to a log pipeline's overall I/O cost. A related mistake worth naming explicitly: a shared character class like [0-9a-fA-F:.%] for "anything IP-like" will also match ordinary hex-looking words (letters a-f) and will swallow a trailing :port into what it thinks is one IPv6 token, silently dropping the whole match when validation then rejects the combined string. Keeping the IPv4 and bracketed-IPv6 branches structurally separate, as above, avoids that trap.
A pod crash-looped and restarted several times during last night's incident, and its stdout/stderr from before the final restart is gone along with the container. Walk through what you'd still check to reconstruct what happened (previous-container logs, any shipped copies, cluster events, sibling pods), and what you'd flag afterward about how this workload's logs are captured.
Sample Answer
Direct answer
The final crashed container's raw logs being gone doesn't mean the evidence is gone; a pod (Kubernetes' smallest deployable unit, one or more containers sharing storage and network) crash-looping leaves a trail across at least four other places: the kubelet's still-retained previous-container logs, anything already shipped off-node before the crash, the cluster's own event history, and any sibling replicas that hit the same failure.
Structured elaboration
- Previous-container logs. Kubernetes keeps the just-terminated container's logs available even after a restart, specifically for this situation:
kubectl logs <pod> --previous -c <container>fetches the log from the container instance immediately before the current one. This only works if the kubelet (the per-node agent that manages containers) hasn't already garbage-collected that container, which is governed by its own log-retention limits (containerLogMaxSize/containerLogMaxFiles), so this is time- and space-bounded, not guaranteed, especially after "several" restarts have already cycled through that budget. - Any shipped copies. If a log-shipping agent (a DaemonSet running Fluent Bit, Fluentd, or Vector, one instance per node, forwarding container logs to a central store) was already running on this node, the early crash-loop output may already be durably stored centrally, even though the node-local copy and the container itself are both gone. This is usually the single most valuable check, since it's the one place the data might survive independent of anything Kubernetes itself retained.
- Cluster events.
kubectl describe pod <pod>surfaces the pod'sLast State: Terminatedblock (with aReason, e.g.OOMKilledfor out-of-memory kill, orErrorwith an exit code) and its recentEvents(scheduling, image pulls, liveness probe failures, restarts).kubectl get events --field-selector involvedObject.name=<pod>pulls the same event history directly. Even without a single log line, the reason for termination and the timing of each restart often narrows the failure mode substantially on its own. - Sibling pods. If this pod is one replica of a Deployment/ReplicaSet, check whether other replicas hit the same symptom around the same time. If they did, the failure is likely environmental (a bad deploy, a dependency outage, a node-level resource issue) rather than something specific to this one pod instance, which changes both the diagnosis and the confidence you can have in whatever partial logs you do recover.
- Node-level container runtime logs, as a last resort. The kubelet also writes symlinked log files under
/var/log/containers/(pointing at/var/log/pods/<namespace>_<pod>_<uid>/<container>/), and the container runtime's own logs (accessible viacrictl logs <container-id>on the node) may still hold data even ifkubectl logs --previousno longer does, since these live on different retention clocks.
Worked example
# 1. Try the previous-container logs first, cheapest and most direct if available
kubectl logs my-app-7d9f8 --previous -c my-app
# 2. Pull the termination reason and recent event history even if logs are gone
kubectl describe pod my-app-7d9f8
kubectl get events --field-selector involvedObject.name=my-app-7d9f8 --sort-by='.lastTimestamp'
# 3. Check whether other replicas of the same workload saw the same symptom
kubectl get pods -l app=my-app -o wide
kubectl logs -l app=my-app --since=12h --prefix
# 4. If a log shipper runs on this cluster, search the central store for this
# pod's UID or name across the incident window rather than the node itself
If describe pod shows Reason: OOMKilled on every restart and sibling pods show the same pattern starting at the same timestamp, you likely have enough to diagnose a memory regression or a traffic-driven memory spike without ever recovering the missing raw log lines, the event history and cross-pod pattern are the evidence here.
What to flag afterward about log capture
- If nothing was shipping this workload's logs off-node, that's the actual finding: the workload's evidence for a crash loop depends entirely on how many restarts happened before someone looked, which is not a property you want to depend on during an incident. Getting a log shipper (DaemonSet-based, so it applies cluster-wide without per-workload setup) in front of this workload is the concrete fix.
- If a shipper was running but this data still wasn't recoverable, check for a gap between "container writes to stdout" and "shipper picks it up and forwards it durably," a crash that happens faster than the shipper's flush/batch interval can lose the last few seconds of output even with shipping enabled.
- Consider whether this workload needs any log capture that survives a fast, repeated crash loop by design (e.g. writing to an
emptyDirvolume that outlives individual container restarts within the pod's lifetime, not just relying on stdout capture timing).
Trade-offs & pitfalls
--previousonly ever gives you the immediately prior container instance; if the pod has crash-looped many times since the interesting failure, that specific evidence may already be several restarts gone, which is exactly why shipped copies matter so much more than node-local state for anything that isn't caught within minutes.- Treating
OOMKilledor a generic exit code as a full diagnosis without corroborating log content risks a wrong root cause; use the event/exit-code evidence to narrow the hypothesis, not to close the investigation outright.
Coding task in Python: build a memory-efficient utility that reads multiple large newline-delimited JSON log files and extracts, in chronological order, all events that have a given request_id between two timestamps. The function signature should be extract_events(file_paths, request_id, start_ts, end_ts) and must stream files without loading them entirely into memory. Describe edge cases you would handle in production.
Sample Answer
Direct answer
Stream each file line by line rather than reading it into memory, filter to the matching request_id and timestamp range as you go, and merge the per-file streams with a k-way merge, using a min-heap (a small structure that always exposes the smallest of the items it holds) keyed by timestamp, so the combined output comes out globally chronological without ever holding more than one buffered record per file at once.
Approach
import json
import heapq
from datetime import datetime
def _parse_ts(raw):
return datetime.fromisoformat(raw.replace("Z", "+00:00"))
def _file_matches(path, request_id, start_ts, end_ts):
"""Stream one file line by line. Assumes each file is internally
time-ordered (true of an append-only log file), so this generator
yields matching events in ascending timestamp order for THIS file.
Never holds more than the current line in memory.
"""
with open(path, "r") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
continue
if rec.get("request_id") != request_id:
continue
raw_ts = rec.get("ts")
if raw_ts is None:
continue
try:
event_ts = _parse_ts(raw_ts)
except ValueError:
continue
if start_ts <= event_ts <= end_ts:
rec["_ts_parsed"] = event_ts
yield rec
def extract_events(file_paths, request_id, start_ts, end_ts):
"""Yield events with the given request_id whose timestamp falls in
[start_ts, end_ts], in chronological order, across multiple large
NDJSON log files, without loading any file fully into memory.
Each file is scanned by its own generator (bounded to one open line
at a time); a min-heap does a k-way merge across those generators so
only one buffered record per file is held at once, not all matches.
Memory is O(number of files), not O(file size) or O(match count).
"""
heap = []
iterators = [iter(_file_matches(p, request_id, start_ts, end_ts)) for p in file_paths]
for idx, it in enumerate(iterators):
first = next(it, None)
if first is not None:
heapq.heappush(heap, (first["_ts_parsed"], idx, first))
while heap:
ts, idx, rec = heapq.heappop(heap)
del rec["_ts_parsed"]
yield rec
nxt = next(iterators[idx], None)
if nxt is not None:
heapq.heappush(heap, (nxt["_ts_parsed"], idx, nxt))
if __name__ == "__main__":
import os
file_a = "/tmp/loganalysis_b1_s13_a.ndjson"
file_b = "/tmp/loganalysis_b1_s13_b.ndjson"
# file_a and file_b are each individually time-ordered, as a real
# per-host or per-service log file would be, even though the SET of
# files is not globally ordered relative to each other.
with open(file_a, "w") as f:
f.write(json.dumps({"ts": "2025-01-01T00:00:01Z", "request_id": "req-1", "service": "C", "msg": "too_early"}) + "\n")
f.write(json.dumps({"ts": "2025-01-01T00:00:02Z", "request_id": "req-1", "service": "A", "msg": "start"}) + "\n")
f.write(json.dumps({"ts": "2025-01-01T00:00:09Z", "request_id": "req-2", "service": "A", "msg": "other_request"}) + "\n")
f.write("not valid json\n")
f.write(json.dumps({"ts": "2025-01-01T00:00:20Z", "request_id": "req-1", "service": "A", "msg": "too_late"}) + "\n")
with open(file_b, "w") as f:
f.write(json.dumps({"ts": "2025-01-01T00:00:05Z", "request_id": "req-1", "service": "B", "msg": "middle"}) + "\n")
start = _parse_ts("2025-01-01T00:00:02Z")
end = _parse_ts("2025-01-01T00:00:10Z")
for rec in extract_events([file_a, file_b], "req-1", start, end):
print(rec)
os.remove(file_a)
os.remove(file_b)
Worked example (executed output)
{'ts': '2025-01-01T00:00:02Z', 'request_id': 'req-1', 'service': 'A', 'msg': 'start'}
{'ts': '2025-01-01T00:00:05Z', 'request_id': 'req-1', 'service': 'B', 'msg': 'middle'}
file_a and file_b are individually time-ordered (as a real append-only log file would be), but the two files are not ordered relative to each other, and file_a even has an out-of-range line, a wrong-request-id line, and a malformed line mixed in. The output above still comes out correctly filtered and in chronological order across both files.
Key points
- Memory usage is
O(number of files), notO(file size)orO(number of matches): at any moment, the heap holds at most one buffered record per file, since each file is its own generator that only produces its next matching record when asked. - The k-way merge relies on one precondition worth stating explicitly: each individual file must already be time-ordered internally. That's a safe assumption for a normal append-only application log, but it would need to be checked (or the approach would need to change) for a log source that writes out of order within a single file.
extract_eventsis itself a generator (ityields rather than returning a list), so a caller can start consuming matching events immediately and stop early if it only needs the first few, rather than waiting for the entire multi-file scan to finish.
Complexity
Let N be the total number of matching events across all files and k be the number of files. Each heap push/pop is O(log k), and there are O(N) of them (one per yielded event, plus refills), so total time is O(N log k). Space is O(k) for the heap itself, plus the fixed per-line buffer each open file uses while being read, which is independent of file size.
Edge cases
- Malformed JSON lines: skipped rather than raising, so one corrupt line doesn't abort the entire scan (shown above:
file_a's "not valid json" line is silently passed over). - A file with no matches at all: its generator is simply exhausted immediately and it drops out of the heap; the merge continues correctly with the remaining files.
- Ties on identical timestamps across files: the heap comparison falls through to the file index as a tiebreaker, so results are still deterministic (files earlier in the input list sort first on an exact tie) rather than raising a comparison error on the underlying dict.
- A file that isn't actually time-ordered internally: this breaks the merge's chronological guarantee silently (each file's own stream would emit an out-of-order record without the heap detecting it), which is worth flagging explicitly rather than assuming away.
Trade-offs & pitfalls
A simpler alternative, collect every match into a list and sort once at the end, is easier to write and is fine when you expect a bounded, small number of matches for one request_id (usually true; a single request rarely touches millions of log lines). The heap-based merge shown here is the more defensible answer for a "must stream, must not load fully into memory" prompt because it doesn't rely on that assumption: it stays memory-bounded even in the pathological case where one request_id matches a huge number of events, which the simpler collect-and-sort approach would not.
Write a Python 3 script that reads a newline-delimited log file where each line is a JSON object with keys: "timestamp" (ISO 8601), "service", "level", "message". The script should output per-minute error counts (level == "ERROR") for a given service over the last 60 minutes, printing lines like: 2025-03-12 14:05 3. The log may be out-of-order and can be large (~10GB): prioritize streaming and bounded memory.
Sample Answer
Direct answer
Stream the file line by line and keep only a small dict of per-minute counters, never the raw lines, so memory stays bounded regardless of file size. Because the file can be out of order, the true 60-minute window boundary (which minute is "now") isn't known until you've seen the latest timestamp, so prune the counters dict opportunistically as later, larger timestamps arrive, rather than needing a full second pass over the file.
Approach
import json
from datetime import datetime, timedelta, timezone
WINDOW_MINUTES = 60
def _parse_ts(raw):
if raw.endswith("Z"):
raw = raw[:-1] + "+00:00"
dt = datetime.fromisoformat(raw)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def per_minute_error_counts(path, service):
"""Stream a large NDJSON log file and return per-minute ERROR counts
for `service` over the last WINDOW_MINUTES relative to the latest
timestamp seen, as a dense list of (minute, count) tuples (one entry
per minute, zero-filled).
The file may be out of order. Rather than buffering every distinct
minute seen across the whole file, the counts dict is pruned WHILE
streaming: every time a new maximum timestamp arrives, anything
older than (new_max - WINDOW_MINUTES) is dropped. Memory therefore
stays close to O(WINDOW_MINUTES) even over a ~10GB file, at the cost
of one bookkeeping check per line rather than a second pass.
"""
counts = {}
newest = None
with open(path, "r") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
rec = json.loads(line)
except json.JSONDecodeError:
continue
if rec.get("service") != service:
continue
raw_ts = rec.get("timestamp")
if not raw_ts:
continue
try:
ts = _parse_ts(raw_ts)
except ValueError:
continue
if newest is None or ts > newest:
newest = ts
cutoff = newest - timedelta(minutes=WINDOW_MINUTES - 1)
for bucket in [b for b in counts if b < cutoff.replace(second=0, microsecond=0)]:
del counts[bucket]
if rec.get("level") == "ERROR":
bucket = ts.replace(second=0, microsecond=0)
counts[bucket] = counts.get(bucket, 0) + 1
if newest is None:
return []
window_start = (newest - timedelta(minutes=WINDOW_MINUTES - 1)).replace(second=0, microsecond=0)
result = []
cur = window_start
end = newest.replace(second=0, microsecond=0)
while cur <= end:
result.append((cur.strftime("%Y-%m-%d %H:%M"), counts.get(cur, 0)))
cur += timedelta(minutes=1)
return result
if __name__ == "__main__":
import os
path = "/tmp/loganalysis_b1_s18.ndjson"
lines = [
{"timestamp": "2025-03-12T14:05:10Z", "service": "checkout", "level": "ERROR", "message": "timeout"},
{"timestamp": "2025-03-12T13:00:00Z", "service": "checkout", "level": "ERROR", "message": "too old, outside 60min window"},
{"timestamp": "2025-03-12T14:05:40Z", "service": "checkout", "level": "ERROR", "message": "timeout"},
{"timestamp": "2025-03-12T14:59:59Z", "service": "checkout", "level": "ERROR", "message": "latest event, defines the window end"},
{"timestamp": "2025-03-12T14:05:20Z", "service": "checkout", "level": "INFO", "message": "not an error"},
{"timestamp": "2025-03-12T14:07:00Z", "service": "checkout", "level": "ERROR", "message": "timeout"},
{"timestamp": "2025-03-12T14:07:05Z", "service": "auth", "level": "ERROR", "message": "different service"},
]
with open(path, "w") as f:
for rec in lines:
f.write(json.dumps(rec) + "\n")
for bucket, cnt in per_minute_error_counts(path, "checkout"):
if cnt:
print(f"{bucket} {cnt}")
os.remove(path)
Worked example (executed output)
2025-03-12 14:05 2
2025-03-12 14:07 1
2025-03-12 14:59 1
The synthetic input above is deliberately out of order and mixes services and levels: a 13:00 entry (an hour before the latest timestamp, correctly excluded from the 60-minute window), an INFO-level entry at 14:05 (correctly excluded, not an ERROR), and an entry from a different service, auth (correctly excluded). The three surviving lines match exactly what the question's example output format specifies: YYYY-MM-DD HH:MM count.
Key points
- Memory stays close to
O(WINDOW_MINUTES), notO(distinct minutes in the whole file): every time a new maximum timestamp arrives, the code immediately drops any bucket older thannew_max - 60 minutes, rather than waiting until the end of the file to filter. Over a ~10 GB file that could span far more than an hour of wall-clock time, this keeps the counters dict small throughout the scan instead of growing with every minute the file happens to contain. - The underlying function returns a dense per-minute series (one entry per minute of the 60-minute window, zero-filled where there were no errors), which is more useful than a sparse series if you intend to plot it; the driver code above only prints non-zero minutes to match the requested output format, but the zero-filled minutes are still available from the function's return value.
- Out-of-order tolerance comes from tracking the running maximum timestamp seen so far, not from assuming the file's line order matches timestamp order at all.
Complexity
Time is O(n) for n lines in the file, one pass, no sorting. Space is bounded by the number of distinct minute buckets retained at any point, which the opportunistic pruning keeps close to the 60-minute window rather than proportional to the file's total time span.
Edge cases
- No matching lines at all (wrong service, or no ERROR-level entries): the function returns an empty list rather than guessing a window relative to wall-clock "now," since there's no timestamp evidence in the file to anchor a window to.
- All events in the same single minute: the loop still correctly walks the full 60-minute range and reports that one minute's count alongside 59 zero counts in the dense series.
- A timestamp that fails to parse: skipped rather than raised, so one malformed line doesn't abort processing of a 10 GB file.
Trade-offs & pitfalls
The main design decision worth defending is anchoring "last 60 minutes" to the latest timestamp actually seen in the file, rather than to wall-clock now(). For a file being analyzed after the fact (which is the realistic scenario for "you're handed a log file and asked to find the error pattern"), the data's own latest timestamp is the meaningful reference point; anchoring to now() instead would silently produce an empty or wrong window whenever the file isn't from the current moment, which is the common case during an investigation. The trade-off of opportunistic pruning versus a simpler two-pass approach (scan once for the max timestamp, then scan again to aggregate) is one pass of I/O saved at the cost of slightly more bookkeeping per line; for a 10 GB file, halving the required I/O is worth that complexity.
After a critical outage you discover that some logs were lost due to a rotation misconfiguration. Describe the forensic investigation steps you would take: evidence preservation, reconstructing timelines, correlating remaining traces and metrics, and long-term fixes to logging and retention to prevent recurrence.
Sample Answer
Direct answer
Treat this like any forensic investigation: preserve what still exists before it decays further, rebuild the timeline from every surviving source (not just the logs that were lost), and only then fix the rotation problem so it cannot happen again. The rotation loss itself becomes part of the incident, not just an inconvenience while investigating the original outage.
Structured elaboration
Evidence preservation (first)
- Immediately copy every surviving log file, rotated archive, and journal export to a separate location before anything else runs rotation, cleanup cron jobs, or disk pressure that could delete more evidence. Treat the current state as read-only.
- Snapshot configuration alongside the data: the
logrotateconfig (logrotateis the standard Linux utility that rotates, compresses, and deletes log files on a schedule) or thejournaldconfig (journaldis systemd's log-collecting daemon) / container logging driver config, any log-shipping agent config, and the retention policy in effect at the time. You will need these to explain the loss, not just to fix it. - Preserve non-log evidence that was not affected by the rotation bug: metrics/dashboards, alert history, deploy and change-management records, and any centralized copy a shipping agent may have already forwarded before local deletion happened.
Reconstructing the timeline from what remains
- Anchor on sources the rotation bug did not touch: monitoring alert timestamps, deploy pipeline timestamps, load balancer or gateway logs (often on a separate host/retention policy), and database slow-query or error logs.
- If a log-shipping agent was running, check whether a central log store already has a copy of what was lost locally; local rotation deleting a file does not delete an already-shipped copy.
- Build the timeline in UTC from the start, and note explicitly which windows have zero surviving log coverage, so the gap itself is documented rather than silently absent from the narrative.
Correlating remaining traces and metrics
- Use metric anomalies (error-rate spikes, latency jumps, restart counts) to bound the incident window even where no log line survives inside it.
- Cross-reference any request or correlation IDs that do survive (in a load balancer log, for instance) against the services you have log coverage for, to partially reconstruct request flow through the gap.
- Where two independent sources agree on a timestamp and a plausible causal link, treat that as higher confidence than a single source alone.
Long-term fixes to logging and retention
- Separate rotation from retention: local rotation should only ever archive or compress, never delete, until a shipping agent has confirmed successful delivery to a durable off-host store.
- Ship logs off the host in near real time so a single host's local rotation bug is not a single point of failure for the evidence.
- Add an explicit alert on log-volume dropping to near zero for a service that is still running; a rotation misconfiguration usually shows up as a sudden, suspicious silence before anyone notices logs are actually gone.
- Periodically test the rotation and retention configuration itself (a scheduled check that logs from N days ago are still retrievable where policy says they should be), rather than only discovering it is broken during an incident.
Worked example
Say the rotation config was set to keep only 2 rotated files at 10 MB each (20 MB total), but the service was logging at a rate that filled 20 MB in under an hour during the incident. By the time anyone started investigating hours later, the relevant window had already rotated out. The forensic reconstruction in this case would lean on the load balancer's access log (a separate host, unaffected), the deploy record showing a config push 12 minutes before the first alert, and the database's own slow-query log, which together bound the incident to a 20-minute window and identify the deploy as the most likely trigger, even though the application's own logs for that window are gone.
Trade-offs & pitfalls
The biggest mistake here is spending the first hour trying to recover the deleted logs (which is usually not possible once rotation has removed and overwritten disk blocks) instead of immediately preserving what remains and pivoting to alternate sources. A second common mistake is fixing only the rotation size/count without addressing the systemic issue: any purely host-local retention policy is one misconfiguration away from doing this again, which is why the durable long-term fix is decoupling deletion from local rotation entirely.
Unlock Full Question Bank
Get access to all 28 Log Analysis and Diagnostic Data Gathering interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.