Log Analysis and Diagnostic Data Gathering Questions

Direct answer

Start from the config: read the logrotate (the standard Linux utility that rotates, compresses, and deletes log files on a schedule) stanza for this application and work out how many days of history it's actually supposed to retain, since "only 2 days survive" might be exactly what the config says, just not what anyone expected. If the config genuinely promises more than 2 days, the next question is whether rotation is happening more often than assumed, most commonly because it's triggered by file size, not purely by a daily schedule, and a burst of logging quietly burned through the retention budget faster than usual.

Structured elaboration

1. Read the actual config, don't assume it says what people remember. Find the stanza for this app (typically under /etc/logrotate.d/) and check rotate N (how many old copies to keep before deleting), and whether rotation is daily/weekly or size-triggered (e.g. size 100M). A config with rotate 14 genuinely only promises 14 rotation cycles, not 14 calendar days, those are the same thing only if rotation is purely time-based and never fires early.
2. Check whether rotation is size-triggered and could have fired more than once a day. This is the most common surprise: if the stanza has a size directive (or maxsize alongside daily), a burst of unusually verbose logging (a bug spamming warnings, a traffic spike) can trigger multiple rotations in a single day. If that happened, rotate 14 might only cover 3-4 calendar days instead of 14, and the config was never wrong, the assumption that "rotation count equals days" was.
3. Check when rotation actually last ran and how often. Logrotate records the last rotation timestamp per config stanza in a status file (commonly /var/lib/logrotate/status on newer Debian/Ubuntu packaging, or /var/lib/logrotate.status on some older systems, check whichever exists on this host). Comparing that against the timestamps on the surviving rotated files (ls -la --time-style=full-iso /var/log/myapp/*.gz) tells you the actual rotation cadence over the retention window, not the cadence the config implies.
4. Rule out something deleting files outside logrotate entirely. A disk-pressure cleanup cron job, a container image rebuild that wiped a non-persistent volume, or a manual cleanup someone ran and forgot about can all produce the exact same symptom as a rotation misconfiguration. Check for other scheduled jobs touching that directory (grep -r logrotate /etc/cron*, and anything else with find/rm against /var/log/myapp) before concluding it's purely a logrotate issue.
5. Dry-run the config to see what it would actually do right now. logrotate -d /etc/logrotate.d/myapp runs a dry run (debug mode: shows what actions would be taken without taking them), which is the fastest way to confirm your reading of the stanza matches what logrotate itself thinks it should do, catching a typo or an unexpected included/overriding config you missed on a manual read.

Worked example

# 1. What does the config actually promise?
cat /etc/logrotate.d/myapp
# e.g.:
#   /var/log/myapp/application.log {
#       daily
#       rotate 14
#       size 100M
#       compress
#   }
# This promises "up to 14 rotations, whichever of daily/100M triggers first"
# -- NOT unconditionally 14 calendar days.

# 2. When did each surviving rotation actually happen, and how big were they?
ls -la --time-style=full-iso /var/log/myapp/application.log*.gz

# 3. What does logrotate's own status file say about last-rotation timing?
cat /var/lib/logrotate/status 2>/dev/null || cat /var/lib/logrotate.status 2>/dev/null

# 4. Confirm the config parses the way you think it does, without applying it.
logrotate -d /etc/logrotate.d/myapp

If step 2's timestamps show several rotations within a single day around the time of a known traffic spike or noisy deploy, that's a strong, checkable signal that size 100M fired early and repeatedly, burning through the 14-rotation budget in far fewer than 14 days, exactly the "expected behavior, unexpected outcome" case rather than a misconfiguration.

What to change so it doesn't bite the next investigation

• If size-triggered rotation is the cause, either raise rotate N enough to cover your actual worst-case rotation cadence during a noisy period, or add maxage semantics via a longer retention window, or (often the real fix) ship logs off-host to a central store where retention isn't coupled to a single host's disk and a burst of local logging can't shrink your effective history.
• If the config is genuinely fine and the real gap is "logrotate's local retention was never meant to serve as your incident-investigation retention," that's a signal the two need to be decoupled: local rotation exists to bound disk usage, not to be the system of record for four-day-old incident evidence.
• Whatever the root cause, document the actual effective retention (in calendar days, under realistic worst-case log volume) somewhere a future investigator will find it before they hit the same surprise.

Trade-offs & pitfalls

• rotate N with a size trigger is a genuinely reasonable config for bounding disk usage; the mistake isn't the config, it's treating "N rotations" and "N days" as interchangeable when planning how far back an investigation can reach.
• Raising rotate N just enough to cover today's worst case doesn't protect against tomorrow's worse case (a bigger traffic spike, a noisier bug); if four-day-plus retention actually matters for incident response, local rotation on a single host is the wrong system to depend on at all.

Direct answer

Before touching the language or the parser, profile the parsing step itself with a sampling profiler to find out whether the cost is really JSON decoding, or something adjacent like object/dict allocation, string encoding conversions, or field extraction logic sitting next to the parse call. Only after that's confirmed as the actual hot path is a rewrite (to a compiled language, or to a SIMD-accelerated JSON parser) worth considering, and even then, the cheapest fix is often to see whether the shell-tooling tier (grep/awk/sed) can filter or pre-shape the data so less JSON ever needs full parsing at all.

Structured elaboration

1. Diagnose where the time is actually going. "Parsing is the bottleneck" from a coarse profile (CPU is pegged, I/O is idle) still leaves several distinct possibilities:

• Use a sampling profiler (py-spy for Python, pprof for Go, perf at the OS level) to get a flame graph (a visualization showing which function calls consumed the most CPU time, stacked by call depth, so the widest bars are the hottest code paths) and see whether time is in the actual decode call (e.g. json.loads) or in what your code does with the result (building dict-of-dicts, converting to your internal schema, string encode/decode churn).
• Check allocation behavior separately from CPU time (Python's tracemalloc, Go's pprof -alloc_objects): a lot of "parsing is slow" turns out to be garbage-collector pressure from allocating a new dict and set of strings per line, not the byte-level scanning itself.
• Isolate parsing from I/O properly: feed the parser from an in-memory buffer or /dev/shm for the profiling run so disk/network jitter can't leak into the measurement, confirming the earlier finding that I/O isn't the bottleneck.

2. The cheaper step before a rewrite. A full language rewrite is a multi-week investment with real risk; before reaching for it, check whether the shell-tooling tier can do enough of the job:

• grep/mawk/sed are compiled C programs with tight, allocation-light loops. If most lines can be filtered out (e.g. you only care about level=ERROR lines, or only a handful of fields) before they ever reach a JSON parser, a grep-then-parse pipeline can cut the number of lines that pay the full JSON-decode cost by an order of magnitude for free.
• If you need real field extraction rather than just filtering, tools like jq or a awk-based fixed-position extractor can pull specific fields without a general-purpose JSON parse, at the cost of being fragile to schema changes.
• This step matters because it changes the actual problem: "parse 200k lines/sec" might become "parse the 20k lines/sec that survived filtering," which a scripting-language parser may already handle fine, no rewrite needed at all.

3. If a rewrite is still justified, what class of tooling. Once you've confirmed the decode itself (not allocation, not I/O, not unnecessary work) is the bottleneck, and pre-filtering doesn't reduce the problem enough, the realistic options are:

• A compiled, garbage-collected language (Go): a large step up in throughput with a manageable operational profile; still has GC pauses under sustained allocation, which pooling (reusing buffers/structs via a pool instead of allocating fresh ones per line) mitigates.
• A systems language with manual memory control (Rust): highest achievable throughput and predictable latency (no GC pauses), at the cost of a steeper team ramp-up.
• A SIMD-accelerated JSON parser (e.g. simdjson and its bindings): uses CPU vector instructions (single instruction, multiple data, meaning one instruction processes several bytes in parallel) to scan JSON far faster than a byte-at-a-time parser; typically wrapped in a small Go or Rust service since simdjson itself is a C++ library.
  The right choice depends on how much of the rest of your pipeline is already in that language: a Go rewrite that reuses your existing Go services is a smaller organizational lift than introducing Rust or C++ purely for this one hot path.

4. Benchmarking before committing. A microbenchmark that doesn't resemble production traffic will mislead you:

• Build the benchmark corpus from real captured log samples, not synthetic data, including your actual field cardinality, nesting depth, and line-length distribution, not just an average-case line repeated a million times.
• Measure more than one thing: lines/sec throughput, but also p50/p95/p99 latency per line (a parser that's fast on average but spikes on deeply-nested lines will still cause tail-latency problems downstream), and memory allocation rate.
• Run sustained (multi-minute) trials, not single-shot timings; JIT (just-in-time compilation, where the runtime compiles and speeds up hot code paths only after a warm-up period)/GC-based runtimes behave very differently warm vs. cold.
• Shadow the candidate against real production traffic (processing a copy of the live stream without acting on the output) before cutting over, so the decision is validated against real, current data shapes rather than the benchmark corpus alone.

I'm not going to state specific throughput numbers here (lines/sec, milliseconds) because they're entirely dependent on your hardware, log shapes, and even background load at benchmark time; the value is in the benchmark harness's shape (representative data, percentile tracking, sustained runs), which is what you'd actually run to get numbers you can trust for your own environment. A benchmark skeleton looks like this:

import time, statistics

def benchmark(parse_fn, corpus, warmup=5000):
    for line in corpus[:warmup]:
        parse_fn(line)  # warm up JIT/caches, discard timing
    latencies = []
    start = time.perf_counter()
    for line in corpus[warmup:]:
        t0 = time.perf_counter()
        parse_fn(line)
        latencies.append(time.perf_counter() - t0)
    total = time.perf_counter() - start
    return {
        "lines_per_sec": len(corpus[warmup:]) / total,
        "p50_us": statistics.median(latencies) * 1e6,
        "p99_us": sorted(latencies)[int(len(latencies) * 0.99)] * 1e6,
    }

Run this on your own captured corpus against each candidate parser to get numbers specific to your hardware and data, that comparison is the deliverable, not any single absolute number.

Trade-offs & pitfalls

• The most common mistake here is skipping straight to "rewrite in Rust" because 200k/sec sounds intimidating, without ever confirming that decode itself (not allocation, not unnecessary downstream work, not I/O) is actually the cost. That can burn weeks rewriting the wrong bottleneck.
• simdjson-class parsers are fastest at raw scanning but usually parse into a DOM-like view (a tree of in-memory objects you then navigate to pull out fields, the same idea as the DOM a browser builds from parsed HTML) you then have to extract fields from; if your real cost is field extraction and schema mapping, a faster raw parser alone won't fix it.
• A rewrite that's faster in isolation but harder to operate (a language your team doesn't run in production elsewhere) can be a net loss even if the benchmark numbers look good; factor operational cost into the decision, not just throughput.

Direct answer

The common application-level severity levels, from least to most severe, are debug, info, notice, warning, error, critical, alert, and emergency; these map directly onto syslog's numeric priority scale (RFC 5424), where a lower number means more severe. In production, the challenge isn't picking a level per log line, it's controlling the volume of the noisiest levels (mainly debug) without losing the context you'd need during an actual incident.

Structured elaboration

The number ordering is deliberately inverted from how severity intuitively reads: 0 is the worst outcome (system unusable), 7 is the most trivial (debug trace). This is why a journalctl -p warning style filter for "warning and above" actually means "priority number 4 or lower."

Controlling debug volume in production

• Sample rather than suppress entirely: keep 100% of error and above, but only log a small fraction (say 1%) of debug/info events, so you retain a representative slice of normal-operation detail instead of losing it completely.
• Correlation IDs as a safety net for sampling: always attach a request or trace ID even to sampled-out events' surrounding context, so if a specific request later turns out to matter, you can request full-detail logging for that one flow rather than needing every debug line for every request.
• Runtime-adjustable verbosity: make log level a live-toggleable setting (a config flag, a feature flag) per service or per instance, so you can turn debug on temporarily for the specific component you're investigating instead of running at high verbosity everywhere, all the time.
• Rate limiting: cap how many times an identical message can be logged per interval; a single misbehaving code path that logs the same error thousands of times per second drowns out everything else in the same window.

Worked example

A payments service normally logs at info level (a few hundred lines a minute) but, during a debugging session, someone left debug enabled in production. Debug volume alone is now tens of thousands of lines a minute, filling the retention window in under an hour and pushing out the error-level lines from the actual incident being investigated three hours earlier before anyone could pull them. Sampling debug output at 1% instead of logging every line, combined with keeping 100% of error, would have kept the incident's own error lines intact for the full retention window while still preserving a usable slice of the debug detail.

Trade-offs & pitfalls

The most common production mistake is treating log level purely as a filter for what a human reads on a terminal, rather than as a volume/cost control. Leaving debug logging on everywhere "just in case" feels safe but actively works against incident response: it accelerates log rotation, increases the cost of every downstream aggregation query, and buries the handful of error/warning lines that actually matter in noise. The opposite mistake, running everything at error only, is just as damaging: when an incident does happen, there's no info/debug context around it to reconstruct what led up to the failure, which is exactly why sampling (some debug, always) tends to beat an all-or-nothing choice.

Direct answer

Centralizing logs trades local simplicity for a single, searchable, cross-host view, at the cost of introducing a new dependency and a new potential single point of failure into the very system you rely on during an incident. Whether that trade is worth it depends on the team's actual scale and pain today, not on centralized logging being a generically "better" architecture.

Structured elaboration

Benefits

• One search surface across every host, instead of SSHing into individual machines to grep local files, which becomes painful fast once you have more than a handful of hosts or any ephemeral/autoscaled infrastructure where a host might not even exist anymore by the time you go looking.
• Consistent retention and access control applied in one place, rather than as many slightly-different local configurations as there are hosts.
• Cross-service correlation (a request spanning several services) becomes a single query instead of manually stitching together several hosts' files by hand.

Risks

• Reliability: the centralized system becomes a dependency of last resort during an incident; if it's degraded at the exact moment you need it (which happens more often than teams expect, since incidents and infrastructure stress correlate), you've lost your primary investigation tool.
• Latency: shipping logs off-host adds a delay between an event happening and it being searchable centrally; for a live, fast-moving incident, that lag matters.
• Privacy and compliance: centralizing means log data, which can contain user identifiers or other sensitive fields, now flows through and is stored in one additional system, which is a new surface to get access control, encryption, and retention policy right on, and potentially a new data-residency concern if that store lives in a different region than the data originated.
• Single points of failure: if the central pipeline or store goes down, you don't just lose the aggregation, you can lose the ability to investigate anything happening right now, unless hosts still retain their own local copies as a fallback.

Worked example

A team running 6 long-lived hosts, where an engineer can still reasonably SSH into each one and grep during an incident, gets comparatively little benefit from centralizing today, the local-file workflow isn't actually painful yet, while taking on real risk (a new dependency, new compliance surface) for benefit they're not using. A team running 200 short-lived, autoscaled containers where a host that logged the interesting event may no longer exist an hour later has effectively no working alternative to centralizing; without it, evidence disappears on its own, unrelated to any bug.

Trade-offs & pitfalls

The decision isn't "centralize or don't," it's "how do you decide whether centralizing addresses this team's actual pain." A reasonable framing: ask whether the team is already spending real, recurring time and pain on the local-file workflow (SSHing into many hosts, losing logs when instances recycle), whether the team has the operational capacity to run or pay for a reliable central pipeline (including its own failure and backpressure handling), and whether the compliance and access-control requirements are things a centralized store would actually satisfy or, worse, entrench a bad practice around handling sensitive fields. If those answers point toward hybrid, keep local retention as a fallback even after centralizing, so an outage in the central pipeline doesn't leave the team with zero evidence for the exact incident that pipeline outage might itself be causing.

Direct answer

The final crashed container's raw logs being gone doesn't mean the evidence is gone; a pod (Kubernetes' smallest deployable unit, one or more containers sharing storage and network) crash-looping leaves a trail across at least four other places: the kubelet's still-retained previous-container logs, anything already shipped off-node before the crash, the cluster's own event history, and any sibling replicas that hit the same failure.

Structured elaboration

1. Previous-container logs. Kubernetes keeps the just-terminated container's logs available even after a restart, specifically for this situation: kubectl logs <pod> --previous -c <container> fetches the log from the container instance immediately before the current one. This only works if the kubelet (the per-node agent that manages containers) hasn't already garbage-collected that container, which is governed by its own log-retention limits (containerLogMaxSize/containerLogMaxFiles), so this is time- and space-bounded, not guaranteed, especially after "several" restarts have already cycled through that budget.
2. Any shipped copies. If a log-shipping agent (a DaemonSet running Fluent Bit, Fluentd, or Vector, one instance per node, forwarding container logs to a central store) was already running on this node, the early crash-loop output may already be durably stored centrally, even though the node-local copy and the container itself are both gone. This is usually the single most valuable check, since it's the one place the data might survive independent of anything Kubernetes itself retained.
3. Cluster events. kubectl describe pod <pod> surfaces the pod's Last State: Terminated block (with a Reason, e.g. OOMKilled for out-of-memory kill, or Error with an exit code) and its recent Events (scheduling, image pulls, liveness probe failures, restarts). kubectl get events --field-selector involvedObject.name=<pod> pulls the same event history directly. Even without a single log line, the reason for termination and the timing of each restart often narrows the failure mode substantially on its own.
4. Sibling pods. If this pod is one replica of a Deployment/ReplicaSet, check whether other replicas hit the same symptom around the same time. If they did, the failure is likely environmental (a bad deploy, a dependency outage, a node-level resource issue) rather than something specific to this one pod instance, which changes both the diagnosis and the confidence you can have in whatever partial logs you do recover.
5. Node-level container runtime logs, as a last resort. The kubelet also writes symlinked log files under /var/log/containers/ (pointing at /var/log/pods/<namespace>_<pod>_<uid>/<container>/), and the container runtime's own logs (accessible via crictl logs <container-id> on the node) may still hold data even if kubectl logs --previous no longer does, since these live on different retention clocks.

Worked example

# 1. Try the previous-container logs first, cheapest and most direct if available
kubectl logs my-app-7d9f8 --previous -c my-app

# 2. Pull the termination reason and recent event history even if logs are gone
kubectl describe pod my-app-7d9f8
kubectl get events --field-selector involvedObject.name=my-app-7d9f8 --sort-by='.lastTimestamp'

# 3. Check whether other replicas of the same workload saw the same symptom
kubectl get pods -l app=my-app -o wide
kubectl logs -l app=my-app --since=12h --prefix

# 4. If a log shipper runs on this cluster, search the central store for this
#    pod's UID or name across the incident window rather than the node itself

If describe pod shows Reason: OOMKilled on every restart and sibling pods show the same pattern starting at the same timestamp, you likely have enough to diagnose a memory regression or a traffic-driven memory spike without ever recovering the missing raw log lines, the event history and cross-pod pattern are the evidence here.

What to flag afterward about log capture

• If nothing was shipping this workload's logs off-node, that's the actual finding: the workload's evidence for a crash loop depends entirely on how many restarts happened before someone looked, which is not a property you want to depend on during an incident. Getting a log shipper (DaemonSet-based, so it applies cluster-wide without per-workload setup) in front of this workload is the concrete fix.
• If a shipper was running but this data still wasn't recoverable, check for a gap between "container writes to stdout" and "shipper picks it up and forwards it durably," a crash that happens faster than the shipper's flush/batch interval can lose the last few seconds of output even with shipping enabled.
• Consider whether this workload needs any log capture that survives a fast, repeated crash loop by design (e.g. writing to an emptyDir volume that outlives individual container restarts within the pod's lifetime, not just relying on stdout capture timing).

Trade-offs & pitfalls

• --previous only ever gives you the immediately prior container instance; if the pod has crash-looped many times since the interesting failure, that specific evidence may already be several restarts gone, which is exactly why shipped copies matter so much more than node-local state for anything that isn't caught within minutes.
• Treating OOMKilled or a generic exit code as a full diagnosis without corroborating log content risks a wrong root cause; use the event/exit-code evidence to narrow the hypothesis, not to close the investigation outright.