Log Analysis and Diagnostic Data Gathering Questions

Direct answer

For an occasional, ad-hoc 90-day investigation, reach for Athena (a serverless SQL query engine that reads directly from files in object storage, like Amazon S3) over logs converted to partitioned Parquet (a column-oriented file format, meaning it stores each field together across rows rather than row-by-row, which lets a query skip whole columns and files it doesn't need). It has no idle cost between investigations, and partition pruning plus columnar scanning is usually enough to hit "minutes" on a well-laid-out dataset. Reach for a small, narrowly-scoped Elasticsearch (or OpenSearch) tier only if you need frequent full-text or fuzzy search across raw unstructured message text; reach for Presto/EMR (a distributed SQL engine, Presto or its fork Trino, run on a managed Hadoop/Spark cluster service like Amazon EMR) only if your queries are heavier than Athena's engine handles well, e.g. large multi-way joins across datasets, or you already operate that cluster for other workloads.

Structured elaboration

The three options solve different problems, and the mistake is picking based on familiarity rather than the actual query shape:

• Athena over Parquet. You pay per byte scanned, not for idle cluster time, so it's ideal for a query pattern that's bursty (someone runs a handful of ad-hoc queries during an incident, then nothing for days). Its speed comes almost entirely from two things: converting raw JSON/text logs to a columnar format (so a query touching 3 of 20 fields only reads those 3 columns) and partitioning by date, and ideally also by a high-cardinality field like service, so a query for "the last 3 days, payments-api" physically skips every file outside that slice before scanning a single byte.
• A frozen/cold Elasticsearch (or OpenSearch) tier. Full-text search across raw, unstructured message text (fuzzy matching, "search for anything containing this stack trace fragment") is something a SQL engine over structured Parquet genuinely can't do well. A cold tier keeps that capability available for older data at lower cost than a fully hot cluster, but it's still a cluster you operate continuously, with real fixed cost, so it only earns its keep if full-text search across old data is a frequent, not occasional, need.
• Presto/EMR. Gives you the most raw compute and flexibility (arbitrary joins, UDFs (user-defined functions, custom logic you plug directly into the query), heavier transforms) but you're paying for cluster time even when idle, or paying the startup latency each time you spin one up on demand. For a single active investigation needing ad-hoc queries "back in minutes," standing up a cluster is usually a worse cost/latency trade than Athena unless the specific queries are things Athena's engine struggles with.

Worked example

Partitioning and format conversion aren't just an optimization, they're what makes "minutes not hours" possible at all. Say an investigation targets a 3-day window for one service. Without date/service partitioning, a query has to scan the entire 90-day corpus and filter afterward; with dt (date) and service as partition columns, the query planner discards every file outside that slice before reading anything, since partition values are encoded in the S3 key path itself (e.g. s3://bucket/logs/dt=2026-05-01/service=payments-api/) and never require opening the file to check:

-- Illustrative Athena/Presto SQL over a Glue-cataloged, partitioned Parquet table.
-- I have not executed this against a live Athena endpoint; the syntax (date
-- literals, partition predicates) is standard Presto/Trino SQL I'm confident in.
SELECT request_id, timestamp, level, message
FROM logs_parquet
WHERE dt BETWEEN date '2026-05-01' AND date '2026-05-03'
  AND service = 'payments-api'
  AND level = 'ERROR'
ORDER BY timestamp
LIMIT 500;

For illustration only (not a measured figure from a real system): if the full 90-day corpus is, say, 30 TB, and this query's partition predicate confines it to 3 days of one service's data, that's roughly a 30x reduction in candidate files before columnar pruning even applies, and columnar pruning (only reading the 4 selected columns instead of all 20) shrinks the scanned bytes further still. Both Athena's cost (billed per byte scanned) and its latency scale with bytes scanned, so partitioning and format conversion attack the same number that both cost and speed depend on.

Keeping it affordable without making cold logs unusable

• Run a daily batch job (Glue, Lambda, or a scheduled Spark job) that converts that day's raw logs to compacted, partitioned Parquet shortly after ingestion, so by the time anyone needs to query "yesterday," it's already in the fast format, not something you convert reactively during an incident.
• Keep the Parquet-converted data itself in S3's cheaper storage classes as it ages (moving from frequently-accessed to infrequent-access tiers) since Athena can query Parquet directly from those classes; you're not forced to choose between "cheap" and "queryable in minutes," only between "cheap and queryable in minutes" and "even cheaper but with retrieval latency," which is a separate, later archive tier for data you rarely touch.
• Maintain the Glue Data Catalog (the partition/schema metadata Athena queries against) so partition pruning keeps working as new partitions are added; a catalog that's out of sync with what's actually in S3 forces full scans and quietly destroys the whole point of partitioning.

Trade-offs & pitfalls

• Athena's per-query cost model means a badly-written query (missing partition predicates, SELECT * over unpruned data) can be surprisingly expensive and slow on the same dataset that a well-written query handles cheaply in seconds; enforcing partition predicates (and warning or blocking scans above a size threshold) matters operationally, not just as a style preference.
• A frozen Elasticsearch tier is still infrastructure you run 24/7; don't stand one up "just in case" if full-text search across 90-day-old logs is actually a rare need, the ongoing fixed cost won't be worth it.
• If schema drifts over time (new fields added, types changed), Parquet conversion and the catalog both need to handle schema evolution deliberately, or old partitions become unreadable or return nulls for fields that didn't exist yet when they were written.

Direct answer

Pull per-service error and request counts for the last 24 hours and the 24 hours before that, compute each service's error rate (errors divided by requests) in both windows, and rank services by how much that rate increased, with a minimum absolute-error floor so a service going from 1 error to 2 doesn't dominate the list.

Structured elaboration

• Inputs: two time-bounded queries against your metrics or log backend (Prometheus, an ELK-style log store, or a data warehouse table): current_window covering [now - 24h, now] and previous_window covering [now - 48h, now - 24h], each returning (service, error_count, request_count) rows.
• Aggregation method: sum errors and requests per service within each window (this is a single GROUP BY service in whichever backend you're querying), then compute error_rate = errors / requests for each service in each window.
• Threshold logic: flag a service as "rising" only if both hold: (1) the percentage increase in error rate versus the previous window exceeds a chosen threshold (e.g. 50%), and (2) the current window has at least some minimum absolute error count (e.g. 20). The second condition matters because a rate can double from a statistically meaningless base (1 error to 2) without being operationally significant; the floor filters that noise out.
• Output: a ranked list of (service, percent_increase, current_rate, current_error_count), truncated to the top N (e.g. top 3), suitable for a Slack message or a dashboard panel.

Worked example

def find_rising_services(current, previous, top_n=3, pct_threshold=50.0, min_errors=20):
    results = []
    for service in set(current) | set(previous):
        cur = current.get(service, {"errors": 0, "requests": 1})
        prev = previous.get(service, {"errors": 0, "requests": 1})
        rate_cur = cur["errors"] / (cur["requests"] or 1)
        rate_prev = prev["errors"] / (prev["requests"] or 1)
        pct_increase = ((rate_cur - rate_prev) / rate_prev * 100 if rate_prev > 0
                         else 100.0 if rate_cur > 0 else 0.0)
        if cur["errors"] >= min_errors and pct_increase >= pct_threshold:
            results.append((service, pct_increase, rate_cur, cur["errors"]))
    results.sort(key=lambda r: r[1], reverse=True)
    return results[:top_n]

Run against pinned example data for four services (current vs. previous 24h window):

current_window = {
    "checkout":  {"errors": 480, "requests": 40000},
    "search":    {"errors": 55,  "requests": 50000},
    "auth":      {"errors": 12,  "requests": 30000},
    "recommend": {"errors": 300, "requests": 20000},
}
previous_window = {
    "checkout":  {"errors": 200, "requests": 39000},
    "search":    {"errors": 50,  "requests": 49000},
    "auth":      {"errors": 10,  "requests": 29500},
    "recommend": {"errors": 40,  "requests": 19800},
}
for service, pct, rate, errs in find_rising_services(current_window, previous_window):
    print(f"{service:<10} +{pct:6.1f}%   current rate={rate:.4%}   errors={errs}")

Output:

recommend  + 642.5%   current rate=1.5000%   errors=300
checkout   + 134.0%   current rate=1.2000%   errors=480

search (7.8% increase) and auth (18% increase, and only 12 errors, below the floor) are correctly excluded: their rate barely moved and/or their volume is too low to act on.

Trade-offs & pitfalls

• Comparing only two adjacent 24-hour windows is sensitive to day-of-week effects (a Monday will look "worse" than a quiet Sunday even with no real regression); a production version would compare against the same weekday a week prior, or a rolling baseline, instead.
• A hard percentage threshold treats a jump from 0.001% to 0.002% the same as 10% to 20% unless the minimum-error floor is tuned well; the floor is doing real work here, not just cosmetic filtering.
• This is a scan-and-report utility, not a real-time alert; running it every few minutes as a scheduled job is usually enough, since "rising over 24h" is inherently a slow-moving signal, not something that needs sub-second reaction.

Direct answer

Centralizing logs trades local simplicity for a single, searchable, cross-host view, at the cost of introducing a new dependency and a new potential single point of failure into the very system you rely on during an incident. Whether that trade is worth it depends on the team's actual scale and pain today, not on centralized logging being a generically "better" architecture.

Structured elaboration

Benefits

• One search surface across every host, instead of SSHing into individual machines to grep local files, which becomes painful fast once you have more than a handful of hosts or any ephemeral/autoscaled infrastructure where a host might not even exist anymore by the time you go looking.
• Consistent retention and access control applied in one place, rather than as many slightly-different local configurations as there are hosts.
• Cross-service correlation (a request spanning several services) becomes a single query instead of manually stitching together several hosts' files by hand.

Risks

• Reliability: the centralized system becomes a dependency of last resort during an incident; if it's degraded at the exact moment you need it (which happens more often than teams expect, since incidents and infrastructure stress correlate), you've lost your primary investigation tool.
• Latency: shipping logs off-host adds a delay between an event happening and it being searchable centrally; for a live, fast-moving incident, that lag matters.
• Privacy and compliance: centralizing means log data, which can contain user identifiers or other sensitive fields, now flows through and is stored in one additional system, which is a new surface to get access control, encryption, and retention policy right on, and potentially a new data-residency concern if that store lives in a different region than the data originated.
• Single points of failure: if the central pipeline or store goes down, you don't just lose the aggregation, you can lose the ability to investigate anything happening right now, unless hosts still retain their own local copies as a fallback.

Worked example

A team running 6 long-lived hosts, where an engineer can still reasonably SSH into each one and grep during an incident, gets comparatively little benefit from centralizing today, the local-file workflow isn't actually painful yet, while taking on real risk (a new dependency, new compliance surface) for benefit they're not using. A team running 200 short-lived, autoscaled containers where a host that logged the interesting event may no longer exist an hour later has effectively no working alternative to centralizing; without it, evidence disappears on its own, unrelated to any bug.

Trade-offs & pitfalls

The decision isn't "centralize or don't," it's "how do you decide whether centralizing addresses this team's actual pain." A reasonable framing: ask whether the team is already spending real, recurring time and pain on the local-file workflow (SSHing into many hosts, losing logs when instances recycle), whether the team has the operational capacity to run or pay for a reliable central pipeline (including its own failure and backpressure handling), and whether the compliance and access-control requirements are things a centralized store would actually satisfy or, worse, entrench a bad practice around handling sensitive fields. If those answers point toward hybrid, keep local retention as a fallback even after centralizing, so an outage in the central pipeline doesn't leave the team with zero evidence for the exact incident that pipeline outage might itself be causing.

Direct answer

Grok is the pattern-matching DSL (domain-specific language, a small purpose-built syntax rather than a general programming language) that Logstash's grok filter uses to name regex fragments and compose them, so you write field names instead of raw regex. The Logstash grok pattern below is illustrative configuration (not something this environment can execute against a real Logstash pipeline); the field boundaries it encodes are verified separately with an equivalent, executed Python regex against a real nginx combined log line.

Approach

Logstash grok pattern

%{IPORHOST:client_ip} - %{USER:ident} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:path} %{DATA:protocol}" %{NUMBER:status} %{NUMBER:bytes_sent} "%{DATA:referrer}" "%{DATA:user_agent}"

Each %{PATTERN:field_name} is a named, reusable regex fragment from grok's built-in pattern library: %{IPORHOST} matches an IP address or hostname, %{USER} matches a username token, %{HTTPDATE} matches Apache's bracketed date format, %{WORD} matches one alphabetic token, and %{NUMBER} matches a numeric string, each expanding to a pre-built regex fragment so you don't have to write it by hand. %{DATA} is grok's non-greedy "anything" pattern, used here for path, protocol, referrer, and user_agent because those fields can contain almost any character. Note that grok's own built-in COMBINEDAPACHELOG pattern names the trailing HTTP/1.1 token httpversion and only captures the version number, not the scheme; since the question asks specifically for a field named protocol, this pattern captures the whole HTTP/1.1 token (scheme and version together) under that name instead, which is the more literal reading of "protocol."

Verifying the field boundaries (executed)

import re

# Same field boundaries as the grok pattern
# %{IPORHOST:client_ip} - %{USER:ident} \[%{HTTPDATE:timestamp}\]
# "%{WORD:method} %{DATA:path} %{DATA:protocol}"
# %{NUMBER:status} %{NUMBER:bytes_sent} "%{DATA:referrer}" "%{DATA:user_agent}"
PATTERN = re.compile(
    r'^(?P<client_ip>\S+) - (?P<ident>\S+) '
    r'\[(?P<timestamp>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+) (?P<protocol>[A-Z]+/[\d.]+)" '
    r'(?P<status>\d+) (?P<bytes_sent>\d+|-) '
    r'"(?P<referrer>[^"]*)" '
    r'"(?P<user_agent>[^"]*)"$'
)

line = ('203.0.113.5 - - [10/Oct/2024:13:55:36 -0700] '
        '"GET /search?q=test%20query HTTP/1.1" 200 5231 '
        '"https://example.com/" '
        '"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"')

m = PATTERN.match(line)
if m:
    for k, v in m.groupdict().items():
        print(f"{k}: {v}")
else:
    print("no match")

Worked example (executed output)

client_ip: 203.0.113.5
ident: -
timestamp: 10/Oct/2024:13:55:36 -0700
method: GET
path: /search?q=test%20query
protocol: HTTP/1.1
status: 200
bytes_sent: 5231
referrer: https://example.com/
user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

Key points

• Query strings in the path: %{DATA:path} is non-greedy, so it stops at the first space rather than trying to consume the rest of the line; a query string like ?q=test%20query stays part of the path field intact, exactly as shown above (/search?q=test%20query). If you need path and query as separate fields, add a dissect or mutate step after the grok match to split on the first ?.
• Percent-encoding: neither grok nor this regex decodes %20 into a space; that's expected; percent-encoding is part of the raw URL and decoding it is a separate, deliberate step (only do it if you actually need the decoded form for display or analysis, since the encoded form is what the server actually received).
• Very long user agents and mapping explosion: in Elasticsearch, a text or default-mapped keyword field indexes every unique value it sees; browser and bot user-agent strings are extremely high-cardinality (having a very large number of distinct values) and some are very long, which can blow up index size and even hit Elasticsearch's field-length limits. The practical mitigations are: map user_agent as keyword with ignore_above set (so overlong values are stored but not indexed for aggregation), truncate the field before indexing if you only need it for display, and, if you need to group by user agent at all, index a hash of the full string instead of the raw string for aggregation purposes.
• Naming protocol vs. a library's default field name: this is a small but real pitfall in its own right. A team pulling in a stock COMBINEDAPACHELOG-style pattern will get httpversion, not protocol, and a downstream query or dashboard written against "protocol" will silently return nothing until someone notices the field doesn't exist. When a spec (a question, a ticket, a schema doc) names a field explicitly, match that name exactly, or add an explicit rename/alias step, rather than assuming a library's default naming is close enough.

Trade-offs & pitfalls

The main pitfall with grok specifically is that a malformed line (one that doesn't match the pattern at all) produces no match and typically gets tagged as a parse failure rather than partially parsed; a production pipeline needs a fallback path for those lines rather than silently dropping them. The other common mistake is treating query strings as low-cardinality: a path with a highly variable query string (a session token, a timestamp) indexed as a single keyword field will itself cause the same kind of cardinality blowup as an unbounded user-agent field, so the same truncate-or-split discipline usually needs to apply to path as well, not just to user_agent.

Direct answer

Stream each file line by line rather than reading it into memory, filter to the matching request_id and timestamp range as you go, and merge the per-file streams with a k-way merge, using a min-heap (a small structure that always exposes the smallest of the items it holds) keyed by timestamp, so the combined output comes out globally chronological without ever holding more than one buffered record per file at once.

Approach

import json
import heapq
from datetime import datetime


def _parse_ts(raw):
    return datetime.fromisoformat(raw.replace("Z", "+00:00"))


def _file_matches(path, request_id, start_ts, end_ts):
    """Stream one file line by line. Assumes each file is internally
    time-ordered (true of an append-only log file), so this generator
    yields matching events in ascending timestamp order for THIS file.
    Never holds more than the current line in memory.
    """
    with open(path, "r") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue
            if rec.get("request_id") != request_id:
                continue
            raw_ts = rec.get("ts")
            if raw_ts is None:
                continue
            try:
                event_ts = _parse_ts(raw_ts)
            except ValueError:
                continue
            if start_ts <= event_ts <= end_ts:
                rec["_ts_parsed"] = event_ts
                yield rec


def extract_events(file_paths, request_id, start_ts, end_ts):
    """Yield events with the given request_id whose timestamp falls in
    [start_ts, end_ts], in chronological order, across multiple large
    NDJSON log files, without loading any file fully into memory.

    Each file is scanned by its own generator (bounded to one open line
    at a time); a min-heap does a k-way merge across those generators so
    only one buffered record per file is held at once, not all matches.
    Memory is O(number of files), not O(file size) or O(match count).
    """
    heap = []
    iterators = [iter(_file_matches(p, request_id, start_ts, end_ts)) for p in file_paths]
    for idx, it in enumerate(iterators):
        first = next(it, None)
        if first is not None:
            heapq.heappush(heap, (first["_ts_parsed"], idx, first))

    while heap:
        ts, idx, rec = heapq.heappop(heap)
        del rec["_ts_parsed"]
        yield rec
        nxt = next(iterators[idx], None)
        if nxt is not None:
            heapq.heappush(heap, (nxt["_ts_parsed"], idx, nxt))


if __name__ == "__main__":
    import os

    file_a = "/tmp/loganalysis_b1_s13_a.ndjson"
    file_b = "/tmp/loganalysis_b1_s13_b.ndjson"

    # file_a and file_b are each individually time-ordered, as a real
    # per-host or per-service log file would be, even though the SET of
    # files is not globally ordered relative to each other.
    with open(file_a, "w") as f:
        f.write(json.dumps({"ts": "2025-01-01T00:00:01Z", "request_id": "req-1", "service": "C", "msg": "too_early"}) + "\n")
        f.write(json.dumps({"ts": "2025-01-01T00:00:02Z", "request_id": "req-1", "service": "A", "msg": "start"}) + "\n")
        f.write(json.dumps({"ts": "2025-01-01T00:00:09Z", "request_id": "req-2", "service": "A", "msg": "other_request"}) + "\n")
        f.write("not valid json\n")
        f.write(json.dumps({"ts": "2025-01-01T00:00:20Z", "request_id": "req-1", "service": "A", "msg": "too_late"}) + "\n")

    with open(file_b, "w") as f:
        f.write(json.dumps({"ts": "2025-01-01T00:00:05Z", "request_id": "req-1", "service": "B", "msg": "middle"}) + "\n")

    start = _parse_ts("2025-01-01T00:00:02Z")
    end = _parse_ts("2025-01-01T00:00:10Z")

    for rec in extract_events([file_a, file_b], "req-1", start, end):
        print(rec)

    os.remove(file_a)
    os.remove(file_b)

Worked example (executed output)

{'ts': '2025-01-01T00:00:02Z', 'request_id': 'req-1', 'service': 'A', 'msg': 'start'}
{'ts': '2025-01-01T00:00:05Z', 'request_id': 'req-1', 'service': 'B', 'msg': 'middle'}

file_a and file_b are individually time-ordered (as a real append-only log file would be), but the two files are not ordered relative to each other, and file_a even has an out-of-range line, a wrong-request-id line, and a malformed line mixed in. The output above still comes out correctly filtered and in chronological order across both files.

Key points

• Memory usage is O(number of files), not O(file size) or O(number of matches): at any moment, the heap holds at most one buffered record per file, since each file is its own generator that only produces its next matching record when asked.
• The k-way merge relies on one precondition worth stating explicitly: each individual file must already be time-ordered internally. That's a safe assumption for a normal append-only application log, but it would need to be checked (or the approach would need to change) for a log source that writes out of order within a single file.
• extract_events is itself a generator (it yields rather than returning a list), so a caller can start consuming matching events immediately and stop early if it only needs the first few, rather than waiting for the entire multi-file scan to finish.

Complexity

Let N be the total number of matching events across all files and k be the number of files. Each heap push/pop is O(log k), and there are O(N) of them (one per yielded event, plus refills), so total time is O(N log k). Space is O(k) for the heap itself, plus the fixed per-line buffer each open file uses while being read, which is independent of file size.

Edge cases

• Malformed JSON lines: skipped rather than raising, so one corrupt line doesn't abort the entire scan (shown above: file_a's "not valid json" line is silently passed over).
• A file with no matches at all: its generator is simply exhausted immediately and it drops out of the heap; the merge continues correctly with the remaining files.
• Ties on identical timestamps across files: the heap comparison falls through to the file index as a tiebreaker, so results are still deterministic (files earlier in the input list sort first on an exact tie) rather than raising a comparison error on the underlying dict.
• A file that isn't actually time-ordered internally: this breaks the merge's chronological guarantee silently (each file's own stream would emit an out-of-order record without the heap detecting it), which is worth flagging explicitly rather than assuming away.

Trade-offs & pitfalls

A simpler alternative, collect every match into a list and sort once at the end, is easier to write and is fine when you expect a bounded, small number of matches for one request_id (usually true; a single request rarely touches millions of log lines). The heap-based merge shown here is the more defensible answer for a "must stream, must not load fully into memory" prompt because it doesn't rely on that assumption: it stays memory-bounded even in the pathological case where one request_id matches a huge number of events, which the simpler collect-and-sort approach would not.