Security Monitoring, SIEM, and Detection Engineering Questions
Building and operating the detection stack: the SOC and detection-engineering practice that answers 'can we see an attack happening.' Covers SIEM platform selection and architecture, use-case and detection-rule query development (for example Splunk SPL, KQL, or Sigma), alert triage and tuning to reduce false positives, detection engineering and closing coverage gaps, mapping detections to the MITRE ATT&CK framework and scoring detection coverage, log analysis and anomaly and baseline development, network and endpoint telemetry sourcing, malware and compromise-indicator recognition, and security operations center (SOC) alert escalation workflows. Distinct from the reactive work of containing, remediating, and communicating during a confirmed incident (incident response and postmortem topics own that ground; this topic stops at 'the alert fired and here is the detection logic', not 'here is how we contained and recovered from it'). Distinct from generic system-reliability monitoring and observability (SLOs, error budgets, uptime dashboards), a separate discipline even when the underlying ingestion mechanics look similar; the anomaly or signal here must be framed as adversarial or security-relevant. Distinct from hardening a software delivery pipeline against supply-chain compromise (SBOM generation, artifact signing, dependency and build-permission controls); this topic only touches the delivery pipeline from the detection side, spotting a compromised build or tainted artifact via telemetry, not the preventive-controls side. Distinct from designing security control architecture and governance (security architecture and cloud security architecture topics own the design-time question of what controls should exist); this topic is the run-time operation of the detection stack once those controls are in place.
Design a simple detection rule to identify potential brute-force login attempts on an authentication service. Specify thresholds (for example, failed attempts per account and per source IP over a time window), explain why both account- and source-based thresholds matter, and propose techniques to avoid noisy alerts in shared or high-traffic services.
Sample Answer
Direct answer
A basic brute-force detection needs TWO independent thresholds, not one, failed attempts per ACCOUNT over a time window (catches an attacker targeting one specific account from anywhere) and failed attempts per SOURCE IP over a time window (catches an attacker spraying many accounts from one place); either threshold alone misses the attack pattern the other is designed to catch.
Structured elaboration
Account-based threshold: for example, 10 failed attempts for the SAME account within 10 minutes, regardless of source IP. This catches an attacker who has one target account and is guessing passwords against it, possibly rotating through multiple source IPs specifically to evade an IP-only threshold.
Source-based threshold: for example, 20 failed attempts from the SAME source IP within 10 minutes, regardless of which account is targeted. This catches an attacker spraying many different account/password combinations from one location, a pattern the account-based threshold alone would miss entirely, since no single account individually crosses its own threshold.
Why both matter: an attacker aware of only an account-based threshold defeats it by spreading attempts across MANY accounts from one source (password spraying); an attacker aware of only a source-based threshold defeats it by rotating source IPs while hammering one account. Requiring both thresholds, evaluated independently, closes each evasion path the other threshold alone would leave open.
Techniques to avoid noisy alerts in shared or high-traffic services: for a shared authentication service (a VPN gateway, a public-facing login endpoint used by many legitimate users), a fixed absolute threshold calibrated for a low-traffic internal service will be far too sensitive; scale the threshold relative to the service's own observed baseline failure rate (a service with naturally high failure volume, due to normal user error at scale, needs a proportionally higher threshold than a low-traffic internal tool), and exclude known, legitimate high-volume automated sources (health checks, load-balancer probes) from the source-based count entirely.
Worked example
Concretely, a shared corporate VPN gateway naturally sees several hundred failed login attempts per day from ordinary user typos and expired passwords, spread across many accounts and many source IPs. A fixed threshold of "10 failures per IP in 10 minutes" calibrated for a low-traffic internal tool would fire constantly against this gateway's normal background noise. Recalibrating the source-based threshold to this specific service's own observed baseline (for instance, requiring failure volume from a single IP to exceed several times the service's typical per-IP failure rate within the window, derived from real historical data rather than an arbitrary round number) brings the detection back to a usable signal-to-noise ratio for THIS specific service, while the account-based threshold, which is inherently less affected by a shared service's overall traffic volume (since it is scoped to one account at a time), needs comparatively less service-specific recalibration.
Trade-offs and pitfalls
- Common mistake: implementing only one of the two thresholds, since it is simpler, and treating the resulting detection as complete brute-force coverage; as the elaboration above shows, each threshold has a specific, distinct evasion path the other is designed to close.
- Common mistake: using the SAME absolute threshold value across every service regardless of its own traffic volume and baseline noise level; the worked example demonstrates directly why a threshold appropriate for a low-traffic tool floods a high-traffic shared service with noise.
- This is the entry-level, non-correlated version of brute-force detection: two independent per-account and per-source thresholds are a reasonable starting point, while a fuller multi-device correlation approach adds explicit cross-host and cross-account correlation logic on top of it, catching a coordinated attacker who deliberately keeps each individual account or source just under ITS OWN threshold while the combined pattern across many accounts and sources is still clearly anomalous.
- Both thresholds should be tunable per service, not global constants: a single organization-wide threshold value, hardcoded once, cannot serve a genuinely diverse set of services with very different legitimate traffic volumes, which is exactly the noisy-shared-service problem the question specifically asks how to avoid.
Define Endpoint Detection and Response (EDR). List common types of EDR telemetry (process creation, network connections, file/registry changes, memory artifacts), three typical automated EDR response actions, and explain how EDR differs from traditional antivirus in detection approach.
Sample Answer
Direct answer
Endpoint Detection and Response (EDR) is host-based security software that continuously records detailed endpoint activity (not just known-bad file signatures), applies behavioral and signature-based detection to that activity, and gives responders the ability to investigate and act directly on the endpoint, capabilities that traditional antivirus, focused narrowly on scanning files against known-malware signatures, does not provide.
Structured elaboration
Common types of EDR telemetry:
- Process creation events: every process launched, its command line, parent process, and the user context it ran under, the single most information-dense EDR signal for reconstructing what actually happened on a host.
- Network connections: outbound and inbound connections initiated by monitored processes, including destination IP/port and the specific process responsible, tying network activity back to a specific piece of software rather than just the host as a whole.
- File/registry changes: file creation, modification, and deletion, plus (on Windows) registry key changes, which is how persistence mechanisms and configuration tampering are typically observed.
- Memory artifacts: information about loaded modules, injected code, or suspicious memory regions, which is what lets EDR see fileless or in-memory-only attack techniques that leave no trace on disk for a file-based signature to ever match.
Three typical automated EDR response actions:
- Kill the malicious process immediately upon high-confidence detection, halting further execution without waiting for human review.
- Isolate the host from the network (while typically preserving a management channel for the EDR agent itself), containing the blast radius while an analyst investigates.
- Quarantine the malicious file, moving or encrypting it so it cannot be executed again even if the process itself has already been killed, while preserving it for later forensic analysis rather than deleting it outright.
How EDR differs from traditional antivirus (AV) in detection approach: traditional AV is primarily signature-based, matching files against a database of known-malware hashes or patterns, scanning at rest or on execution, with relatively limited behavioral visibility beyond that scan. EDR combines lighter-weight signature matching with continuous behavioral monitoring across process, network, file, and memory activity, is built to detect NOVEL and fileless threats that have no matching signature, and, critically, retains a rich forensic record of endpoint activity that an investigator can query after the fact, something traditional AV, designed to prevent and clean rather than to support investigation, was never built to provide.
Worked example
A fileless attack technique that abuses a legitimate, signed system tool to execute malicious code entirely in memory (no new file ever written to disk) is functionally invisible to a pure file-signature AV engine, since there is no malicious file for a hash-based signature to ever match, the "malicious" thing here is a sequence of BEHAVIOR (an unusual parent-child process relationship, a suspicious in-memory code injection into that trusted tool), not a static artifact. EDR's process-creation and memory-artifact telemetry is specifically what makes this visible: a behavioral rule watching for "this trusted system tool's process just had code injected into it from an unrelated parent process" would fire on the pattern even though no file-based signature exists for this specific attack at all.
Trade-offs and pitfalls
- EDR is not a replacement for every AV capability: some traditional AV functions (fast, low-overhead signature scanning of files at rest, useful for catching known commodity malware cheaply) remain valuable as a complementary first layer; most modern endpoint platforms actually bundle both under one product rather than treating them as competing choices.
- Automated response actions carry real operational risk: automatically isolating a production server on a false positive is itself an outage; the confidence threshold for fully automated action (versus alert-and-await-analyst-confirmation) should be set deliberately higher than the threshold for merely generating an alert.
- Common mistake: assuming EDR coverage is complete because "the agent is installed," without validating the agent is actually reporting all the telemetry types described above; a misconfigured or degraded agent can silently drop entire categories of telemetry (memory visibility in particular is often the first thing to degrade under resource pressure) while still showing as "healthy" in a basic connectivity check.
- Common mistake: treating EDR telemetry as sufficient on its own without network- and identity-layer telemetry; EDR sees what happens ON the endpoint, but lateral movement and account-based attacks often need network and identity telemetry (as covered by HIDS/NIDS and identity-log-focused questions) to see the full picture across multiple hosts.
Explain why mapping detection use cases to the MITRE ATT&CK framework is valuable. Provide three concrete examples showing how mapping to ATT&CK techniques influences the telemetry you collect and the specific detection logic you would implement.
Sample Answer
Direct answer
Mapping detection use cases to MITRE ATT&CK is valuable because it turns "what should we build detections for" from an open-ended, intuition-driven question into a structured one grounded in real, observed adversary behavior, and it does this concretely by shaping BOTH what telemetry a team collects and what specific detection logic it writes, not just how coverage gets reported afterward.
Structured elaboration
The mechanism is direct: a technique entry in ATT&CK describes a specific adversary behavior at a level of detail that implies specific, checkable telemetry requirements and specific, checkable detection logic, rather than a vague threat category. Three concrete examples showing this influence explicitly:
Example 1: T1003.001 (OS Credential Dumping: LSASS Memory). Mapping to this specific sub-technique tells a team exactly what telemetry to prioritize collecting, process-level visibility into which processes access LSASS memory, and with what access rights, since generic process-creation logging alone does not capture this. It also tells the team exactly what detection logic to build: a rule watching for non-standard processes requesting high-privilege memory access to the LSASS process, not a generic "suspicious process" rule.
Example 2: T1071.004 (Application Layer Protocol: DNS). Mapping to this technique tells a team that DNS query telemetry (not just DNS server logs, but query-level detail including subdomain content and frequency) needs to be collected with enough fidelity to support entropy and length-based analysis, and it tells the team the corresponding detection logic needs to score DNS query PATTERNS, not just match known-bad domains from a static blocklist.
Example 3: T1053.005 (Scheduled Task/Job: Scheduled Task). Mapping to this technique tells a team that Windows Event ID 4698 (scheduled task creation) needs to be collected and, critically, that the detection logic needs to distinguish EXPECTED scheduled-task creation (from known administrative or automation accounts) from unexpected creation, which in turn implies the team also needs an enrichment source identifying which accounts are legitimately expected to create scheduled tasks, a requirement that would not have been obvious from a generic "watch for persistence" goal alone.
Worked example
Without ATT&CK mapping, a team asked to "improve credential-theft detection" might reasonably start anywhere, broad login-anomaly detection, password-policy auditing, or generic process monitoring, all defensible but unfocused starting points. With the T1003.001 mapping specifically, the team's very first engineering task is unambiguous: confirm LSASS memory-access telemetry is actually being collected at all (frequently it is not, by default, on a standard endpoint configuration), and if not, that becomes the FIRST, concrete, prioritized backlog item, derived directly from the technique mapping rather than from a general sense that "credential theft is bad."
Trade-offs and pitfalls
- Mapping fields belong in the rule and playbook artifacts themselves, not just a separate tracking spreadsheet: each detection rule's own metadata should carry its mapped technique ID(s) directly, and incident-response playbooks referencing a given technique should link back to which specific rules provide coverage for it, so the mapping is a living, queryable part of the operational tooling rather than a document that goes stale the moment it is written.
- A practical use of the mapping in sprint/backlog prioritization: when a detection-engineering team plans its next work cycle, technique-level coverage gaps (a technique relevant to the organization's threat model with zero or weak mapped detections) are a concrete, defensible prioritization input, arguably a stronger one than "which rule idea sounds most interesting," since it ties engineering time directly to a documented, real gap rather than to intuition.
- Common mistake: treating the mapping as a one-time labeling exercise disconnected from actual engineering work; the value described above only materializes if the mapping genuinely DRIVES telemetry and detection-logic decisions, not if it is applied retroactively as a label on rules that were designed independently of it.
- A technique mapping does not by itself guarantee the mapped rule actually works: a mapping records intent, not proof, and should be validated (ideally via red-team or purple-team testing) to confirm the mapped detection genuinely fires against a realistic instance of the technique, not just that someone tagged a rule with the right ID.
Architect a multi-tenant SIEM for a SaaS provider expected to ingest 1,000,000 events/sec. Describe how you would handle tenant isolation (logical and physical), routing and partitioning of data, index/tenant mapping, query latency expectations, encryption at rest/in transit, access control and RBAC, schema/versioning, and cost allocation between tenants. Address operational concerns like scaling, backups, cross-region compliance, and tenant admin functions.
Sample Answer
Direct answer
A 1,000,000-events/sec multi-tenant SaaS SIEM needs the same tiered-ingestion and tiered-storage backbone as a single-tenant design, plus a tenant identity that is enforced at every layer (routing, indexing, query, billing), not just at the application's login screen; the single most consequential design decision is whether tenant isolation is LOGICAL (shared infrastructure, isolated by access control and index scoping) or PHYSICAL (separate infrastructure per tenant or tenant tier), because that choice cascades into every other requirement listed.
Structured elaboration
Tenant isolation, logical vs. physical. Logical isolation (shared clusters, per-tenant index prefixes or namespaces, enforced by RBAC at query time) is far more cost-efficient at scale and is the default for small-to-mid tenants. Physical isolation (dedicated infrastructure, sometimes a fully separate deployment) costs more per tenant but is often required for the largest or most regulated tenants (financial services, government), where a shared-infrastructure security or compliance argument is not acceptable to their auditors regardless of how well the logical controls are implemented. A practical design offers a tiered model: most tenants on shared, logically-isolated infrastructure, with a physically-isolated tier available for tenants whose contracts or regulatory posture require it.
Routing and partitioning. Partition ingestion by tenant ID as the primary key, sharded further by time, so no single tenant's traffic spike can starve another tenant sharing the same ingestion partition, and so a single tenant's data can be located, rebalanced, or (if needed) physically relocated without touching every other tenant's data.
Index/tenant mapping. Each tenant maps to its own index (or index-per-tenant-per-day for time-rolled indices), never a shared index filtered at query time by a tenant field alone; a query-time-only filter is one bug away from a cross-tenant data leak, whereas a genuinely separate index makes that class of bug structurally impossible, not just policy-forbidden.
Query latency expectations. Set an explicit service-level objective (SLO) tiered by data age, consistent with the storage tiers: sub-second to low-single-digit-second latency for recent (hot) data typical of interactive analyst search, and a higher latency budget (seconds to tens of seconds) accepted for warm/cold-tier historical queries, communicated to tenants as part of the platform's contract rather than left implicit.
Encryption at rest/in transit. Transport Layer Security (TLS) for every hop (as in a single-tenant design) plus, for at-rest encryption, tenant-scoped or tenant-provided (customer-managed) encryption keys for tenants whose compliance requirements demand cryptographic isolation, not just access-control isolation, of their data from every other tenant, including the platform operator's own other customers.
Access control and role-based access control (RBAC). Two RBAC layers: platform-level (which tenant can a given credential even see) and within-tenant (which roles inside that tenant can view raw events versus only dashboards, or administer detection rules versus only read alerts), since a tenant's own internal separation-of-duties requirements do not disappear just because they are a customer of a shared platform.
Schema/versioning. A shared normalized schema across tenants (so platform-wide detection content, like out-of-the-box detection rules, works identically for every tenant) with explicit schema versioning, so a schema change can roll out to some tenants before others and a rule author can target a specific schema version rather than assuming every tenant is on the latest one at all times.
Cost allocation between tenants. Meter ingestion volume, storage footprint, and query compute per tenant (the three cost drivers that scale with usage) and allocate shared infrastructure overhead (the ingestion bus, the control plane) either evenly or proportionally to usage, so a heavy tenant's costs are visible and attributable rather than silently subsidized by lighter tenants.
Operational concerns:
- Scaling: horizontal, partition-based scaling as in a single-tenant design, but with tenant-aware auto-scaling triggers, since a spike from one large tenant should not trigger platform-wide scale-up if smaller tenants are unaffected.
- Backups: per-tenant backup scoping (so a single tenant's data can be restored independently, and so a tenant offboarding can be cleanly and completely deleted, including from backups, for compliance) rather than one platform-wide backup blob.
- Cross-region compliance: tenants in regulated jurisdictions may require their data to never leave a specific region (data residency); this needs to be a routing-time decision (which region's cluster ingests this tenant's data) enforced structurally, not a post-hoc replication policy.
- Tenant admin functions: expose tenant-scoped self-service capabilities (view usage/cost, manage within-tenant RBAC, configure their own detection rules and retention within platform-allowed bounds) without ever granting cross-tenant visibility through the admin surface itself, which is a common and easy-to-miss privilege-escalation path if the admin API is not scoped as strictly as the data API.
Worked example
At 1,000,000 events/sec, using the same 500-byte average event size assumption and methodology from a single-tenant sizing exercise: raw ingestion is 1,000,000×500×86,400=4.32×1013 bytes/day, 43.2 TB/day raw. Applying the same 1.3x index overhead and 2x replication used for a single-tenant hot/warm tier gives 43.2×1.3×2=112.32 TB/day for the hot/warm indexed footprint, ten times the single-tenant 100,000-EPS example, which is the expected linear relationship since both event size and overhead assumptions are unchanged and only EPS scaled by 10x. The practical implication: at this scale, per-tenant index-per-day partitioning is not optional, a single unpartitioned index holding 112 TB/day worth of documents would make even simple time-bounded queries prohibitively slow, and per-tenant deletion (for offboarding or retention expiry) would require rewriting a massive shared structure instead of simply dropping that tenant's own daily indices.
Trade-offs and pitfalls
- Logical isolation is cheaper but carries residual risk: a bug in query-time tenant scoping is a genuine cross-tenant breach, not a performance issue, so logical isolation needs both index-level separation (structural) AND RBAC enforcement (policy) as defense in depth, not either alone.
- Common mistake: metering cost per tenant only on storage, while ignoring query compute; a tenant that stores little data but runs expensive, wide-ranging searches constantly can cost more in compute than a tenant with ten times the storage footprint who rarely queries historical data.
- Common mistake: treating cross-region data residency as a replication-layer afterthought; if ingestion ever routes a tenant's raw data through a region it is not allowed to touch, even transiently, the residency guarantee is already broken regardless of where the data is FINALLY stored.
- Common mistake: building tenant admin self-service functions against the same underlying API surface as platform-internal admin tools without a hard tenant-scoping boundary; this is a realistic path to a tenant being able to see or affect another tenant's configuration if the scoping is enforced only by the ADMIN UI and not by the underlying API itself.
With very limited endpoint telemetry (process creation events and summarized netflow only), propose statistical heuristics and feature engineering to detect abuse of LOLBins. Suggest concrete features (for example: process-parent novelty score, command-line entropy proxy, atypical destination score, time-of-day deviation) and describe how you would combine them into a scoring model or anomaly detector.
Sample Answer
Direct answer
With only process-creation events and summarized (non-packet-level) netflow, LOLBin abuse detection has to lean entirely on BEHAVIORAL statistics computed from what little structure those two sources provide, rather than any content inspection, since there is no command-line-argument detail, file-content, or per-packet visibility available; the practical approach is a small set of cheaply-computed numeric features per process-creation event, combined into a single weighted or learned score rather than any one feature acting as a hard rule, since each individual feature alone is only weakly discriminating at this telemetry depth.
Structured elaboration
Process-parent novelty score: for each observed (parent process, child process) pair, maintain a per-host or per-fleet historical frequency count, and score a NEW event by how rarely (or never) that specific parent-child pairing has been observed before, for example score = -log(observed_frequency + epsilon), so a pairing seen thousands of times scores near zero and a never-before-seen pairing scores high; this directly targets the classic LOLBin abuse pattern of a legitimate binary (certutil.exe, mshta.exe, regsvr32.exe) being launched by an UNUSUAL parent (a browser, an Office application, a scripting host) rather than its normal parent (a shell or scheduled task).
Command-line entropy proxy: since full command-line content may not be captured at this telemetry depth, approximate obfuscation signal from whatever IS available, process name length and character-class mix (if partial command-line or image-path data exists), or, if command-line truly is unavailable, a proxy built from process-creation RATE and pattern instead (a burst of LOLBin-family process creations in a short window is itself an entropy-like signal of scripted/automated invocation rather than manual interactive use); where at least a truncated command-line is available, Shannon entropy over the character distribution is a cheap, standard obfuscation proxy (H = -sum(p_i * log2(p_i)) over character frequencies), with a high value pointing toward base64/hex-encoded payloads.
Atypical destination score: pair each process-creation event with the summarized netflow occurring in a short window AFTER it (the process launched, then made a connection), and score the destination by novelty against that HOST's own historical destination set (has this host ever connected to this destination/ASN before) rather than a global allowlist/denylist, since a global list misses host-specific abuse while a per-host novelty baseline catches "this LOLBin just talked to somewhere this host has never gone."
Time-of-day deviation: score each event's timestamp against the HOST's (or the responsible user account's) own historical activity-time distribution, not a fleet-wide baseline, since normal working hours vary by role and geography; a LOLBin invocation at 3am on a workstation with no history of any activity at that hour is a stronger signal than the same invocation timing on a server that runs scheduled maintenance nightly.
Combining into a scoring model: given how weak any single feature is at this telemetry depth, a WEIGHTED linear combination or a simple logistic-regression-style scoring function (score = w1*parent_novelty + w2*entropy_proxy + w3*destination_novelty + w4*time_deviation) calibrated against a labeled or semi-labeled sample of known-benign and known-suspicious LOLBin invocations is more robust than any single hard threshold on one feature; the weights themselves can start as analyst-assigned priors (parent novelty weighted highest, since it is the most directly diagnostic of the four at this telemetry depth) and be refined once enough labeled feedback accumulates to fit them properly.
Trade-offs and pitfalls
- Common mistake: treating any ONE of these four features as sufficient grounds for an alert; at this reduced telemetry depth, each feature individually has a meaningfully high false-positive rate (an unusual but entirely legitimate one-off administrative task can trigger parent novelty, time-of-day deviation, or both), and the combined score exists specifically to require CORROBORATION across independent signals before an alert fires.
- Common mistake: baselining destination or time-of-day novelty against a GLOBAL population instead of per-host or per-account history; this either misses attacks on unusually-active hosts or floods quiet hosts with false positives for entirely ordinary variation.
- Command-line entropy is the weakest of the four features WITHOUT full command-line capture: if only a truncated or partial command-line is available, the entropy proxy degrades significantly, and the design should be explicit that this feature's reliability depends directly on how much command-line detail the endpoint agent actually captures, not treated as equally strong to the other three regardless of capture depth.
- This entire approach is a deliberate telemetry-constrained fallback, not a substitute for richer visibility: if full command-line capture, DLL/module-load events, or packet-level netflow become available later, several of these proxy features (especially the entropy proxy) should be replaced with the direct signal rather than kept as permanent design choices; the scoring-model structure (weighted combination requiring corroboration) is the durable part, the specific proxy features are not.
Unlock Full Question Bank
Get access to all Security Monitoring, SIEM, and Detection Engineering interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.